[jira] [Updated] (HADOOP-15087) S3A to support writing directly to the destination dir without creating temp directory to avoid rename
[ https://issues.apache.org/jira/browse/HADOOP-15087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15087: Affects Version/s: 3.0.0 Issue Type: New Feature (was: Improvement) > S3A to support writing directly to the destination dir without creating temp > directory to avoid rename > --- > > Key: HADOOP-15087 > URL: https://issues.apache.org/jira/browse/HADOOP-15087 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3 >Affects Versions: 3.0.0 >Reporter: Yonger > > Rename in workloads like Teragen/Terasort who use Hadoop default > outputcommitters really hurt performance a lot. > Stocator announce it doesn't create the temporary directories any all, and > still preserves Hadoop's fault tolerance. I add a switch when creating file > via integrating it's code into s3a, I got 5x performance gain in Teragen and > 15% performance improvement in Terasort. > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15087) S3A to support writing directly to the destination dir without creating temp directory to avoid rename
[ https://issues.apache.org/jira/browse/HADOOP-15087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15087: Summary: S3A to support writing directly to the destination dir without creating temp directory to avoid rename (was: Write directly without creating temp directory to avoid rename ) > S3A to support writing directly to the destination dir without creating temp > directory to avoid rename > --- > > Key: HADOOP-15087 > URL: https://issues.apache.org/jira/browse/HADOOP-15087 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.0.0 >Reporter: Yonger > > Rename in workloads like Teragen/Terasort who use Hadoop default > outputcommitters really hurt performance a lot. > Stocator announce it doesn't create the temporary directories any all, and > still preserves Hadoop's fault tolerance. I add a switch when creating file > via integrating it's code into s3a, I got 5x performance gain in Teragen and > 15% performance improvement in Terasort. > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
