[jira] [Commented] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251020#comment-16251020 ] Yonger commented on HADOOP-14475: - [~mackrorysd] Thank you very much. For the test case, I removed the assertion for file status counter while it's not necessary. For hard cord, that I copy from some other place(maybe old azure fs implementation, forgot), now I don't use fix length string to append, so this hard code also removed. At last, I just only put this setting into metrics property file: s3afilesystem.sink.file.filename=s3afilesystem-metrics.out which would print all metrics info into the specify file. > Metrics of S3A don't print out when enable it in Hadoop metrics property file > -- > > Key: HADOOP-14475 > URL: https://issues.apache.org/jira/browse/HADOOP-14475 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: uname -a > Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017 > x86_64 x86_64 x86_64 GNU/Linux > cat /etc/issue > Ubuntu 16.04.2 LTS \n \l >Reporter: Yonger >Assignee: Yonger > Attachments: HADOOP-14475-003.patch, HADOOP-14475.002.patch, > HADOOP-14475.005.patch, HADOOP-14475.006.patch, HADOOP-14475.008.patch, > HADOOP-14775.007.patch, failsafe-report-s3a-it.html, > failsafe-report-s3a-scale.html, failsafe-report-scale.html, > failsafe-report-scale.zip, s3a-metrics.patch1, stdout.zip > > > *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink > #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #*.sink.influxdb.url=http:/xx > #*.sink.influxdb.influxdb_port=8086 > #*.sink.influxdb.database=hadoop > #*.sink.influxdb.influxdb_username=hadoop > #*.sink.influxdb.influxdb_password=hadoop > #*.sink.ingluxdb.cluster=c1 > *.period=10 > #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out > I can't find the out put file even i run a MR job which should be used s3. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15038) Abstract MetadataStore in S3Guard into a common module.
[
https://issues.apache.org/jira/browse/HADOOP-15038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Genmao Yu updated HADOOP-15038:
---
Description:
Open this JIRA to discuss if we should move {{MetadataStore}} in {{S3Guard}}
into a common module.
Based on this work, other filesystem or object store can implement their own
metastore for optimization (known issues like consistency problem and metadata
operation performance). [~ste...@apache.org] and other guys have done many base
and great works in {{S3Guard}}. It is very helpful to start work. I did some
perf test in HADOOP-14098, and started related work for Aliyun OSS. Indeed
there are still works to do for {{S3Guard}}, like metadata cache inconsistent
with S3 and so on. It also will be a problem for other object store. However,
we can do these works in parallel.
cc [~drankye]
was:
Open this JIRA to discuss if we should move {{MetadataStore}} in {{S3Guard}}
into a common module. Based on this work, other filesystem or object store can
implement their own metastore for optimization (known issues like consistency
problem and metadata operation performance). [~ste...@apache.org] and other
guys have done many base and great works in {{S3Guard}}. It is very helpful to
start work. I did some perf test in HADOOP-14098, and started related work for
Aliyun OSS. Indeed there are still works to do for {{S3Guard}}, like metadata
cache inconsistent with S3 and so on. It also will be a problem for other
object store. However, we can do these works in parallel.
cc [~drankye]
> Abstract MetadataStore in S3Guard into a common module.
> ---
>
> Key: HADOOP-15038
> URL: https://issues.apache.org/jira/browse/HADOOP-15038
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs
>Affects Versions: 3.0.0-beta1
>Reporter: Genmao Yu
>
> Open this JIRA to discuss if we should move {{MetadataStore}} in {{S3Guard}}
> into a common module.
> Based on this work, other filesystem or object store can implement their own
> metastore for optimization (known issues like consistency problem and
> metadata operation performance). [~ste...@apache.org] and other guys have
> done many base and great works in {{S3Guard}}. It is very helpful to start
> work. I did some perf test in HADOOP-14098, and started related work for
> Aliyun OSS. Indeed there are still works to do for {{S3Guard}}, like
> metadata cache inconsistent with S3 and so on. It also will be a problem for
> other object store. However, we can do these works in parallel.
> cc [~drankye]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15038) Abstract {{MetadataStore}} in {{S3Guard}} into a common module.
Genmao Yu created HADOOP-15038:
--
Summary: Abstract {{MetadataStore}} in {{S3Guard}} into a common
module.
Key: HADOOP-15038
URL: https://issues.apache.org/jira/browse/HADOOP-15038
Project: Hadoop Common
Issue Type: New Feature
Components: fs
Affects Versions: 3.0.0-beta1
Reporter: Genmao Yu
Open this JIRA to discuss if we should move {{MetadataStore}} in {{S3Guard}}
into a common module. Based on this work, other filesystem or object store can
implement their own metastore for optimization (known issues like consistency
problem and metadata operation performance). [~ste...@apache.org] and other
guys have done many base and great works in {{S3Guard}}. It is very helpful to
start work. I did some perf test in HADOOP-14098, and started related work for
Aliyun OSS. Indeed there are still works to do for {{S3Guard}}, like metadata
cache inconsistent with S3 and so on. It also will be a problem for other
object store. However, we can do these works in parallel.
cc [~drankye]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15038) Abstract MetadataStore in S3Guard into a common module.
[
https://issues.apache.org/jira/browse/HADOOP-15038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Genmao Yu updated HADOOP-15038:
---
Summary: Abstract MetadataStore in S3Guard into a common module. (was:
Abstract {{MetadataStore}} in {{S3Guard}} into a common module.)
> Abstract MetadataStore in S3Guard into a common module.
> ---
>
> Key: HADOOP-15038
> URL: https://issues.apache.org/jira/browse/HADOOP-15038
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs
>Affects Versions: 3.0.0-beta1
>Reporter: Genmao Yu
>
> Open this JIRA to discuss if we should move {{MetadataStore}} in {{S3Guard}}
> into a common module. Based on this work, other filesystem or object store
> can implement their own metastore for optimization (known issues like
> consistency problem and metadata operation performance). [~ste...@apache.org]
> and other guys have done many base and great works in {{S3Guard}}. It is very
> helpful to start work. I did some perf test in HADOOP-14098, and started
> related work for Aliyun OSS. Indeed there are still works to do for
> {{S3Guard}}, like metadata cache inconsistent with S3 and so on. It also will
> be a problem for other object store. However, we can do these works in
> parallel.
> cc [~drankye]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yonger updated HADOOP-14475: Attachment: HADOOP-14475.008.patch update: 1.chang prefix and context name to dfs style 2.give each record a different name by the generated metric source name 3.add s3a metric system destroy logic in file system close method > Metrics of S3A don't print out when enable it in Hadoop metrics property file > -- > > Key: HADOOP-14475 > URL: https://issues.apache.org/jira/browse/HADOOP-14475 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: uname -a > Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017 > x86_64 x86_64 x86_64 GNU/Linux > cat /etc/issue > Ubuntu 16.04.2 LTS \n \l >Reporter: Yonger >Assignee: Yonger > Attachments: HADOOP-14475-003.patch, HADOOP-14475.002.patch, > HADOOP-14475.005.patch, HADOOP-14475.006.patch, HADOOP-14475.008.patch, > HADOOP-14775.007.patch, failsafe-report-s3a-it.html, > failsafe-report-s3a-scale.html, failsafe-report-scale.html, > failsafe-report-scale.zip, s3a-metrics.patch1, stdout.zip > > > *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink > #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #*.sink.influxdb.url=http:/xx > #*.sink.influxdb.influxdb_port=8086 > #*.sink.influxdb.database=hadoop > #*.sink.influxdb.influxdb_username=hadoop > #*.sink.influxdb.influxdb_password=hadoop > #*.sink.ingluxdb.cluster=c1 > *.period=10 > #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out > I can't find the out put file even i run a MR job which should be used s3. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15027) Improvements for Hadoop read from AliyunOSS
[
https://issues.apache.org/jira/browse/HADOOP-15027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251008#comment-16251008
]
Genmao Yu commented on HADOOP-15027:
[~wujinhu] backport {{Hadoop-aliyun}} module to which branch is still discuss
in progress, HADOOP-14964 FYI.
> Improvements for Hadoop read from AliyunOSS
> ---
>
> Key: HADOOP-15027
> URL: https://issues.apache.org/jira/browse/HADOOP-15027
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/oss
>Affects Versions: 3.0.0
>Reporter: wujinhu
>Assignee: wujinhu
> Attachments: HADOOP-15027.001.patch, HADOOP-15027.002.patch,
> HADOOP-15027.003.patch
>
>
> Currently, read performance is poor when Hadoop reads from AliyunOSS. It
> needs about 1min to read 1GB from OSS.
> Class AliyunOSSInputStream uses single thread to read data from AliyunOSS,
> so we can refactor this by using multi-thread pre read to improve this.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (HADOOP-15027) Improvements for Hadoop read from AliyunOSS
[ https://issues.apache.org/jira/browse/HADOOP-15027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wujinhu updated HADOOP-15027: - Comment: was deleted (was: [~stevel]) > Improvements for Hadoop read from AliyunOSS > --- > > Key: HADOOP-15027 > URL: https://issues.apache.org/jira/browse/HADOOP-15027 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/oss >Affects Versions: 3.0.0 >Reporter: wujinhu >Assignee: wujinhu > Attachments: HADOOP-15027.001.patch, HADOOP-15027.002.patch, > HADOOP-15027.003.patch > > > Currently, read performance is poor when Hadoop reads from AliyunOSS. It > needs about 1min to read 1GB from OSS. > Class AliyunOSSInputStream uses single thread to read data from AliyunOSS, > so we can refactor this by using multi-thread pre read to improve this. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15027) Improvements for Hadoop read from AliyunOSS
[ https://issues.apache.org/jira/browse/HADOOP-15027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16250965#comment-16250965 ] wujinhu commented on HADOOP-15027: -- [~stevel] > Improvements for Hadoop read from AliyunOSS > --- > > Key: HADOOP-15027 > URL: https://issues.apache.org/jira/browse/HADOOP-15027 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/oss >Affects Versions: 3.0.0 >Reporter: wujinhu >Assignee: wujinhu > Attachments: HADOOP-15027.001.patch, HADOOP-15027.002.patch, > HADOOP-15027.003.patch > > > Currently, read performance is poor when Hadoop reads from AliyunOSS. It > needs about 1min to read 1GB from OSS. > Class AliyunOSSInputStream uses single thread to read data from AliyunOSS, > so we can refactor this by using multi-thread pre read to improve this. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15027) Improvements for Hadoop read from AliyunOSS
[ https://issues.apache.org/jira/browse/HADOOP-15027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16250933#comment-16250933 ] wujinhu commented on HADOOP-15027: -- Yes, agree with [~uncleGen] . We could optimize random IO step by step. As we all known, Hadoop 2.7.5 will be released soon. We hope this patch can be released so that we can solve the sequential IO issue(single thread read) in current implementation. Random IO remains the same, and we can solve this later. I have read class _*SemaphoredDelegatingExecutor*_ and it's good enough. I think [~uncleGen] you can do this job, thanks. Besides, I will provide more detailed test results later. > Improvements for Hadoop read from AliyunOSS > --- > > Key: HADOOP-15027 > URL: https://issues.apache.org/jira/browse/HADOOP-15027 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/oss >Affects Versions: 3.0.0 >Reporter: wujinhu >Assignee: wujinhu > Attachments: HADOOP-15027.001.patch, HADOOP-15027.002.patch, > HADOOP-15027.003.patch > > > Currently, read performance is poor when Hadoop reads from AliyunOSS. It > needs about 1min to read 1GB from OSS. > Class AliyunOSSInputStream uses single thread to read data from AliyunOSS, > so we can refactor this by using multi-thread pre read to improve this. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14964) AliyunOSS: backport Aliyun OSS module to branch-2 and 2.7+ branches
[ https://issues.apache.org/jira/browse/HADOOP-14964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16250931#comment-16250931 ] Kai Zheng commented on HADOOP-14964: Thanks Steve for the comments. bq. We can worry about 2.9, but I don't expect any patches going in to 2.7-2.8, which are essentially maintenance only right now. Looking at the latest discussions, I roughly agree. It sounds reasonable to freeze 2.7 branch, but for 2.8, I'm not very sure. It is more reasonable to support 2.8 branch longer time like it did for 2.7 branch. Will discuss in the common-dev thread. > AliyunOSS: backport Aliyun OSS module to branch-2 and 2.7+ branches > --- > > Key: HADOOP-14964 > URL: https://issues.apache.org/jira/browse/HADOOP-14964 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss >Reporter: Genmao Yu >Assignee: SammiChen > Attachments: HADOOP-14964-branch-2.000.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (HADOOP-15027) Improvements for Hadoop read from AliyunOSS
[ https://issues.apache.org/jira/browse/HADOOP-15027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wujinhu updated HADOOP-15027: - Comment: was deleted (was: ste...@apache.org ) > Improvements for Hadoop read from AliyunOSS > --- > > Key: HADOOP-15027 > URL: https://issues.apache.org/jira/browse/HADOOP-15027 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/oss >Affects Versions: 3.0.0 >Reporter: wujinhu >Assignee: wujinhu > Attachments: HADOOP-15027.001.patch, HADOOP-15027.002.patch, > HADOOP-15027.003.patch > > > Currently, read performance is poor when Hadoop reads from AliyunOSS. It > needs about 1min to read 1GB from OSS. > Class AliyunOSSInputStream uses single thread to read data from AliyunOSS, > so we can refactor this by using multi-thread pre read to improve this. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15027) Improvements for Hadoop read from AliyunOSS
[
https://issues.apache.org/jira/browse/HADOOP-15027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16250920#comment-16250920
]
Genmao Yu commented on HADOOP-15027:
How to optimize random IO is a major piece of work. IIUC, [~ste...@apache.org]
you said you would focus on performance of column format file, i.e. random IO.
Is there any jira to trace?
[~wujinhu] Let us focus on how to improve sequential IO, but not random IO.
IMHO, {{SemaphoredDelegatingExecutor}} is a good common class, we may move this
class to hadoop-common. I will open a jira to do this work if you guys do not
mind. So this jira will be pending for a while. Besides, could you please post
a more detailed performance test result?
> Improvements for Hadoop read from AliyunOSS
> ---
>
> Key: HADOOP-15027
> URL: https://issues.apache.org/jira/browse/HADOOP-15027
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/oss
>Affects Versions: 3.0.0
>Reporter: wujinhu
>Assignee: wujinhu
> Attachments: HADOOP-15027.001.patch, HADOOP-15027.002.patch,
> HADOOP-15027.003.patch
>
>
> Currently, read performance is poor when Hadoop reads from AliyunOSS. It
> needs about 1min to read 1GB from OSS.
> Class AliyunOSSInputStream uses single thread to read data from AliyunOSS,
> so we can refactor this by using multi-thread pre read to improve this.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15027) Improvements for Hadoop read from AliyunOSS
[ https://issues.apache.org/jira/browse/HADOOP-15027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16250921#comment-16250921 ] wujinhu commented on HADOOP-15027: -- ste...@apache.org > Improvements for Hadoop read from AliyunOSS > --- > > Key: HADOOP-15027 > URL: https://issues.apache.org/jira/browse/HADOOP-15027 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/oss >Affects Versions: 3.0.0 >Reporter: wujinhu >Assignee: wujinhu > Attachments: HADOOP-15027.001.patch, HADOOP-15027.002.patch, > HADOOP-15027.003.patch > > > Currently, read performance is poor when Hadoop reads from AliyunOSS. It > needs about 1min to read 1GB from OSS. > Class AliyunOSSInputStream uses single thread to read data from AliyunOSS, > so we can refactor this by using multi-thread pre read to improve this. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14999) AliyunOSS: provide one asynchronous multi-part based uploading mechanism
[
https://issues.apache.org/jira/browse/HADOOP-14999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16250906#comment-16250906
]
Genmao Yu commented on HADOOP-14999:
pending on refactoring: move the {{TaskEngine}} from output stream to oss
filesystem .
Just as discussed in HADOOP-15027, I think
{{org.apache.hadoop.fs.s3a.SemaphoredDelegatingExecutor}} is a good common
class, and we may move it to hadoop-common.
Then, I will refactor the {{TaskEngine}} to use
{{SemaphoredDelegatingExecutor}}.
[~ste...@apache.org] Do you mind if I open jira to do this work?
> AliyunOSS: provide one asynchronous multi-part based uploading mechanism
>
>
> Key: HADOOP-14999
> URL: https://issues.apache.org/jira/browse/HADOOP-14999
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/oss
>Affects Versions: 3.0.0-beta1
>Reporter: Genmao Yu
>Assignee: Genmao Yu
> Attachments: HADOOP-14999.001.patch, HADOOP-14999.002.patch
>
>
> This mechanism is designed for uploading file in parallel and asynchronously:
> - improve the performance of uploading file to OSS server. Firstly, this
> mechanism splits result to multiple small blocks and upload them in parallel.
> Then, getting result and uploading blocks are asynchronous.
> - avoid buffering too large result into local disk. To cite an extreme
> example, there is a task which will output 100GB or even larger, we may need
> to output this 100GB to local disk and then upload it. Sometimes, it is
> inefficient and limited to disk space.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15032) Enable Optimize Hadoop RPC encryption performance for branch-2
[
https://issues.apache.org/jira/browse/HADOOP-15032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16250901#comment-16250901
]
Hadoop QA commented on HADOOP-15032:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 14m
50s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
48s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
49s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
29s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
52s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
57s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
55s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
48s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 10m
38s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 10m 38s{color}
| {color:red} root generated 2 new + 1433 unchanged - 5 fixed = 1435 total (was
1438) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 50s{color} | {color:orange} root: The patch generated 34 new + 862 unchanged
- 31 fixed = 896 total (was 893) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
0s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
46s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 15m 55s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
24s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 78m 59s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}178m 34s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Unreaped Processes | hadoop-common:1 |
| | hadoop-hdfs:20 |
| Failed junit tests |
hadoop.hdfs.server.blockmanagement.TestPendingReplication |
| | hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks |
| Timed out junit tests | org.apache.hadoop.log.TestLogLevel |
| | org.apache.hadoop.hdfs.TestWriteRead |
| | org.apache.hadoop.hdfs.server.blockmanagement.TestHeartbeatHandling |
| | org.apache.hadoop.hdfs.server.blockmanagement.TestReplicationPolicy |
| | org.apache.hadoop.hdfs.TestReservedRawPaths |
| |
org.apache.hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain
|
| | org.apache.hadoop.hdfs.TestAclsEndToEnd |
| |
[jira] [Issue Comment Deleted] (HADOOP-14999) AliyunOSS: provide one asynchronous multi-part based uploading mechanism
[ https://issues.apache.org/jira/browse/HADOOP-14999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Genmao Yu updated HADOOP-14999: --- Comment: was deleted (was: [~ste...@apache.org] and [~drankye] Any feedback is appreciated) > AliyunOSS: provide one asynchronous multi-part based uploading mechanism > > > Key: HADOOP-14999 > URL: https://issues.apache.org/jira/browse/HADOOP-14999 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/oss >Affects Versions: 3.0.0-beta1 >Reporter: Genmao Yu >Assignee: Genmao Yu > Attachments: HADOOP-14999.001.patch, HADOOP-14999.002.patch > > > This mechanism is designed for uploading file in parallel and asynchronously: > - improve the performance of uploading file to OSS server. Firstly, this > mechanism splits result to multiple small blocks and upload them in parallel. > Then, getting result and uploading blocks are asynchronous. > - avoid buffering too large result into local disk. To cite an extreme > example, there is a task which will output 100GB or even larger, we may need > to output this 100GB to local disk and then upload it. Sometimes, it is > inefficient and limited to disk space. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-15028) Got errors while running org.apache.hadoop.io.TestSequenceFileAppend
[ https://issues.apache.org/jira/browse/HADOOP-15028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] bd17kaka reopened HADOOP-15028: --- > Got errors while running org.apache.hadoop.io.TestSequenceFileAppend > > > Key: HADOOP-15028 > URL: https://issues.apache.org/jira/browse/HADOOP-15028 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.6.4 > Environment: Linux 2.6.32-642.el6.x86_64 >Reporter: bd17kaka > > I ran the test case org.apache.hadoop.io.TestSequenceFileAppend in > branch-2.6.4, I got the following errors: > Running org.apache.hadoop.io.TestSequenceFileAppend > Tests run: 4, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 0.801 sec <<< > FAILURE! - in org.apache.hadoop.io.TestSequenceFileAppend > testAppendBlockCompression(org.apache.hadoop.io.TestSequenceFileAppend) Time > elapsed: 0.117 sec <<< ERROR! > java.io.IOException: File is corrupt! > at > org.apache.hadoop.io.SequenceFile$Reader.readBlock(SequenceFile.java:2179) > at > org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2583) > at > org.apache.hadoop.io.TestSequenceFileAppend.verifyAll4Values(TestSequenceFileAppend.java:309) > at > org.apache.hadoop.io.TestSequenceFileAppend.testAppendBlockCompression(TestSequenceFileAppend.java:205) > testAppendSort(org.apache.hadoop.io.TestSequenceFileAppend) Time elapsed: > 0.013 sec <<< ERROR! > java.io.IOException: File is corrupt! > at > org.apache.hadoop.io.SequenceFile$Reader.readBlock(SequenceFile.java:2179) > at > org.apache.hadoop.io.SequenceFile$Reader.nextRaw(SequenceFile.java:2488) > at > org.apache.hadoop.io.SequenceFile$Sorter$SortPass.run(SequenceFile.java:2923) > at > org.apache.hadoop.io.SequenceFile$Sorter.sortPass(SequenceFile.java:2861) > at > org.apache.hadoop.io.SequenceFile$Sorter.sort(SequenceFile.java:2809) > at > org.apache.hadoop.io.SequenceFile$Sorter.sort(SequenceFile.java:2850) > at > org.apache.hadoop.io.TestSequenceFileAppend.testAppendSort(TestSequenceFileAppend.java:286) > But everything is OK in branch-2.6.5.. > The maven command is 'mvn test -Pnative -Dtest=TestSequenceFileAppend' -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-15028) Got errors while running org.apache.hadoop.io.TestSequenceFileAppend
[ https://issues.apache.org/jira/browse/HADOOP-15028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] bd17kaka resolved HADOOP-15028. --- Resolution: Fixed > Got errors while running org.apache.hadoop.io.TestSequenceFileAppend > > > Key: HADOOP-15028 > URL: https://issues.apache.org/jira/browse/HADOOP-15028 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.6.4 > Environment: Linux 2.6.32-642.el6.x86_64 >Reporter: bd17kaka > > I ran the test case org.apache.hadoop.io.TestSequenceFileAppend in > branch-2.6.4, I got the following errors: > Running org.apache.hadoop.io.TestSequenceFileAppend > Tests run: 4, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 0.801 sec <<< > FAILURE! - in org.apache.hadoop.io.TestSequenceFileAppend > testAppendBlockCompression(org.apache.hadoop.io.TestSequenceFileAppend) Time > elapsed: 0.117 sec <<< ERROR! > java.io.IOException: File is corrupt! > at > org.apache.hadoop.io.SequenceFile$Reader.readBlock(SequenceFile.java:2179) > at > org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2583) > at > org.apache.hadoop.io.TestSequenceFileAppend.verifyAll4Values(TestSequenceFileAppend.java:309) > at > org.apache.hadoop.io.TestSequenceFileAppend.testAppendBlockCompression(TestSequenceFileAppend.java:205) > testAppendSort(org.apache.hadoop.io.TestSequenceFileAppend) Time elapsed: > 0.013 sec <<< ERROR! > java.io.IOException: File is corrupt! > at > org.apache.hadoop.io.SequenceFile$Reader.readBlock(SequenceFile.java:2179) > at > org.apache.hadoop.io.SequenceFile$Reader.nextRaw(SequenceFile.java:2488) > at > org.apache.hadoop.io.SequenceFile$Sorter$SortPass.run(SequenceFile.java:2923) > at > org.apache.hadoop.io.SequenceFile$Sorter.sortPass(SequenceFile.java:2861) > at > org.apache.hadoop.io.SequenceFile$Sorter.sort(SequenceFile.java:2809) > at > org.apache.hadoop.io.SequenceFile$Sorter.sort(SequenceFile.java:2850) > at > org.apache.hadoop.io.TestSequenceFileAppend.testAppendSort(TestSequenceFileAppend.java:286) > But everything is OK in branch-2.6.5.. > The maven command is 'mvn test -Pnative -Dtest=TestSequenceFileAppend' -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14999) AliyunOSS: provide one asynchronous multi-part based uploading mechanism
[ https://issues.apache.org/jira/browse/HADOOP-14999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16250772#comment-16250772 ] Genmao Yu commented on HADOOP-14999: [~ste...@apache.org] and [~drankye] Any feedback is appreciated > AliyunOSS: provide one asynchronous multi-part based uploading mechanism > > > Key: HADOOP-14999 > URL: https://issues.apache.org/jira/browse/HADOOP-14999 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/oss >Affects Versions: 3.0.0-beta1 >Reporter: Genmao Yu >Assignee: Genmao Yu > Attachments: HADOOP-14999.001.patch, HADOOP-14999.002.patch > > > This mechanism is designed for uploading file in parallel and asynchronously: > - improve the performance of uploading file to OSS server. Firstly, this > mechanism splits result to multiple small blocks and upload them in parallel. > Then, getting result and uploading blocks are asynchronous. > - avoid buffering too large result into local disk. To cite an extreme > example, there is a task which will output 100GB or even larger, we may need > to output this 100GB to local disk and then upload it. Sometimes, it is > inefficient and limited to disk space. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14993) AliyunOSS: Override listFiles and listLocatedStatus
[ https://issues.apache.org/jira/browse/HADOOP-14993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16250770#comment-16250770 ] Genmao Yu commented on HADOOP-14993: [~drankye] and [~ste...@apache.org] Any feedback is appreciated > AliyunOSS: Override listFiles and listLocatedStatus > > > Key: HADOOP-14993 > URL: https://issues.apache.org/jira/browse/HADOOP-14993 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/oss >Affects Versions: 3.0.0-beta1 >Reporter: Genmao Yu >Assignee: Genmao Yu > Attachments: HADOOP-14993.001.patch, HADOOP-14993.002.patch, > HADOOP-14993.003.patch > > > Do a bulk listing off all entries under a path in one single operation, there > is no need to recursively walk the directory tree. > Updates: > - override listFiles and listLocatedStatus by using bulk listing > - some minor updates in hadoop-aliyun index.md -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15037) Add site release notes for OrgQueue and resource types
[ https://issues.apache.org/jira/browse/HADOOP-15037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16250749#comment-16250749 ] Hudson commented on HADOOP-15037: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13234 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13234/]) HADOOP-15037. Add site release notes for OrgQueue and resource types. (wang: rev 8b125741659a825c71877bd1b1cb8f7e3ef26436) * (edit) hadoop-project/src/site/markdown/index.md.vm > Add site release notes for OrgQueue and resource types > -- > > Key: HADOOP-15037 > URL: https://issues.apache.org/jira/browse/HADOOP-15037 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Andrew Wang >Assignee: Andrew Wang > Fix For: 3.0.0 > > Attachments: HADOOP-15037.001.patch > > > Let's add some small blurbs and doc links to the site release notes for these > features. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15037) Add site release notes for OrgQueue and resource types
[ https://issues.apache.org/jira/browse/HADOOP-15037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HADOOP-15037: - Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Thanks for reviewing Wei-Chiu, I've committed this to trunk and branch-3.0. > Add site release notes for OrgQueue and resource types > -- > > Key: HADOOP-15037 > URL: https://issues.apache.org/jira/browse/HADOOP-15037 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Andrew Wang >Assignee: Andrew Wang > Fix For: 3.0.0 > > Attachments: HADOOP-15037.001.patch > > > Let's add some small blurbs and doc links to the site release notes for these > features. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14840) Tool to estimate resource requirements of an application pipeline based on prior executions
[
https://issues.apache.org/jira/browse/HADOOP-14840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Subru Krishnan updated HADOOP-14840:
Release Note: The first version of Resource Estimator service, a tool that
captures the historical resource usage of an app and predicts its future
resource requirement. (was: The first version of Resource Estimator service, a
tool that captures the historical resource usage of an app and predicts it's
future resource requirement.)
> Tool to estimate resource requirements of an application pipeline based on
> prior executions
> ---
>
> Key: HADOOP-14840
> URL: https://issues.apache.org/jira/browse/HADOOP-14840
> Project: Hadoop Common
> Issue Type: New Feature
> Components: tools
>Reporter: Subru Krishnan
>Assignee: Rui Li
> Fix For: 2.9.0, 3.0.0, 3.1.0
>
> Attachments: HADOOP-14840-branch-2-v4.patch, HADOOP-14840-v1.patch,
> HADOOP-14840-v2.patch, HADOOP-14840-v3.patch, HADOOP-14840-v4.patch,
> ResourceEstimator-design-v1.pdf
>
>
> We have been working on providing SLAs for job execution on Hadoop. At high
> level this involves 2 parts: deriving the resource requirements of a job and
> guaranteeing the estimated resources at runtime. The {{YARN
> ReservationSystem}} (YARN-1051/YARN-2572/YARN-5326) enable the latter and in
> this JIRA, we propose to add a tool to Hadoop to predict the resource
> requirements of a job based on past executions of the job. The system (aka
> *Morpheus*) deep dive can be found in our OSDI'16 paper
> [here|https://www.usenix.org/conference/osdi16/technical-sessions/presentation/jyothi].
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15003) Merge S3A committers into trunk: Yetus patch checker
[ https://issues.apache.org/jira/browse/HADOOP-15003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16250698#comment-16250698 ] Aaron Fabbri commented on HADOOP-15003: --- I'm running the latest patch through my "repeat integration and scale tests overnight" script again. I'll post an update tomorrow morning-ish Pacific time. > Merge S3A committers into trunk: Yetus patch checker > > > Key: HADOOP-15003 > URL: https://issues.apache.org/jira/browse/HADOOP-15003 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13786-041.patch, HADOOP-13786-042.patch, > HADOOP-13786-043.patch, HADOOP-13786-044.patch, HADOOP-13786-045.patch, > HADOOP-13786-046.patch, HADOOP-13786-047.patch, HADOOP-13786-048.patch > > > This is a Yetus only JIRA created to have Yetus review the > HADOOP-13786/HADOOP-14971 patch as a .patch file, as the review PR > [https://github.com/apache/hadoop/pull/282] is stopping this happening in > HADOOP-14971. > Reviews should go into the PR/other task -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15032) Enable Optimize Hadoop RPC encryption performance for branch-2
[ https://issues.apache.org/jira/browse/HADOOP-15032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dapeng Sun updated HADOOP-15032: Attachment: HADOOP-15032-branch-2-002.patch > Enable Optimize Hadoop RPC encryption performance for branch-2 > -- > > Key: HADOOP-15032 > URL: https://issues.apache.org/jira/browse/HADOOP-15032 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.8.1 >Reporter: Dapeng Sun >Assignee: Dapeng Sun > Attachments: HADOOP-15032-branch-2-001.patch, > HADOOP-15032-branch-2-002.patch > > > Enable Optimize Hadoop RPC encryption performance for branch-2 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15037) Add site release notes for OrgQueue and resource types
[
https://issues.apache.org/jira/browse/HADOOP-15037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16250689#comment-16250689
]
Hadoop QA commented on HADOOP-15037:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
9s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
25m 59s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 58s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 37m 57s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-15037 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12897423/HADOOP-15037.001.patch
|
| Optional Tests | asflicense mvnsite |
| uname | Linux 66766d647201 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 5323b00 |
| maven | version: Apache Maven 3.3.9 |
| Max. process+thread count | 336 (vs. ulimit of 5000) |
| modules | C: hadoop-project U: hadoop-project |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13672/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add site release notes for OrgQueue and resource types
> --
>
> Key: HADOOP-15037
> URL: https://issues.apache.org/jira/browse/HADOOP-15037
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Andrew Wang
>Assignee: Andrew Wang
> Attachments: HADOOP-15037.001.patch
>
>
> Let's add some small blurbs and doc links to the site release notes for these
> features.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15037) Add site release notes for OrgQueue and resource types
[ https://issues.apache.org/jira/browse/HADOOP-15037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16250660#comment-16250660 ] Wei-Chiu Chuang commented on HADOOP-15037: -- +1 LGTM. Thanks [~andrew.wang]! > Add site release notes for OrgQueue and resource types > -- > > Key: HADOOP-15037 > URL: https://issues.apache.org/jira/browse/HADOOP-15037 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: HADOOP-15037.001.patch > > > Let's add some small blurbs and doc links to the site release notes for these > features. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15037) Add site release notes for OrgQueue and resource types
[ https://issues.apache.org/jira/browse/HADOOP-15037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HADOOP-15037: - Attachment: HADOOP-15037.001.patch Patch attached, [~eddyxu] / [~templedf] could you review? > Add site release notes for OrgQueue and resource types > -- > > Key: HADOOP-15037 > URL: https://issues.apache.org/jira/browse/HADOOP-15037 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: HADOOP-15037.001.patch > > > Let's add some small blurbs and doc links to the site release notes for these > features. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15037) Add site release notes for OrgQueue and resource types
[ https://issues.apache.org/jira/browse/HADOOP-15037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HADOOP-15037: - Target Version/s: 3.0.0 > Add site release notes for OrgQueue and resource types > -- > > Key: HADOOP-15037 > URL: https://issues.apache.org/jira/browse/HADOOP-15037 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: HADOOP-15037.001.patch > > > Let's add some small blurbs and doc links to the site release notes for these > features. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15037) Add site release notes for OrgQueue and resource types
[ https://issues.apache.org/jira/browse/HADOOP-15037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HADOOP-15037: - Status: Patch Available (was: Open) > Add site release notes for OrgQueue and resource types > -- > > Key: HADOOP-15037 > URL: https://issues.apache.org/jira/browse/HADOOP-15037 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: HADOOP-15037.001.patch > > > Let's add some small blurbs and doc links to the site release notes for these > features. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15037) Add site release notes for OrgQueue and resource types
Andrew Wang created HADOOP-15037: Summary: Add site release notes for OrgQueue and resource types Key: HADOOP-15037 URL: https://issues.apache.org/jira/browse/HADOOP-15037 Project: Hadoop Common Issue Type: Improvement Reporter: Andrew Wang Assignee: Andrew Wang Let's add some small blurbs and doc links to the site release notes for these features. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15008) Metrics sinks may emit too frequently if multiple sink periods are configured
[
https://issues.apache.org/jira/browse/HADOOP-15008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16249863#comment-16249863
]
Eric Yang edited comment on HADOOP-15008 at 11/13/17 11:39 PM:
---
+1 Verified the test failure is not related to this patch. The value output
from different intervals seem to have correct values.
was (Author: eyang):
+1 Verified the test failure is not related to this patch. The value output
from different different intervals seem to have correct values.
> Metrics sinks may emit too frequently if multiple sink periods are configured
> -
>
> Key: HADOOP-15008
> URL: https://issues.apache.org/jira/browse/HADOOP-15008
> Project: Hadoop Common
> Issue Type: Bug
> Components: metrics
>Affects Versions: 2.2.0, 3.0.0-beta1
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Minor
> Fix For: 3.1.0
>
> Attachments: HADOOP-15008.000.patch
>
>
> If there are multiple metrics sink periods configured, depending on what
> those periods are, some sinks may emit too frequently. For example with the
> following:
> {code:title=hadoop-metrics2.properties}
> namenode.sink.file10.class=org.apache.hadoop.metrics2.sink.FileSink
> namenode.sink.file5.class=org.apache.hadoop.metrics2.sink.FileSink
> namenode.sink.file10.filename=namenode-metrics_per10.out
> namenode.sink.file5.filename=namenode-metrics_per5.out
> namenode.sink.file10.period=10
> namenode.sink.file5.period=5
> {code}
> I get the following:
> {code}
> ± for f in namenode-metrics_per*.out; do echo "$f" && grep
> "metricssystem.MetricsSystem" $f | awk '{last=curr; curr=$1} END { print
> curr-last }'; done
> namenode-metrics_per10.out
> 5000
> namenode-metrics_per5.out
> 5000
> {code}
> i.e., for both metrics files, each record is 5000 ms apart, even though one
> of the sinks has been configured to emit at 10s intervals
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14840) Tool to estimate resource requirements of an application pipeline based on prior executions
[
https://issues.apache.org/jira/browse/HADOOP-14840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16250455#comment-16250455
]
Hudson commented on HADOOP-14840:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13231 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13231/])
HADOOP-15036. Update LICENSE.txt for HADOOP-14840. (asuresh) (arun suresh: rev
f871b7541a5375eb117eafb9a091e4f59401231f)
* (edit) LICENSE.txt
> Tool to estimate resource requirements of an application pipeline based on
> prior executions
> ---
>
> Key: HADOOP-14840
> URL: https://issues.apache.org/jira/browse/HADOOP-14840
> Project: Hadoop Common
> Issue Type: New Feature
> Components: tools
>Reporter: Subru Krishnan
>Assignee: Rui Li
> Fix For: 2.9.0, 3.0.0, 3.1.0
>
> Attachments: HADOOP-14840-branch-2-v4.patch, HADOOP-14840-v1.patch,
> HADOOP-14840-v2.patch, HADOOP-14840-v3.patch, HADOOP-14840-v4.patch,
> ResourceEstimator-design-v1.pdf
>
>
> We have been working on providing SLAs for job execution on Hadoop. At high
> level this involves 2 parts: deriving the resource requirements of a job and
> guaranteeing the estimated resources at runtime. The {{YARN
> ReservationSystem}} (YARN-1051/YARN-2572/YARN-5326) enable the latter and in
> this JIRA, we propose to add a tool to Hadoop to predict the resource
> requirements of a job based on past executions of the job. The system (aka
> *Morpheus*) deep dive can be found in our OSDI'16 paper
> [here|https://www.usenix.org/conference/osdi16/technical-sessions/presentation/jyothi].
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15036) Update LICENSE.txt for HADOOP-14840
[ https://issues.apache.org/jira/browse/HADOOP-15036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16250454#comment-16250454 ] Hudson commented on HADOOP-15036: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13231 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13231/]) HADOOP-15036. Update LICENSE.txt for HADOOP-14840. (asuresh) (arun suresh: rev f871b7541a5375eb117eafb9a091e4f59401231f) * (edit) LICENSE.txt > Update LICENSE.txt for HADOOP-14840 > --- > > Key: HADOOP-15036 > URL: https://issues.apache.org/jira/browse/HADOOP-15036 > Project: Hadoop Common > Issue Type: Bug >Reporter: Arun Suresh >Assignee: Arun Suresh > Fix For: 2.9.0, 3.0.0 > > Attachments: HADOOP-15036.001.patch > > > As noticed by [~anu]: > Looks like HADOOP-14840 added a dependency on “oj! Algorithms - version > 43.0”, but we have just added “oj! Algorithms - version 43.0” to the > “LICENSE.txt”. The right addition to the LICENESE.txt should contain the > original MIT License, especially “Copyright (c) 2003-2017 Optimatika”. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15003) Merge S3A committers into trunk: Yetus patch checker
[
https://issues.apache.org/jira/browse/HADOOP-15003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16250430#comment-16250430
]
Hadoop QA commented on HADOOP-15003:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 58 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 51s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
33s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
40s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 11m 40s{color}
| {color:red} root generated 2 new + 1234 unchanged - 0 fixed = 1236 total (was
1234) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 9s{color} | {color:orange} root: The patch generated 55 new + 117 unchanged
- 22 fixed = 172 total (was 139) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
32s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 75 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
2s{color} | {color:red} The patch 3 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
6s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 56s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
59s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
24s{color} | {color:red} hadoop-tools_hadoop-aws generated 2 new + 1 unchanged
- 0 fixed = 3 total (was 1) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 45s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
53s{color} | {color:green} hadoop-yarn-registry in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
54s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
36s{color} | {color:green} hadoop-mapreduce-client-app in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
36s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License
[jira] [Resolved] (HADOOP-15036) Update LICENSE.txt for HADOOP-14840
[ https://issues.apache.org/jira/browse/HADOOP-15036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh resolved HADOOP-15036. -- Resolution: Fixed Fix Version/s: 3.0.0 2.9.0 Thanks for the rev [~anu] Committed to trunk, branch-3.0, branch-2, branch-2.9 & branch-2.9.0 > Update LICENSE.txt for HADOOP-14840 > --- > > Key: HADOOP-15036 > URL: https://issues.apache.org/jira/browse/HADOOP-15036 > Project: Hadoop Common > Issue Type: Bug >Reporter: Arun Suresh >Assignee: Arun Suresh > Fix For: 2.9.0, 3.0.0 > > Attachments: HADOOP-15036.001.patch > > > As noticed by [~anu]: > Looks like HADOOP-14840 added a dependency on “oj! Algorithms - version > 43.0”, but we have just added “oj! Algorithms - version 43.0” to the > “LICENSE.txt”. The right addition to the LICENESE.txt should contain the > original MIT License, especially “Copyright (c) 2003-2017 Optimatika”. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15036) Update LICENSE.txt for HADOOP-14840
[ https://issues.apache.org/jira/browse/HADOOP-15036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16250386#comment-16250386 ] Anu Engineer commented on HADOOP-15036: --- +1, Feel free to commit when you get a chance. Thanks for fixing this so quickly. > Update LICENSE.txt for HADOOP-14840 > --- > > Key: HADOOP-15036 > URL: https://issues.apache.org/jira/browse/HADOOP-15036 > Project: Hadoop Common > Issue Type: Bug >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: HADOOP-15036.001.patch > > > As noticed by [~anu]: > Looks like HADOOP-14840 added a dependency on “oj! Algorithms - version > 43.0”, but we have just added “oj! Algorithms - version 43.0” to the > “LICENSE.txt”. The right addition to the LICENESE.txt should contain the > original MIT License, especially “Copyright (c) 2003-2017 Optimatika”. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15036) Update LICENSE.txt for HADOOP-14840
[ https://issues.apache.org/jira/browse/HADOOP-15036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated HADOOP-15036: - Attachment: HADOOP-15036.001.patch Uploading initial patch. [~anu] Can you please take a look ? > Update LICENSE.txt for HADOOP-14840 > --- > > Key: HADOOP-15036 > URL: https://issues.apache.org/jira/browse/HADOOP-15036 > Project: Hadoop Common > Issue Type: Bug >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: HADOOP-15036.001.patch > > > As noticed by [~anu]: > Looks like HADOOP-14840 added a dependency on “oj! Algorithms - version > 43.0”, but we have just added “oj! Algorithms - version 43.0” to the > “LICENSE.txt”. The right addition to the LICENESE.txt should contain the > original MIT License, especially “Copyright (c) 2003-2017 Optimatika”. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15036) Update LICENSE.txt for HADOOP-14840
Arun Suresh created HADOOP-15036: Summary: Update LICENSE.txt for HADOOP-14840 Key: HADOOP-15036 URL: https://issues.apache.org/jira/browse/HADOOP-15036 Project: Hadoop Common Issue Type: Bug Reporter: Arun Suresh Assignee: Arun Suresh As noticed by [~anu]: Looks like HADOOP-14840 added a dependency on “oj! Algorithms - version 43.0”, but we have just added “oj! Algorithms - version 43.0” to the “LICENSE.txt”. The right addition to the LICENESE.txt should contain the original MIT License, especially “Copyright (c) 2003-2017 Optimatika”. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14104) Client should always ask namenode for kms provider path.
[
https://issues.apache.org/jira/browse/HADOOP-14104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16250323#comment-16250323
]
Xiao Chen commented on HADOOP-14104:
Thanks Rushabh and all for the contribution here.
Just a note:
We have had a downstream application broken, due to the 'cache the nameservice
to provider mapping into UGI credentials' logic:
- application operates with 2 clusters, which have the same NN nameservice.
- application loads to configuration objects with corresponding cluster, and
creates 2 separate dfsclients.
- All these are done using the same UGI.
This worked for them before.
Upon going into a version with HADOOP-14104, accessing one of the clusters
would fail with 'key not found'. This is due to there is only 1 mapping from
nameservice -> kms in the UGI credentials. So
{{[DFSClient#getKeyProviderUri|https://github.com/apache/hadoop/blob/branch-3.0.0-beta1/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java#L2991]}}
always find the same KMS provider uri, for both clusters.
Having identical nameservices for multiple clusters is arguably a
mis-configuration (and is how we moved over the issue at the time - luckily one
of the cluster could be changed without too much trouble). But ideally this
should work regardless. I don't have a great idea on how to fix this, but
figured I'd at least share the problem statement.
> Client should always ask namenode for kms provider path.
>
>
> Key: HADOOP-14104
> URL: https://issues.apache.org/jira/browse/HADOOP-14104
> Project: Hadoop Common
> Issue Type: Improvement
> Components: kms
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
> Fix For: 2.9.0, 3.0.0-alpha4, 2.8.2
>
> Attachments: HADOOP-14104-branch-2.8.patch,
> HADOOP-14104-branch-2.patch, HADOOP-14104-trunk-v1.patch,
> HADOOP-14104-trunk-v2.patch, HADOOP-14104-trunk-v3.patch,
> HADOOP-14104-trunk-v4.patch, HADOOP-14104-trunk-v5.patch,
> HADOOP-14104-trunk.patch
>
>
> According to current implementation of kms provider in client conf, there can
> only be one kms.
> In multi-cluster environment, if a client is reading encrypted data from
> multiple clusters it will only get kms token for local cluster.
> Not sure whether the target version is correct or not.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15032) Enable Optimize Hadoop RPC encryption performance for branch-2
[
https://issues.apache.org/jira/browse/HADOOP-15032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16250279#comment-16250279
]
Hadoop QA commented on HADOOP-15032:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 22m
30s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
26s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
28s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
53s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
6s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
20s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
11s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 12m
8s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 12m 8s{color}
| {color:red} root generated 5 new + 1433 unchanged - 5 fixed = 1438 total (was
1438) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 3s{color} | {color:orange} root: The patch generated 25 new + 886 unchanged
- 7 fixed = 911 total (was 893) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 20m 31s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
33s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 77m 29s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}194m 39s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Unreaped Processes | hadoop-common:1 |
| | hadoop-hdfs:20 |
| Failed junit tests | hadoop.ipc.TestRPCCallBenchmark |
| | hadoop.hdfs.server.datanode.TestBatchIbr |
| Timed out junit tests | org.apache.hadoop.log.TestLogLevel |
| | org.apache.hadoop.hdfs.TestWriteRead |
| | org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
|
| | org.apache.hadoop.hdfs.server.datanode.TestDataNodeMetrics |
| | org.apache.hadoop.hdfs.TestPread |
| | org.apache.hadoop.hdfs.TestFileAppend4 |
| | org.apache.hadoop.hdfs.TestRollingUpgradeDowngrade |
| | org.apache.hadoop.hdfs.TestDecommission |
| | org.apache.hadoop.hdfs.TestApplyingStoragePolicy |
| |
[jira] [Commented] (HADOOP-15035) S3Guard to perform retry and translation of exceptions
[ https://issues.apache.org/jira/browse/HADOOP-15035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16250205#comment-16250205 ] Steve Loughran commented on HADOOP-15035: - There is some minor retry about some of the calls. and the back update process does the incremental updates needed to work with very large files *and process throttled connections*. Nothing obvious in terms of other network problems though. > S3Guard to perform retry and translation of exceptions > -- > > Key: HADOOP-15035 > URL: https://issues.apache.org/jira/browse/HADOOP-15035 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0 >Reporter: Steve Loughran > > S3Guard doesn't translate DDB exceptions, nor does it do much in the way of > retry of network problems. Some queries do a bit. HADOOP-13786 marks up the > code with its new @ attribute; this JIRA is "decide what to do and fix" > Be good to have some stack traces of failures -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15003) Merge S3A committers into trunk: Yetus patch checker
[ https://issues.apache.org/jira/browse/HADOOP-15003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15003: Status: Patch Available (was: Open) > Merge S3A committers into trunk: Yetus patch checker > > > Key: HADOOP-15003 > URL: https://issues.apache.org/jira/browse/HADOOP-15003 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13786-041.patch, HADOOP-13786-042.patch, > HADOOP-13786-043.patch, HADOOP-13786-044.patch, HADOOP-13786-045.patch, > HADOOP-13786-046.patch, HADOOP-13786-047.patch, HADOOP-13786-048.patch > > > This is a Yetus only JIRA created to have Yetus review the > HADOOP-13786/HADOOP-14971 patch as a .patch file, as the review PR > [https://github.com/apache/hadoop/pull/282] is stopping this happening in > HADOOP-14971. > Reviews should go into the PR/other task -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15003) Merge S3A committers into trunk: Yetus patch checker
[ https://issues.apache.org/jira/browse/HADOOP-15003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15003: Attachment: HADOOP-13786-048.patch Yetus didn't run the last patch, so lets try again. Changes here minor; apart from staging committer opt out of uploads, & test tuning, this is about documentation and marking up the DDB code for its retry policy. Looks to me like we need to implement exception translation there as well as retrying ops where appropriate. * staging committers can opt out of aborting all uploads in job cleanup; consistent with the original netflix committer. This allows >1 partitioned committer to write to the same directory tree as long as you are confident that they really are writing to different partitions (race conditions in parallel job commits, see) * Add SocketTimeoutException as retryable on idempotent calls * add Retries policy to DynamoDBMetadataStore. This highlights there's little exception translation taking place, and not enough retries. Not added any more retry/translation; that's a S3Guard issue. Just documented what's happening to see what is needed and filed HADOOP-15035. * DynamoDB uses same idempotency policy on delete as S3 (i.e. use the same constant in retry(delete)) * update committer documentation on configuration, more troubleshooting, whole section in index.md on retries and recovery * Tests: ITestS3AHugeMagicCommits checks pending upload count before (==1) and after (==0) commit. Didn't find any problems, as expected. > Merge S3A committers into trunk: Yetus patch checker > > > Key: HADOOP-15003 > URL: https://issues.apache.org/jira/browse/HADOOP-15003 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13786-041.patch, HADOOP-13786-042.patch, > HADOOP-13786-043.patch, HADOOP-13786-044.patch, HADOOP-13786-045.patch, > HADOOP-13786-046.patch, HADOOP-13786-047.patch, HADOOP-13786-048.patch > > > This is a Yetus only JIRA created to have Yetus review the > HADOOP-13786/HADOOP-14971 patch as a .patch file, as the review PR > [https://github.com/apache/hadoop/pull/282] is stopping this happening in > HADOOP-14971. > Reviews should go into the PR/other task -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15035) S3Guard to perform retry and translation of exceptions
Steve Loughran created HADOOP-15035: --- Summary: S3Guard to perform retry and translation of exceptions Key: HADOOP-15035 URL: https://issues.apache.org/jira/browse/HADOOP-15035 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: 3.0.0 Reporter: Steve Loughran S3Guard doesn't translate DDB exceptions, nor does it do much in the way of retry of network problems. Some queries do a bit. HADOOP-13786 marks up the code with its new @ attribute; this JIRA is "decide what to do and fix" Be good to have some stack traces of failures -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15034) S3Guard instrumentation to include cost of DynamoDB ops as metric
Steve Loughran created HADOOP-15034:
---
Summary: S3Guard instrumentation to include cost of DynamoDB ops
as metric
Key: HADOOP-15034
URL: https://issues.apache.org/jira/browse/HADOOP-15034
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 3.0.0
Reporter: Steve Loughran
Priority: Minor
DynamoDB ops can return the cost of the operation in {{ConsumedCapacity}}
methods.
by switching to the API calls which include this in the results are used in
{{DynamoDBMetadataStore}}, then we could provide live/aggregate stats on IO
capacity used. This could aid in live monitoring S3Guard load, and help assess
the cost of queries
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14854) DistCp should not issue file status calls for files in the filter list
[
https://issues.apache.org/jira/browse/HADOOP-14854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16250038#comment-16250038
]
Hadoop QA commented on HADOOP-14854:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 9m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 22s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 11s{color} | {color:orange} hadoop-tools/hadoop-distcp: The patch generated
1 new + 15 unchanged - 0 fixed = 16 total (was 15) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 57s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m
58s{color} | {color:green} hadoop-distcp in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 65m 2s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-14854 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12897315/HADOOP-14854.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 0433356b1c65 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 0d6bab9 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13669/artifact/out/diff-checkstyle-hadoop-tools_hadoop-distcp.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13669/testReport/ |
| Max. process+thread count | 301 (vs. ulimit of 5000) |
| modules | C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp |
| Console output |
[jira] [Commented] (HADOOP-15031) Fix javadoc issues in Hadoop Common
[
https://issues.apache.org/jira/browse/HADOOP-15031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16249957#comment-16249957
]
Hudson commented on HADOOP-15031:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13226 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13226/])
HADOOP-15031. Fix javadoc issues in Hadoop Common. Contributed by Mukul
(aajisaka: rev 975a57a6886e81e412bea35bf597beccc807a66f)
* (edit)
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Options.java
* (edit)
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
> Fix javadoc issues in Hadoop Common
> ---
>
> Key: HADOOP-15031
> URL: https://issues.apache.org/jira/browse/HADOOP-15031
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
>Priority: Minor
> Fix For: 3.1.0
>
> Attachments: HADOOP-15031.001.patch, HADOOP-15031.002.patch,
> HADOOP-15031.003.patch
>
>
> Fix the following javadocs warning in Hadoop Common
> {code}
> [WARNING] Javadoc Warnings
> [WARNING]
> /Users/msingh/code/work/apache/trunk/trunk1/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java:982:
> warning - Tag @link: can't find createPathHandle(FileStatus, HandleOpt[]) in
> org.apache.hadoop.fs.FileSystem
> [WARNING]
> /Users/msingh/code/work/apache/trunk/trunk1/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Options.java:348:
> warning - Tag @link: can't find getPathHandle(FileStatus, HandleOpt[]) in
> org.apache.hadoop.fs.FileSystem
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15008) Metrics sinks may emit too frequently if multiple sink periods are configured
[
https://issues.apache.org/jira/browse/HADOOP-15008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16249958#comment-16249958
]
Hudson commented on HADOOP-15008:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13226 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13226/])
HADOOP-15008. Fixed period unit calculation for Hadoop Metrics V2. (eyang: rev
782681c73e4ae7a02206d4d26635bb1e4984fa24)
* (edit)
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSinkAdapter.java
* (edit)
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java
* (edit)
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestMetricsSystemImpl.java
HADOOP-15008. Fixed period unit calculation for Hadoop Metrics V2. (eyang:
rev 1b68b8ff2c6d4704f748d47fc0b903636f3e98c7)
* (edit)
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSinkAdapter.java
* (edit)
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestMetricsSystemImpl.java
* (edit)
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java
> Metrics sinks may emit too frequently if multiple sink periods are configured
> -
>
> Key: HADOOP-15008
> URL: https://issues.apache.org/jira/browse/HADOOP-15008
> Project: Hadoop Common
> Issue Type: Bug
> Components: metrics
>Affects Versions: 2.2.0, 3.0.0-beta1
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Minor
> Fix For: 3.1.0
>
> Attachments: HADOOP-15008.000.patch
>
>
> If there are multiple metrics sink periods configured, depending on what
> those periods are, some sinks may emit too frequently. For example with the
> following:
> {code:title=hadoop-metrics2.properties}
> namenode.sink.file10.class=org.apache.hadoop.metrics2.sink.FileSink
> namenode.sink.file5.class=org.apache.hadoop.metrics2.sink.FileSink
> namenode.sink.file10.filename=namenode-metrics_per10.out
> namenode.sink.file5.filename=namenode-metrics_per5.out
> namenode.sink.file10.period=10
> namenode.sink.file5.period=5
> {code}
> I get the following:
> {code}
> ± for f in namenode-metrics_per*.out; do echo "$f" && grep
> "metricssystem.MetricsSystem" $f | awk '{last=curr; curr=$1} END { print
> curr-last }'; done
> namenode-metrics_per10.out
> 5000
> namenode-metrics_per5.out
> 5000
> {code}
> i.e., for both metrics files, each record is 5000 ms apart, even though one
> of the sinks has been configured to emit at 10s intervals
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15008) Metrics sinks may emit too frequently if multiple sink periods are configured
[
https://issues.apache.org/jira/browse/HADOOP-15008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erik Krogen updated HADOOP-15008:
-
Release Note: Previously if multiple metrics sinks were configured with
different periods, they may emit more frequently than configured, at a period
as low as the GCD of the configured periods. This change makes all metrics
sinks emit at their configured period.
> Metrics sinks may emit too frequently if multiple sink periods are configured
> -
>
> Key: HADOOP-15008
> URL: https://issues.apache.org/jira/browse/HADOOP-15008
> Project: Hadoop Common
> Issue Type: Bug
> Components: metrics
>Affects Versions: 2.2.0, 3.0.0-beta1
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Minor
> Fix For: 3.1.0
>
> Attachments: HADOOP-15008.000.patch
>
>
> If there are multiple metrics sink periods configured, depending on what
> those periods are, some sinks may emit too frequently. For example with the
> following:
> {code:title=hadoop-metrics2.properties}
> namenode.sink.file10.class=org.apache.hadoop.metrics2.sink.FileSink
> namenode.sink.file5.class=org.apache.hadoop.metrics2.sink.FileSink
> namenode.sink.file10.filename=namenode-metrics_per10.out
> namenode.sink.file5.filename=namenode-metrics_per5.out
> namenode.sink.file10.period=10
> namenode.sink.file5.period=5
> {code}
> I get the following:
> {code}
> ± for f in namenode-metrics_per*.out; do echo "$f" && grep
> "metricssystem.MetricsSystem" $f | awk '{last=curr; curr=$1} END { print
> curr-last }'; done
> namenode-metrics_per10.out
> 5000
> namenode-metrics_per5.out
> 5000
> {code}
> i.e., for both metrics files, each record is 5000 ms apart, even though one
> of the sinks has been configured to emit at 10s intervals
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15008) Metrics sinks may emit too frequently if multiple sink periods are configured
[
https://issues.apache.org/jira/browse/HADOOP-15008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16249922#comment-16249922
]
Erik Krogen commented on HADOOP-15008:
--
Thanks [~eyang]!
> Metrics sinks may emit too frequently if multiple sink periods are configured
> -
>
> Key: HADOOP-15008
> URL: https://issues.apache.org/jira/browse/HADOOP-15008
> Project: Hadoop Common
> Issue Type: Bug
> Components: metrics
>Affects Versions: 2.2.0, 3.0.0-beta1
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Minor
> Fix For: 3.1.0
>
> Attachments: HADOOP-15008.000.patch
>
>
> If there are multiple metrics sink periods configured, depending on what
> those periods are, some sinks may emit too frequently. For example with the
> following:
> {code:title=hadoop-metrics2.properties}
> namenode.sink.file10.class=org.apache.hadoop.metrics2.sink.FileSink
> namenode.sink.file5.class=org.apache.hadoop.metrics2.sink.FileSink
> namenode.sink.file10.filename=namenode-metrics_per10.out
> namenode.sink.file5.filename=namenode-metrics_per5.out
> namenode.sink.file10.period=10
> namenode.sink.file5.period=5
> {code}
> I get the following:
> {code}
> ± for f in namenode-metrics_per*.out; do echo "$f" && grep
> "metricssystem.MetricsSystem" $f | awk '{last=curr; curr=$1} END { print
> curr-last }'; done
> namenode-metrics_per10.out
> 5000
> namenode-metrics_per5.out
> 5000
> {code}
> i.e., for both metrics files, each record is 5000 ms apart, even though one
> of the sinks has been configured to emit at 10s intervals
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15008) Metrics sinks may emit too frequently if multiple sink periods are configured
[
https://issues.apache.org/jira/browse/HADOOP-15008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated HADOOP-15008:
---
Affects Version/s: 2.2.0
3.0.0-beta1
Hadoop Flags: Incompatible change,Reviewed
Fix Version/s: 3.1.0
> Metrics sinks may emit too frequently if multiple sink periods are configured
> -
>
> Key: HADOOP-15008
> URL: https://issues.apache.org/jira/browse/HADOOP-15008
> Project: Hadoop Common
> Issue Type: Bug
> Components: metrics
>Affects Versions: 2.2.0, 3.0.0-beta1
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Minor
> Fix For: 3.1.0
>
> Attachments: HADOOP-15008.000.patch
>
>
> If there are multiple metrics sink periods configured, depending on what
> those periods are, some sinks may emit too frequently. For example with the
> following:
> {code:title=hadoop-metrics2.properties}
> namenode.sink.file10.class=org.apache.hadoop.metrics2.sink.FileSink
> namenode.sink.file5.class=org.apache.hadoop.metrics2.sink.FileSink
> namenode.sink.file10.filename=namenode-metrics_per10.out
> namenode.sink.file5.filename=namenode-metrics_per5.out
> namenode.sink.file10.period=10
> namenode.sink.file5.period=5
> {code}
> I get the following:
> {code}
> ± for f in namenode-metrics_per*.out; do echo "$f" && grep
> "metricssystem.MetricsSystem" $f | awk '{last=curr; curr=$1} END { print
> curr-last }'; done
> namenode-metrics_per10.out
> 5000
> namenode-metrics_per5.out
> 5000
> {code}
> i.e., for both metrics files, each record is 5000 ms apart, even though one
> of the sinks has been configured to emit at 10s intervals
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15008) Metrics sinks may emit too frequently if multiple sink periods are configured
[
https://issues.apache.org/jira/browse/HADOOP-15008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated HADOOP-15008:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this, Thank you Erik.
> Metrics sinks may emit too frequently if multiple sink periods are configured
> -
>
> Key: HADOOP-15008
> URL: https://issues.apache.org/jira/browse/HADOOP-15008
> Project: Hadoop Common
> Issue Type: Bug
> Components: metrics
>Affects Versions: 2.2.0, 3.0.0-beta1
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Minor
> Fix For: 3.1.0
>
> Attachments: HADOOP-15008.000.patch
>
>
> If there are multiple metrics sink periods configured, depending on what
> those periods are, some sinks may emit too frequently. For example with the
> following:
> {code:title=hadoop-metrics2.properties}
> namenode.sink.file10.class=org.apache.hadoop.metrics2.sink.FileSink
> namenode.sink.file5.class=org.apache.hadoop.metrics2.sink.FileSink
> namenode.sink.file10.filename=namenode-metrics_per10.out
> namenode.sink.file5.filename=namenode-metrics_per5.out
> namenode.sink.file10.period=10
> namenode.sink.file5.period=5
> {code}
> I get the following:
> {code}
> ± for f in namenode-metrics_per*.out; do echo "$f" && grep
> "metricssystem.MetricsSystem" $f | awk '{last=curr; curr=$1} END { print
> curr-last }'; done
> namenode-metrics_per10.out
> 5000
> namenode-metrics_per5.out
> 5000
> {code}
> i.e., for both metrics files, each record is 5000 ms apart, even though one
> of the sinks has been configured to emit at 10s intervals
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15033) Use java.util.zip.CRC32C for Java 9 and above
[ https://issues.apache.org/jira/browse/HADOOP-15033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16249865#comment-16249865 ] Ravi Prakash commented on HADOOP-15033: --- Hi Dmitry! Thank you for running the tests. I'm sorry I'm not too familiar with this area. Do you know if the results for the checksum were the same? I remember there were some inconsistencies between Hadoop's implementation of CRC32C and the system libraries. If that were to happen, on an upgrade perfectly good blocks may be marked as corrupt. Have you taken a look at https://issues.apache.org/jira/browse/HDFS-3528 ? > Use java.util.zip.CRC32C for Java 9 and above > - > > Key: HADOOP-15033 > URL: https://issues.apache.org/jira/browse/HADOOP-15033 > Project: Hadoop Common > Issue Type: Improvement > Components: performance, util >Affects Versions: 3.0.0 >Reporter: Dmitry Chuyko > > java.util.zip.CRC32C implementation is available since Java 9. > https://docs.oracle.com/javase/9/docs/api/java/util/zip/CRC32C.html > Platform specific assembler intrinsics make it more effective than any pure > Java implementation. > Hadoop is compiled against Java 8 but class constructor may be accessible > with method handle on 9 to instances implementing Checksum in runtime. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15008) Metrics sinks may emit too frequently if multiple sink periods are configured
[
https://issues.apache.org/jira/browse/HADOOP-15008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16249863#comment-16249863
]
Eric Yang commented on HADOOP-15008:
+1 Verified the test failure is not related to this patch. The value output
from different different intervals seem to have correct values.
> Metrics sinks may emit too frequently if multiple sink periods are configured
> -
>
> Key: HADOOP-15008
> URL: https://issues.apache.org/jira/browse/HADOOP-15008
> Project: Hadoop Common
> Issue Type: Bug
> Components: metrics
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Minor
> Attachments: HADOOP-15008.000.patch
>
>
> If there are multiple metrics sink periods configured, depending on what
> those periods are, some sinks may emit too frequently. For example with the
> following:
> {code:title=hadoop-metrics2.properties}
> namenode.sink.file10.class=org.apache.hadoop.metrics2.sink.FileSink
> namenode.sink.file5.class=org.apache.hadoop.metrics2.sink.FileSink
> namenode.sink.file10.filename=namenode-metrics_per10.out
> namenode.sink.file5.filename=namenode-metrics_per5.out
> namenode.sink.file10.period=10
> namenode.sink.file5.period=5
> {code}
> I get the following:
> {code}
> ± for f in namenode-metrics_per*.out; do echo "$f" && grep
> "metricssystem.MetricsSystem" $f | awk '{last=curr; curr=$1} END { print
> curr-last }'; done
> namenode-metrics_per10.out
> 5000
> namenode-metrics_per5.out
> 5000
> {code}
> i.e., for both metrics files, each record is 5000 ms apart, even though one
> of the sinks has been configured to emit at 10s intervals
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15032) Enable Optimize Hadoop RPC encryption performance for branch-2
[ https://issues.apache.org/jira/browse/HADOOP-15032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dapeng Sun updated HADOOP-15032: Status: Patch Available (was: Open) > Enable Optimize Hadoop RPC encryption performance for branch-2 > -- > > Key: HADOOP-15032 > URL: https://issues.apache.org/jira/browse/HADOOP-15032 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.8.1 >Reporter: Dapeng Sun >Assignee: Dapeng Sun > Attachments: HADOOP-15032-branch-2-001.patch > > > Enable Optimize Hadoop RPC encryption performance for branch-2 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15032) Enable Optimize Hadoop RPC encryption performance for branch-2
[ https://issues.apache.org/jira/browse/HADOOP-15032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dapeng Sun updated HADOOP-15032: Attachment: HADOOP-15032-branch-2-001.patch > Enable Optimize Hadoop RPC encryption performance for branch-2 > -- > > Key: HADOOP-15032 > URL: https://issues.apache.org/jira/browse/HADOOP-15032 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.8.1 >Reporter: Dapeng Sun >Assignee: Dapeng Sun > Attachments: HADOOP-15032-branch-2-001.patch > > > Enable Optimize Hadoop RPC encryption performance for branch-2 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14901) ReuseObjectMapper in Hadoop Common
[ https://issues.apache.org/jira/browse/HADOOP-14901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HADOOP-14901: -- Target Version/s: 2.9.0, 3.0.0, 3.1.0 (was: 2.8.2, 3.0.0, 3.1.0) > ReuseObjectMapper in Hadoop Common > -- > > Key: HADOOP-14901 > URL: https://issues.apache.org/jira/browse/HADOOP-14901 > Project: Hadoop Common > Issue Type: Bug >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Minor > Fix For: 2.9.0, 3.0.0 > > Attachments: HADOOP-14901-branch-2.001.patch, > HADOOP-14901-branch-2.002.patch, HADOOP-14901.001.patch > > > It is recommended to reuse ObjectMapper, if possible, for better performance. > We can aslo use ObjectReader or ObjectWriter to replace the ObjectMapper in > some places: they are straightforward and thread safe. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14901) ReuseObjectMapper in Hadoop Common
[ https://issues.apache.org/jira/browse/HADOOP-14901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16249708#comment-16249708 ] Anu Engineer commented on HADOOP-14901: --- Thank you for cherry-picking this into 3.0. I had skipped 3.0 due to this getting fixed very close to the last release. > ReuseObjectMapper in Hadoop Common > -- > > Key: HADOOP-14901 > URL: https://issues.apache.org/jira/browse/HADOOP-14901 > Project: Hadoop Common > Issue Type: Bug >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Minor > Fix For: 2.9.0, 3.0.0 > > Attachments: HADOOP-14901-branch-2.001.patch, > HADOOP-14901-branch-2.002.patch, HADOOP-14901.001.patch > > > It is recommended to reuse ObjectMapper, if possible, for better performance. > We can aslo use ObjectReader or ObjectWriter to replace the ObjectMapper in > some places: they are straightforward and thread safe. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14901) ReuseObjectMapper in Hadoop Common
[ https://issues.apache.org/jira/browse/HADOOP-14901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HADOOP-14901: -- Target Version/s: 2.8.2, 3.0.0, 3.1.0 (was: 2.8.2, 3.1.0) > ReuseObjectMapper in Hadoop Common > -- > > Key: HADOOP-14901 > URL: https://issues.apache.org/jira/browse/HADOOP-14901 > Project: Hadoop Common > Issue Type: Bug >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Minor > Fix For: 2.9.0, 3.0.0 > > Attachments: HADOOP-14901-branch-2.001.patch, > HADOOP-14901-branch-2.002.patch, HADOOP-14901.001.patch > > > It is recommended to reuse ObjectMapper, if possible, for better performance. > We can aslo use ObjectReader or ObjectWriter to replace the ObjectMapper in > some places: they are straightforward and thread safe. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15031) Fix javadoc issues in Hadoop Common
[
https://issues.apache.org/jira/browse/HADOOP-15031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16249649#comment-16249649
]
Mukul Kumar Singh commented on HADOOP-15031:
Thanks for the review and commit [~ajisakaa]
> Fix javadoc issues in Hadoop Common
> ---
>
> Key: HADOOP-15031
> URL: https://issues.apache.org/jira/browse/HADOOP-15031
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
>Priority: Minor
> Fix For: 3.1.0
>
> Attachments: HADOOP-15031.001.patch, HADOOP-15031.002.patch,
> HADOOP-15031.003.patch
>
>
> Fix the following javadocs warning in Hadoop Common
> {code}
> [WARNING] Javadoc Warnings
> [WARNING]
> /Users/msingh/code/work/apache/trunk/trunk1/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java:982:
> warning - Tag @link: can't find createPathHandle(FileStatus, HandleOpt[]) in
> org.apache.hadoop.fs.FileSystem
> [WARNING]
> /Users/msingh/code/work/apache/trunk/trunk1/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Options.java:348:
> warning - Tag @link: can't find getPathHandle(FileStatus, HandleOpt[]) in
> org.apache.hadoop.fs.FileSystem
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15031) Fix javadoc issues in Hadoop Common
[
https://issues.apache.org/jira/browse/HADOOP-15031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated HADOOP-15031:
---
Priority: Minor (was: Major)
Summary: Fix javadoc issues in Hadoop Common (was: Fix javadocs issues in
Hadoop Common)
> Fix javadoc issues in Hadoop Common
> ---
>
> Key: HADOOP-15031
> URL: https://issues.apache.org/jira/browse/HADOOP-15031
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
>Priority: Minor
> Fix For: 3.1.0
>
> Attachments: HADOOP-15031.001.patch, HADOOP-15031.002.patch,
> HADOOP-15031.003.patch
>
>
> Fix the following javadocs warning in Hadoop Common
> {code}
> [WARNING] Javadoc Warnings
> [WARNING]
> /Users/msingh/code/work/apache/trunk/trunk1/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java:982:
> warning - Tag @link: can't find createPathHandle(FileStatus, HandleOpt[]) in
> org.apache.hadoop.fs.FileSystem
> [WARNING]
> /Users/msingh/code/work/apache/trunk/trunk1/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Options.java:348:
> warning - Tag @link: can't find getPathHandle(FileStatus, HandleOpt[]) in
> org.apache.hadoop.fs.FileSystem
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15031) Fix javadocs issues in Hadoop Common
[
https://issues.apache.org/jira/browse/HADOOP-15031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated HADOOP-15031:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.1.0
Status: Resolved (was: Patch Available)
Committed this to trunk. Thanks [~msingh] for the contribution!
> Fix javadocs issues in Hadoop Common
>
>
> Key: HADOOP-15031
> URL: https://issues.apache.org/jira/browse/HADOOP-15031
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Fix For: 3.1.0
>
> Attachments: HADOOP-15031.001.patch, HADOOP-15031.002.patch,
> HADOOP-15031.003.patch
>
>
> Fix the following javadocs warning in Hadoop Common
> {code}
> [WARNING] Javadoc Warnings
> [WARNING]
> /Users/msingh/code/work/apache/trunk/trunk1/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java:982:
> warning - Tag @link: can't find createPathHandle(FileStatus, HandleOpt[]) in
> org.apache.hadoop.fs.FileSystem
> [WARNING]
> /Users/msingh/code/work/apache/trunk/trunk1/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Options.java:348:
> warning - Tag @link: can't find getPathHandle(FileStatus, HandleOpt[]) in
> org.apache.hadoop.fs.FileSystem
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15031) Fix javadocs issues in Hadoop Common
[
https://issues.apache.org/jira/browse/HADOOP-15031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16249633#comment-16249633
]
Akira Ajisaka commented on HADOOP-15031:
001 patch should fix the warning, but it did not. This seems JavaDoc bug.
Anyway, +1 for 003 patch. Checking this in.
> Fix javadocs issues in Hadoop Common
>
>
> Key: HADOOP-15031
> URL: https://issues.apache.org/jira/browse/HADOOP-15031
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Attachments: HADOOP-15031.001.patch, HADOOP-15031.002.patch,
> HADOOP-15031.003.patch
>
>
> Fix the following javadocs warning in Hadoop Common
> {code}
> [WARNING] Javadoc Warnings
> [WARNING]
> /Users/msingh/code/work/apache/trunk/trunk1/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java:982:
> warning - Tag @link: can't find createPathHandle(FileStatus, HandleOpt[]) in
> org.apache.hadoop.fs.FileSystem
> [WARNING]
> /Users/msingh/code/work/apache/trunk/trunk1/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Options.java:348:
> warning - Tag @link: can't find getPathHandle(FileStatus, HandleOpt[]) in
> org.apache.hadoop.fs.FileSystem
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15033) Use java.util.zip.CRC32C for Java 9 and above
[ https://issues.apache.org/jira/browse/HADOOP-15033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15033: Affects Version/s: 3.0.0 > Use java.util.zip.CRC32C for Java 9 and above > - > > Key: HADOOP-15033 > URL: https://issues.apache.org/jira/browse/HADOOP-15033 > Project: Hadoop Common > Issue Type: Improvement > Components: performance, util >Affects Versions: 3.0.0 >Reporter: Dmitry Chuyko > > java.util.zip.CRC32C implementation is available since Java 9. > https://docs.oracle.com/javase/9/docs/api/java/util/zip/CRC32C.html > Platform specific assembler intrinsics make it more effective than any pure > Java implementation. > Hadoop is compiled against Java 8 but class constructor may be accessible > with method handle on 9 to instances implementing Checksum in runtime. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15033) Use java.util.zip.CRC32C for Java 9 and above
[ https://issues.apache.org/jira/browse/HADOOP-15033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15033: Status: Patch Available (was: Open) > Use java.util.zip.CRC32C for Java 9 and above > - > > Key: HADOOP-15033 > URL: https://issues.apache.org/jira/browse/HADOOP-15033 > Project: Hadoop Common > Issue Type: Improvement > Components: performance, util >Reporter: Dmitry Chuyko > > java.util.zip.CRC32C implementation is available since Java 9. > https://docs.oracle.com/javase/9/docs/api/java/util/zip/CRC32C.html > Platform specific assembler intrinsics make it more effective than any pure > Java implementation. > Hadoop is compiled against Java 8 but class constructor may be accessible > with method handle on 9 to instances implementing Checksum in runtime. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14854) DistCp should not issue file status calls for files in the filter list
[ https://issues.apache.org/jira/browse/HADOOP-14854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh updated HADOOP-14854: --- Attachment: HADOOP-14854.001.patch > DistCp should not issue file status calls for files in the filter list > -- > > Key: HADOOP-14854 > URL: https://issues.apache.org/jira/browse/HADOOP-14854 > Project: Hadoop Common > Issue Type: Improvement > Components: tools/distcp >Affects Versions: 2.8.1 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Minor > Attachments: HADOOP-14854.001.patch > > > DistCp currently excludes the files in the filter list only when the files > are added to the copy list. > However distcp can be optimized by not issuing file status/get attr calls for > the files in the filter. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14854) DistCp should not issue file status calls for files in the filter list
[ https://issues.apache.org/jira/browse/HADOOP-14854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh updated HADOOP-14854: --- Status: Patch Available (was: Open) > DistCp should not issue file status calls for files in the filter list > -- > > Key: HADOOP-14854 > URL: https://issues.apache.org/jira/browse/HADOOP-14854 > Project: Hadoop Common > Issue Type: Improvement > Components: tools/distcp >Affects Versions: 2.8.1 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Minor > Attachments: HADOOP-14854.001.patch > > > DistCp currently excludes the files in the filter list only when the files > are added to the copy list. > However distcp can be optimized by not issuing file status/get attr calls for > the files in the filter. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15027) Improvements for Hadoop read from AliyunOSS
[
https://issues.apache.org/jira/browse/HADOOP-15027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16249523#comment-16249523
]
Steve Loughran commented on HADOOP-15027:
-
[~wujinhu]:
have a look at [Dancing with
Elephants|https://www.slideshare.net/steve_l/dancing-elephants-working-with-object-storage-in-apache-spark-and-hive],
which includes some of the summary & trace data from our benchmarking TCP-DS
against S3A. Key perf killers: cost of TCP abort; performance of listing and
getFileStatus() calls. the latter is always done during the sequential
partitioning process, so slows the entire job down.
Common code sequences
often code does open + seek() immediately, even when doing forward reads
(example: partitioned read of a sequential file). Lazy seek code usually delays
the GET request until the first read after a seek.
Sequential file formats, including output of Mappers
# open(file)
# seek(offset)
# readFully(bytes[], len, offset)
While the columnar stores read footers then bounce around the file, using the
PositionedReadable API. (whose default implementation of seek/read/seek) is a
killer unless you do lazy seek
# open(file)
# PositionedReadable.readFully(EOF-offset, bytes[], len, offset)
# PositionedReadable.readFully(offset determined by footer info, bytes[], len,
offset)
# PositionedReadable.readFully(offset + 10s or 100s of KB)
# repeated until done
so: backwards as well as forwards, big leaps through the code. (This is isn't a
real trace BTW; we should really collect some).
[~uncleGen] wrote
bq. 2. move thread pool from InputStream to FileSystem: can this lead to
blocking between multiple input stream? I mean if threads are working on slow
stuff for one abnormal file , other input streams should be waiting for a long
time.
3. expensive cost to use thread pool.
Depends on the #of threads a specific stream can have allocated.
Look at {{org.apache.hadoop.fs.s3a.SemaphoredDelegatingExecutor}}; It's used in
S3ABlockOutputStream to allow >1 thread per stream to upload data, using a fail
Semaphore so allocate threads from the pool fairly across all streams. Then,
when the pool is used up, it blocks the caller, so any thread generating too
much data is the one which gets blocked.
This
* eliminates thread creation overhead on stream creation. (expensive on mempry;
slow).
* offers streams ability to use >1 thread for IO.
* shared pool fairly across threads.
* blocks callers under heavy load (e.g. stops pool expanding until OOM).
If you think it is good we could move that class to hadoop-common and share.
> Improvements for Hadoop read from AliyunOSS
> ---
>
> Key: HADOOP-15027
> URL: https://issues.apache.org/jira/browse/HADOOP-15027
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/oss
>Affects Versions: 3.0.0
>Reporter: wujinhu
>Assignee: wujinhu
> Attachments: HADOOP-15027.001.patch, HADOOP-15027.002.patch,
> HADOOP-15027.003.patch
>
>
> Currently, read performance is poor when Hadoop reads from AliyunOSS. It
> needs about 1min to read 1GB from OSS.
> Class AliyunOSSInputStream uses single thread to read data from AliyunOSS,
> so we can refactor this by using multi-thread pre read to improve this.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15028) Got errors while running org.apache.hadoop.io.TestSequenceFileAppend
[ https://issues.apache.org/jira/browse/HADOOP-15028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16249506#comment-16249506 ] Steve Loughran commented on HADOOP-15028: - think you marked it as a duplicate of the wrong JIRA there. Which one does it duplicate? > Got errors while running org.apache.hadoop.io.TestSequenceFileAppend > > > Key: HADOOP-15028 > URL: https://issues.apache.org/jira/browse/HADOOP-15028 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.6.4 > Environment: Linux 2.6.32-642.el6.x86_64 >Reporter: bd17kaka > > I ran the test case org.apache.hadoop.io.TestSequenceFileAppend in > branch-2.6.4, I got the following errors: > Running org.apache.hadoop.io.TestSequenceFileAppend > Tests run: 4, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 0.801 sec <<< > FAILURE! - in org.apache.hadoop.io.TestSequenceFileAppend > testAppendBlockCompression(org.apache.hadoop.io.TestSequenceFileAppend) Time > elapsed: 0.117 sec <<< ERROR! > java.io.IOException: File is corrupt! > at > org.apache.hadoop.io.SequenceFile$Reader.readBlock(SequenceFile.java:2179) > at > org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2583) > at > org.apache.hadoop.io.TestSequenceFileAppend.verifyAll4Values(TestSequenceFileAppend.java:309) > at > org.apache.hadoop.io.TestSequenceFileAppend.testAppendBlockCompression(TestSequenceFileAppend.java:205) > testAppendSort(org.apache.hadoop.io.TestSequenceFileAppend) Time elapsed: > 0.013 sec <<< ERROR! > java.io.IOException: File is corrupt! > at > org.apache.hadoop.io.SequenceFile$Reader.readBlock(SequenceFile.java:2179) > at > org.apache.hadoop.io.SequenceFile$Reader.nextRaw(SequenceFile.java:2488) > at > org.apache.hadoop.io.SequenceFile$Sorter$SortPass.run(SequenceFile.java:2923) > at > org.apache.hadoop.io.SequenceFile$Sorter.sortPass(SequenceFile.java:2861) > at > org.apache.hadoop.io.SequenceFile$Sorter.sort(SequenceFile.java:2809) > at > org.apache.hadoop.io.SequenceFile$Sorter.sort(SequenceFile.java:2850) > at > org.apache.hadoop.io.TestSequenceFileAppend.testAppendSort(TestSequenceFileAppend.java:286) > But everything is OK in branch-2.6.5.. > The maven command is 'mvn test -Pnative -Dtest=TestSequenceFileAppend' -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14964) AliyunOSS: backport Aliyun OSS module to branch-2 and 2.7+ branches
[ https://issues.apache.org/jira/browse/HADOOP-14964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16249503#comment-16249503 ] Steve Loughran commented on HADOOP-14964: - We can worry about 2.9, but I don't expect any patches going in to 2.7-2.8, which are essentially maintenance only right now. Indeed, there's been a discussion on common-dev about whether to add any new features into branch-2 at all, or have a split of : features in 3.x+ (java 8 language, testing), fixes into branch-2. I don't expect to see my new code going back, and embracing java 8 is a great thing to do. Not least because nobody has java 7 dev environments any more. FWIW, I think the object stores are less risky here, but packaging changes are. > AliyunOSS: backport Aliyun OSS module to branch-2 and 2.7+ branches > --- > > Key: HADOOP-14964 > URL: https://issues.apache.org/jira/browse/HADOOP-14964 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss >Reporter: Genmao Yu >Assignee: SammiChen > Attachments: HADOOP-14964-branch-2.000.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15003) Merge S3A committers into trunk: Yetus patch checker
[ https://issues.apache.org/jira/browse/HADOOP-15003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15003: Status: Open (was: Patch Available) > Merge S3A committers into trunk: Yetus patch checker > > > Key: HADOOP-15003 > URL: https://issues.apache.org/jira/browse/HADOOP-15003 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13786-041.patch, HADOOP-13786-042.patch, > HADOOP-13786-043.patch, HADOOP-13786-044.patch, HADOOP-13786-045.patch, > HADOOP-13786-046.patch, HADOOP-13786-047.patch > > > This is a Yetus only JIRA created to have Yetus review the > HADOOP-13786/HADOOP-14971 patch as a .patch file, as the review PR > [https://github.com/apache/hadoop/pull/282] is stopping this happening in > HADOOP-14971. > Reviews should go into the PR/other task -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15033) Use java.util.zip.CRC32C for Java 9 and above
[ https://issues.apache.org/jira/browse/HADOOP-15033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitry Chuyko updated HADOOP-15033: --- Description: java.util.zip.CRC32C implementation is available since Java 9. https://docs.oracle.com/javase/9/docs/api/java/util/zip/CRC32C.html Platform specific assembler intrinsics make it more effective than any pure Java implementation. Hadoop is compiled against Java 8 but class constructor may be accessible with method handle on 9 to instances implementing Checksum in runtime. was: java.util.zip.CRC32C implementation is available since Java 9. https://docs.oracle.com/javase/9/docs/api/java/util/zip/CRC32C.html Platform specific assembler intrinsics make it more aeefective than any pure Java implementation. Hadoop is compiled against Java 8 but class constructor may be accessible with method handle on 9 to instances implementing Checksum in runtime. > Use java.util.zip.CRC32C for Java 9 and above > - > > Key: HADOOP-15033 > URL: https://issues.apache.org/jira/browse/HADOOP-15033 > Project: Hadoop Common > Issue Type: Improvement > Components: performance, util >Reporter: Dmitry Chuyko > > java.util.zip.CRC32C implementation is available since Java 9. > https://docs.oracle.com/javase/9/docs/api/java/util/zip/CRC32C.html > Platform specific assembler intrinsics make it more effective than any pure > Java implementation. > Hadoop is compiled against Java 8 but class constructor may be accessible > with method handle on 9 to instances implementing Checksum in runtime. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-10768) Optimize Hadoop RPC encryption performance
[
https://issues.apache.org/jira/browse/HADOOP-10768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16249378#comment-16249378
]
Hadoop QA commented on HADOOP-10768:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
9s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 21s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
17s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 11m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
35s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 20s{color} | {color:orange} root: The patch generated 18 new + 877 unchanged
- 9 fixed = 895 total (was 886) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 50s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
32s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
20s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 82m 1s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}182m 42s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.namenode.ha.TestBootstrapStandbyWithQJM |
| | hadoop.hdfs.server.federation.metrics.TestFederationMetrics |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.fs.TestUnbuffer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue
[jira] [Commented] (HADOOP-15033) Use java.util.zip.CRC32C for Java 9 and above
[ https://issues.apache.org/jira/browse/HADOOP-15033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16249379#comment-16249379 ] Dmitry Chuyko commented on HADOOP-15033: In the results above mainline JDK 10 was used. > Use java.util.zip.CRC32C for Java 9 and above > - > > Key: HADOOP-15033 > URL: https://issues.apache.org/jira/browse/HADOOP-15033 > Project: Hadoop Common > Issue Type: Improvement > Components: performance, util >Reporter: Dmitry Chuyko > > java.util.zip.CRC32C implementation is available since Java 9. > https://docs.oracle.com/javase/9/docs/api/java/util/zip/CRC32C.html > Platform specific assembler intrinsics make it more aeefective than any pure > Java implementation. > Hadoop is compiled against Java 8 but class constructor may be accessible > with method handle on 9 to instances implementing Checksum in runtime. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15033) Use java.util.zip.CRC32C for Java 9 and above
[ https://issues.apache.org/jira/browse/HADOOP-15033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16249373#comment-16249373 ] ASF GitHub Bot commented on HADOOP-15033: - GitHub user dchuyko opened a pull request: https://github.com/apache/hadoop/pull/291 HADOOP-15033. Use java.util.zip.CRC32C for Java 9 and above I added isJava9OrAbove() flag in Shell which is statically initialized using version string parsing. If it is true, DataChecksum creates zip implementation instead of PureJavaCrc32C. It is created using constructor method handle that's looked up also if it's 9 or above. Signature check should ignore java.lang.invoke to let method handles code pass build. Performance test now outputs extra column for 9 and above with results of zip CRC32C. Results on Cavium ThunderX server (AArch64) show ~4x improvement and are listed below. The picture is about the same on x86. | bpc | #T || Zip || ZipC | % diff || PureJava | % diff || PureJavaC | % diff | |32 | 1 | 157.6 | 160.1 | 1.6% | 110.3 | -31.1% | 98.9 | -10.3% | |32 | 2 | 171.2 | 184.2 | 7.6% | 107.0 | -41.9% | 112.9 | 5.5% | |32 | 4 | 182.1 | 176.0 | -3.3% | 104.0 | -40.9% | 113.3 | 8.9% | |32 | 8 | 179.3 | 173.0 | -3.6% | 102.1 | -41.0% | 111.5 | 9.3% | |32 | 16 | 95.0 | 91.3 | -3.9% | 53.4 | -41.5% | 57.5 | 7.7% | | bpc | #T || Zip || ZipC | % diff || PureJava | % diff || PureJavaC | % diff | |64 | 1 | 326.5 | 324.6 | -0.6% | 137.4 | -57.7% | 144.7 | 5.3% | |64 | 2 | 298.8 | 293.8 | -1.7% | 133.5 | -54.6% | 139.5 | 4.5% | |64 | 4 | 279.3 | 273.3 | -2.2% | 128.4 | -53.0% | 136.5 | 6.3% | |64 | 8 | 269.8 | 267.2 | -0.9% | 125.9 | -52.9% | 133.6 | 6.1% | |64 | 16 | 140.1 | 139.3 | -0.6% | 65.6 | -52.9% | 69.3 | 5.6% | | bpc | #T || Zip || ZipC | % diff || PureJava | % diff || PureJavaC | % diff | | 128 | 1 | 666.6 | 668.6 | 0.3% | 185.0 | -72.3% | 208.7 | 12.8% | | 128 | 2 | 656.3 | 616.9 | -6.0% | 184.5 | -70.1% | 207.7 | 12.6% | | 128 | 4 | 636.7 | 599.7 | -5.8% | 181.8 | -69.7% | 203.6 | 12.0% | | 128 | 8 | 628.3 | 592.3 | -5.7% | 180.6 | -69.5% | 202.2 | 12.0% | | 128 | 16 | 378.1 | 345.9 | -8.5% | 93.1 | -73.1% | 108.4 | 16.4% | | bpc | #T || Zip || ZipC | % diff || PureJava | % diff || PureJavaC | % diff | | 256 | 1 | 906.4 | 776.1 | -14.4% | 207.9 | -73.2% | 239.2 | 15.0% | | 256 | 2 | 882.9 | 833.0 | -5.6% | 206.6 | -75.2% | 238.6 | 15.5% | | 256 | 4 | 837.9 | 801.0 | -4.4% | 204.2 | -74.5% | 235.9 | 15.5% | | 256 | 8 | 806.1 | 776.7 | -3.7% | 201.7 | -74.0% | 225.6 | 11.8% | | 256 | 16 | 493.6 | 456.0 | -7.6% | 107.8 | -76.4% | 121.3 | 12.5% | | bpc | #T || Zip || ZipC | % diff || PureJava | % diff || PureJavaC | % diff | | 512 | 1 |1016.3 | 850.4 | -16.3% | 214.6 | -74.8% | 248.7 | 15.9% | | 512 | 2 | 969.7 | 894.4 | -7.8% | 212.1 | -76.3% | 241.5 | 13.8% | | 512 | 4 | 894.0 | 829.4 | -7.2% | 207.4 | -75.0% | 232.1 | 11.9% | | 512 | 8 | 886.8 | 831.8 | -6.2% | 205.2 | -75.3% | 238.6 | 16.3% | | 512 | 16 | 518.3 | 525.4 | 1.4% | 111.1 | -78.9% | 124.8 | 12.4% | | bpc | #T || Zip || ZipC | % diff || PureJava | % diff || PureJavaC | % diff | | 1024 | 1 |1120.0 |1130.5 | 0.9% | 219.1 | -80.6% | 258.0 | 17.8% | | 1024 | 2 |1054.9 | 983.9 | -6.7% | 217.5 | -77.9% | 254.8 | 17.1% | | 1024 | 4 | 956.8 | 766.0 | -19.9% | 213.0 | -72.2% | 241.1 | 13.2% | | 1024 | 8 | 835.5 | 864.9 | 3.5% | 209.6 | -75.8% | 247.2 | 18.0% | | 1024 | 16 | 545.0 | 540.5 | -0.8% | 110.1 | -79.6% | 131.8 | 19.7% | | bpc | #T || Zip || ZipC | % diff || PureJava | % diff || PureJavaC | % diff | | 2048 | 1 |1189.2 |1190.5 | 0.1% | 223.3 | -81.2% | 263.3 | 17.9% | | 2048 | 2 |1116.4 |1027.5 | -8.0% | 220.6 | -78.5% | 260.0 | 17.9% | | 2048 | 4 |1009.9 | 931.0 | -7.8% | 215.3 | -76.9% | 253.3 | 17.6% | | 2048 | 8 | 950.4 | 881.0 | -7.3% | 213.2 | -75.8% | 251.5 | 17.9% | | 2048 | 16 | 570.7 | 521.7 | -8.6% |
[jira] [Commented] (HADOOP-15033) Use java.util.zip.CRC32C for Java 9 and above
[ https://issues.apache.org/jira/browse/HADOOP-15033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16249316#comment-16249316 ] Dmitry Chuyko commented on HADOOP-15033: Related issue https://issues.apache.org/jira/browse/HADOOP-10674 > Use java.util.zip.CRC32C for Java 9 and above > - > > Key: HADOOP-15033 > URL: https://issues.apache.org/jira/browse/HADOOP-15033 > Project: Hadoop Common > Issue Type: Improvement > Components: performance, util >Reporter: Dmitry Chuyko > > java.util.zip.CRC32C implementation is available since Java 9. > https://docs.oracle.com/javase/9/docs/api/java/util/zip/CRC32C.html > Platform specific assembler intrinsics make it more aeefective than any pure > Java implementation. > Hadoop is compiled against Java 8 but class constructor may be accessible > with method handle on 9 to instances implementing Checksum in runtime. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15033) Use java.util.zip.CRC32C for Java 9 and above
Dmitry Chuyko created HADOOP-15033: -- Summary: Use java.util.zip.CRC32C for Java 9 and above Key: HADOOP-15033 URL: https://issues.apache.org/jira/browse/HADOOP-15033 Project: Hadoop Common Issue Type: Improvement Components: performance, util Reporter: Dmitry Chuyko java.util.zip.CRC32C implementation is available since Java 9. https://docs.oracle.com/javase/9/docs/api/java/util/zip/CRC32C.html Platform specific assembler intrinsics make it more aeefective than any pure Java implementation. Hadoop is compiled against Java 8 but class constructor may be accessible with method handle on 9 to instances implementing Checksum in runtime. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15027) Improvements for Hadoop read from AliyunOSS
[
https://issues.apache.org/jira/browse/HADOOP-15027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16249292#comment-16249292
]
Hadoop QA commented on HADOOP-15027:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 13s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 9s{color} | {color:orange} hadoop-tools/hadoop-aliyun: The patch generated
9 new + 0 unchanged - 0 fixed = 9 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 28s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
18s{color} | {color:green} hadoop-aliyun in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 43m 32s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-15027 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12897300/HADOOP-15027.003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux e8af8ea80bf2 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 3e26077 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13668/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aliyun.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13668/testReport/ |
| Max. process+thread count | 312 (vs. ulimit of 5000) |
| modules | C: hadoop-tools/hadoop-aliyun U: hadoop-tools/hadoop-aliyun |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13668/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
[jira] [Commented] (HADOOP-15027) Improvements for Hadoop read from AliyunOSS
[ https://issues.apache.org/jira/browse/HADOOP-15027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16249250#comment-16249250 ] wujinhu commented on HADOOP-15027: -- Updated to HADOOP-15027.003.patch. > Improvements for Hadoop read from AliyunOSS > --- > > Key: HADOOP-15027 > URL: https://issues.apache.org/jira/browse/HADOOP-15027 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/oss >Affects Versions: 3.0.0 >Reporter: wujinhu >Assignee: wujinhu > Attachments: HADOOP-15027.001.patch, HADOOP-15027.002.patch, > HADOOP-15027.003.patch > > > Currently, read performance is poor when Hadoop reads from AliyunOSS. It > needs about 1min to read 1GB from OSS. > Class AliyunOSSInputStream uses single thread to read data from AliyunOSS, > so we can refactor this by using multi-thread pre read to improve this. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14993) AliyunOSS: Override listFiles and listLocatedStatus
[
https://issues.apache.org/jira/browse/HADOOP-14993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16249244#comment-16249244
]
Hadoop QA commented on HADOOP-14993:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
9s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 8s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 1s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
16s{color} | {color:green} hadoop-aliyun in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 41m 28s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-14993 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12897295/HADOOP-14993.003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 056438f06eef 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 3e26077 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13667/testReport/ |
| Max. process+thread count | 360 (vs. ulimit of 5000) |
| modules | C: hadoop-tools/hadoop-aliyun U: hadoop-tools/hadoop-aliyun |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13667/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> AliyunOSS: Override listFiles and
[jira] [Updated] (HADOOP-15027) Improvements for Hadoop read from AliyunOSS
[ https://issues.apache.org/jira/browse/HADOOP-15027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wujinhu updated HADOOP-15027: - Attachment: HADOOP-15027.003.patch > Improvements for Hadoop read from AliyunOSS > --- > > Key: HADOOP-15027 > URL: https://issues.apache.org/jira/browse/HADOOP-15027 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/oss >Affects Versions: 3.0.0 >Reporter: wujinhu >Assignee: wujinhu > Attachments: HADOOP-15027.001.patch, HADOOP-15027.002.patch, > HADOOP-15027.003.patch > > > Currently, read performance is poor when Hadoop reads from AliyunOSS. It > needs about 1min to read 1GB from OSS. > Class AliyunOSSInputStream uses single thread to read data from AliyunOSS, > so we can refactor this by using multi-thread pre read to improve this. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-15028) Got errors while running org.apache.hadoop.io.TestSequenceFileAppend
[ https://issues.apache.org/jira/browse/HADOOP-15028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] bd17kaka resolved HADOOP-15028. --- Resolution: Duplicate > Got errors while running org.apache.hadoop.io.TestSequenceFileAppend > > > Key: HADOOP-15028 > URL: https://issues.apache.org/jira/browse/HADOOP-15028 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.6.4 > Environment: Linux 2.6.32-642.el6.x86_64 >Reporter: bd17kaka > > I ran the test case org.apache.hadoop.io.TestSequenceFileAppend in > branch-2.6.4, I got the following errors: > Running org.apache.hadoop.io.TestSequenceFileAppend > Tests run: 4, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 0.801 sec <<< > FAILURE! - in org.apache.hadoop.io.TestSequenceFileAppend > testAppendBlockCompression(org.apache.hadoop.io.TestSequenceFileAppend) Time > elapsed: 0.117 sec <<< ERROR! > java.io.IOException: File is corrupt! > at > org.apache.hadoop.io.SequenceFile$Reader.readBlock(SequenceFile.java:2179) > at > org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2583) > at > org.apache.hadoop.io.TestSequenceFileAppend.verifyAll4Values(TestSequenceFileAppend.java:309) > at > org.apache.hadoop.io.TestSequenceFileAppend.testAppendBlockCompression(TestSequenceFileAppend.java:205) > testAppendSort(org.apache.hadoop.io.TestSequenceFileAppend) Time elapsed: > 0.013 sec <<< ERROR! > java.io.IOException: File is corrupt! > at > org.apache.hadoop.io.SequenceFile$Reader.readBlock(SequenceFile.java:2179) > at > org.apache.hadoop.io.SequenceFile$Reader.nextRaw(SequenceFile.java:2488) > at > org.apache.hadoop.io.SequenceFile$Sorter$SortPass.run(SequenceFile.java:2923) > at > org.apache.hadoop.io.SequenceFile$Sorter.sortPass(SequenceFile.java:2861) > at > org.apache.hadoop.io.SequenceFile$Sorter.sort(SequenceFile.java:2809) > at > org.apache.hadoop.io.SequenceFile$Sorter.sort(SequenceFile.java:2850) > at > org.apache.hadoop.io.TestSequenceFileAppend.testAppendSort(TestSequenceFileAppend.java:286) > But everything is OK in branch-2.6.5.. > The maven command is 'mvn test -Pnative -Dtest=TestSequenceFileAppend' -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14993) AliyunOSS: Override listFiles and listLocatedStatus
[
https://issues.apache.org/jira/browse/HADOOP-14993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16249221#comment-16249221
]
Genmao Yu edited comment on HADOOP-14993 at 11/13/17 8:12 AM:
--
[~drankye] I post a new patch, some updates related to your comments:
1. refactor {{listFiles}} and {{listLocatedStatus}}
2. use {{firstListing}} to avoid the {{null}} check of {{batchIterator}}, so I
kept it.
3.& 4. IMHO it is ok for current naming, copy from existing code.
All test passed against "oss-cn-shanghai.aliyuncs.com".
cc [~ste...@apache.org] please take a review, thanks.
was (Author: unclegen):
[~drankye] I post a new patch, some updates related to your comments:
1. refactor {{listFiles}} and {{listLocatedStatus}}
2. use {{firstListing}} to avoid the {{null}} check of {{batchIterator}}, so I
kept it.
3.& 4. IMHO it is ok for current naming, copy from existing code.
also cc [~ste...@apache.org] please take a review, thanks.
> AliyunOSS: Override listFiles and listLocatedStatus
>
>
> Key: HADOOP-14993
> URL: https://issues.apache.org/jira/browse/HADOOP-14993
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/oss
>Affects Versions: 3.0.0-beta1
>Reporter: Genmao Yu
>Assignee: Genmao Yu
> Attachments: HADOOP-14993.001.patch, HADOOP-14993.002.patch,
> HADOOP-14993.003.patch
>
>
> Do a bulk listing off all entries under a path in one single operation, there
> is no need to recursively walk the directory tree.
> Updates:
> - override listFiles and listLocatedStatus by using bulk listing
> - some minor updates in hadoop-aliyun index.md
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14993) AliyunOSS: Override listFiles and listLocatedStatus
[
https://issues.apache.org/jira/browse/HADOOP-14993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16249221#comment-16249221
]
Genmao Yu commented on HADOOP-14993:
[~drankye] I post a new patch, some updates related to your comments:
1. refactor {{listFiles}} and {{listLocatedStatus}}
2. use {{firstListing}} to avoid the {{null}} check of {{batchIterator}}, so I
kept it.
3.& 4. IMHO it is ok for current naming, copy from existing code.
also cc [~ste...@apache.org] please take a review, thanks.
> AliyunOSS: Override listFiles and listLocatedStatus
>
>
> Key: HADOOP-14993
> URL: https://issues.apache.org/jira/browse/HADOOP-14993
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/oss
>Affects Versions: 3.0.0-beta1
>Reporter: Genmao Yu
>Assignee: Genmao Yu
> Attachments: HADOOP-14993.001.patch, HADOOP-14993.002.patch,
> HADOOP-14993.003.patch
>
>
> Do a bulk listing off all entries under a path in one single operation, there
> is no need to recursively walk the directory tree.
> Updates:
> - override listFiles and listLocatedStatus by using bulk listing
> - some minor updates in hadoop-aliyun index.md
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14993) AliyunOSS: Override listFiles and listLocatedStatus
[ https://issues.apache.org/jira/browse/HADOOP-14993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Genmao Yu updated HADOOP-14993: --- Attachment: HADOOP-14993.003.patch > AliyunOSS: Override listFiles and listLocatedStatus > > > Key: HADOOP-14993 > URL: https://issues.apache.org/jira/browse/HADOOP-14993 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/oss >Affects Versions: 3.0.0-beta1 >Reporter: Genmao Yu >Assignee: Genmao Yu > Attachments: HADOOP-14993.001.patch, HADOOP-14993.002.patch, > HADOOP-14993.003.patch > > > Do a bulk listing off all entries under a path in one single operation, there > is no need to recursively walk the directory tree. > Updates: > - override listFiles and listLocatedStatus by using bulk listing > - some minor updates in hadoop-aliyun index.md -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
