[jira] [Updated] (HADOOP-15032) Enable Optimize Hadoop RPC encryption performance for branch-2
[ https://issues.apache.org/jira/browse/HADOOP-15032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dapeng Sun updated HADOOP-15032: Attachment: HADOOP-15032-branch-2-003.patch > Enable Optimize Hadoop RPC encryption performance for branch-2 > -- > > Key: HADOOP-15032 > URL: https://issues.apache.org/jira/browse/HADOOP-15032 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.8.1 >Reporter: Dapeng Sun >Assignee: Dapeng Sun > Attachments: HADOOP-15032-branch-2-001.patch, > HADOOP-15032-branch-2-002.patch, HADOOP-15032-branch-2-003.patch > > > Enable Optimize Hadoop RPC encryption performance for branch-2 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-10768) Optimize Hadoop RPC encryption performance
[
https://issues.apache.org/jira/browse/HADOOP-10768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dapeng Sun updated HADOOP-10768:
Attachment: HADOOP-10768.007.patch
> Optimize Hadoop RPC encryption performance
> --
>
> Key: HADOOP-10768
> URL: https://issues.apache.org/jira/browse/HADOOP-10768
> Project: Hadoop Common
> Issue Type: Improvement
> Components: performance, security
>Affects Versions: 3.0.0-alpha1
>Reporter: Yi Liu
>Assignee: Dapeng Sun
> Attachments: HADOOP-10768.001.patch, HADOOP-10768.002.patch,
> HADOOP-10768.003.patch, HADOOP-10768.004.patch, HADOOP-10768.005.patch,
> HADOOP-10768.006.patch, HADOOP-10768.007.patch, Optimize Hadoop RPC
> encryption performance.pdf
>
>
> Hadoop RPC encryption is enabled by setting {{hadoop.rpc.protection}} to
> "privacy". It utilized SASL {{GSSAPI}} and {{DIGEST-MD5}} mechanisms for
> secure authentication and data protection. Even {{GSSAPI}} supports using
> AES, but without AES-NI support by default, so the encryption is slow and
> will become bottleneck.
> After discuss with [~atm], [~tucu00] and [~umamaheswararao], we can do the
> same optimization as in HDFS-6606. Use AES-NI with more than *20x* speedup.
> On the other hand, RPC message is small, but RPC is frequent and there may be
> lots of RPC calls in one connection, we needs to setup benchmark to see real
> improvement and then make a trade-off.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252976#comment-16252976 ] Yonger commented on HADOOP-14475: - [~mackrorysd]Thank you. > Metrics of S3A don't print out when enable it in Hadoop metrics property file > -- > > Key: HADOOP-14475 > URL: https://issues.apache.org/jira/browse/HADOOP-14475 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: uname -a > Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017 > x86_64 x86_64 x86_64 GNU/Linux > cat /etc/issue > Ubuntu 16.04.2 LTS \n \l >Reporter: Yonger >Assignee: Yonger > Attachments: HADOOP-14475-003.patch, HADOOP-14475.002.patch, > HADOOP-14475.005.patch, HADOOP-14475.006.patch, HADOOP-14475.008.patch, > HADOOP-14775.007.patch, failsafe-report-s3a-it.html, > failsafe-report-s3a-scale.html, failsafe-report-scale.html, > failsafe-report-scale.zip, s3a-metrics.patch1, stdout.zip > > > *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink > #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #*.sink.influxdb.url=http:/xx > #*.sink.influxdb.influxdb_port=8086 > #*.sink.influxdb.database=hadoop > #*.sink.influxdb.influxdb_username=hadoop > #*.sink.influxdb.influxdb_password=hadoop > #*.sink.ingluxdb.cluster=c1 > *.period=10 > #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out > I can't find the out put file even i run a MR job which should be used s3. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14964) AliyunOSS: backport Aliyun OSS module to branch-2 and 2.7+ branches
[
https://issues.apache.org/jira/browse/HADOOP-14964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252957#comment-16252957
]
Hadoop QA commented on HADOOP-14964:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m
29s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 19 new or modified test
files. {color} |
|| || || || {color:brown} branch-2.8 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
21s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
33s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
19s{color} | {color:green} branch-2.8 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-tools hadoop-tools/hadoop-tools-dist {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} branch-2.8 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
34s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 0s{color} | {color:orange} root: The patch generated 2 new + 0 unchanged -
0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-tools hadoop-tools/hadoop-tools-dist {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
10s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
14s{color} | {color:green} hadoop-aliyun in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 48m 2s{color}
| {color:red} hadoop-tools in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
14s{color} | {color:green} hadoop-tools-dist in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 96m 26s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Unreaped Processes | hadoop-tools:1 |
| Failed junit tests | hadoop.yarn.sls.TestSLSRunner |
| | hadoop.tools.TestIntegration |
| | hadoop.tools.TestDistCpViewFs |
| | hadoop.fs.adl.live.TestAdlFileSystemContractLive |
| Timed out junit tests |
[jira] [Work started] (HADOOP-15039) move SemaphoredDelegatingExecutor to hadoop-common
[
https://issues.apache.org/jira/browse/HADOOP-15039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HADOOP-15039 started by Genmao Yu.
--
> move SemaphoredDelegatingExecutor to hadoop-common
> --
>
> Key: HADOOP-15039
> URL: https://issues.apache.org/jira/browse/HADOOP-15039
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs, fs/oss, fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Genmao Yu
>Assignee: Genmao Yu
>Priority: Minor
>
> Detailed discussions in HADOOP-14999 and HADOOP-15027.
> share {{SemaphoredDelegatingExecutor}} and move it to {{hadoop-common}}.
> cc [~ste...@apache.org]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12725) RPC encryption benchmark and optimization prototypes
[ https://issues.apache.org/jira/browse/HADOOP-12725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252900#comment-16252900 ] Kai Zheng commented on HADOOP-12725: Hello [~shv], Thanks for your interesting on this. We're aligned with HADOOP-10768 and looks like the work is going pretty well over there. Would you look at that one? Thanks! > RPC encryption benchmark and optimization prototypes > > > Key: HADOOP-12725 > URL: https://issues.apache.org/jira/browse/HADOOP-12725 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Kai Zheng >Assignee: Wei Zhou > > This would implement a benchmark tool to measure and compare the performance > of Hadoop IPC/RPC call when security is enabled and different SASL > QOP(Quality of Protection) is enforced. Given the data collected by this > benchmark, it would then be able to know if any performance concern when > considering to enforce privacy, integration, or authenticy protection level, > and do optimization accordingly. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14964) AliyunOSS: backport Aliyun OSS module to branch-2 and 2.7+ branches
[ https://issues.apache.org/jira/browse/HADOOP-14964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SammiChen updated HADOOP-14964: --- Attachment: HADOOP-14964-branch-2.8.000.patch Patch for branch-2.8 > AliyunOSS: backport Aliyun OSS module to branch-2 and 2.7+ branches > --- > > Key: HADOOP-14964 > URL: https://issues.apache.org/jira/browse/HADOOP-14964 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss >Reporter: Genmao Yu >Assignee: SammiChen > Attachments: HADOOP-14964-branch-2.000.patch, > HADOOP-14964-branch-2.8.000.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-10768) Optimize Hadoop RPC encryption performance
[
https://issues.apache.org/jira/browse/HADOOP-10768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252894#comment-16252894
]
Dapeng Sun commented on HADOOP-10768:
-
Here are my test commands:
No SASL:
{{java -cp `./bin/hadoop classpath` org.apache.hadoop.ipc.RPCCallBenchmark -r 4
-c 30 -s 30 -w 60 -t 60 -m 1024}}
SASL INTEGRITY:
{{java -cp `./bin/hadoop classpath` org.apache.hadoop.ipc.RPCCallBenchmark -r 4
-c 30 -s 30 -w 60 -t 60 -m 1024 -a -q INTEGRITY}}
SASL PRIVACY with AES:
{{java -cp `./bin/hadoop classpath` org.apache.hadoop.ipc.RPCCallBenchmark -r 4
-c 30 -s 30 -w 60 -t 60 -m 1024 -a -q PRIVACY -f AES/CTR/NoPadding}}
SASL PRIVACY with original DES:
{{java -cp `./bin/hadoop classpath` org.apache.hadoop.ipc.RPCCallBenchmark -r 4
-c 30 -s 30 -w 60 -t 60 -m 1024 -a -q PRIVACY}}
> Optimize Hadoop RPC encryption performance
> --
>
> Key: HADOOP-10768
> URL: https://issues.apache.org/jira/browse/HADOOP-10768
> Project: Hadoop Common
> Issue Type: Improvement
> Components: performance, security
>Affects Versions: 3.0.0-alpha1
>Reporter: Yi Liu
>Assignee: Dapeng Sun
> Attachments: HADOOP-10768.001.patch, HADOOP-10768.002.patch,
> HADOOP-10768.003.patch, HADOOP-10768.004.patch, HADOOP-10768.005.patch,
> HADOOP-10768.006.patch, Optimize Hadoop RPC encryption performance.pdf
>
>
> Hadoop RPC encryption is enabled by setting {{hadoop.rpc.protection}} to
> "privacy". It utilized SASL {{GSSAPI}} and {{DIGEST-MD5}} mechanisms for
> secure authentication and data protection. Even {{GSSAPI}} supports using
> AES, but without AES-NI support by default, so the encryption is slow and
> will become bottleneck.
> After discuss with [~atm], [~tucu00] and [~umamaheswararao], we can do the
> same optimization as in HDFS-6606. Use AES-NI with more than *20x* speedup.
> On the other hand, RPC message is small, but RPC is frequent and there may be
> lots of RPC calls in one connection, we needs to setup benchmark to see real
> improvement and then make a trade-off.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-10768) Optimize Hadoop RPC encryption performance
[
https://issues.apache.org/jira/browse/HADOOP-10768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252892#comment-16252892
]
Dapeng Sun edited comment on HADOOP-10768 at 11/15/17 3:37 AM:
---
Hi, [~daryn], [~atm], I have updated the patch, please help to review if you
time allow.
{quote}Add some unit tests and results of measuring the performance improvement
{quote}
I have finished the initial Benchmark with {{RPC Call Benchmark}} and also
added two UTs in the latest patch, the benchmark result shows the patch improve
{{~4X}} on the throughput of RPC call, now the performance of PRIVACY is close
to INTEGRITY’s, here is the detail data of total calls per second:
||SASL||r:4 c:1 s:1 m:1024|| r:4 c:30 s:30 m:1024||
|NONE|17102|233726|
|INTEGRITY|11693|114303|
|PRIVACY (AES with HadoopOpensslCodec)|9194|112763|
|PRIVACY (AES with HadoopJceCodec)|7718|111807|
|PRIVACY (Original DES)|1872|34007|
{quote}Why not use javax cipher libraries? Any number of ciphers could be used
now and in the future w/o code change. The aes ciphers are supposed to use
aes-ni intrinsics when available.{quote}
JDK cipher doesn’t take advantage of Intel AES-NI very well until JDK 9
([JDK-8143925|https://bugs.openjdk.java.net/browse/JDK-8143925] Improve JDK 9
AES-CTR with 4~6x gain), so I think using Crypto Stream in Hadoop should be a
good choice.
{quote}Should use a custom sasl client/server that delegates to the actual sasl
instance. The ipc layer changes would be minimal and easier to maintain.{quote}
It is hard to separate current logic to independent sasl client/server with
minimal changes, I have move the logic to separate method for minimizing the
changes.
{quote}The cipher options appears to be present in every packet. If so, it
should only be in the negotiate/initiate messages.{quote}
The cipher option only appears in SASL negotiate/initiate.
Thanks.
was (Author: dapengsun):
Hi, Daryn Sharp, Aaron T. Myers, I have updated the patch, please help to
review if you time allow.
{quote}Add some unit tests and results of measuring the performance improvement
{quote}
I have finished the initial Benchmark with {{RPC Call Benchmark}} and also
added two UTs in the latest patch, the benchmark result shows the patch improve
{{~4X}} on the throughput of RPC call, now the performance of PRIVACY is close
to INTEGRITY’s, here is the detail data of total calls per second:
||SASL||r:4 c:1 s:1 m:1024|| r:4 c:30 s:30 m:1024||
|NONE|17102|233726|
|INTEGRITY|11693|114303|
|PRIVACY (AES with HadoopOpensslCodec)|9194|112763|
|PRIVACY (AES with HadoopJceCodec)|7718|111807|
|PRIVACY (Original DES)|1872|34007|
{quote}Why not use javax cipher libraries? Any number of ciphers could be used
now and in the future w/o code change. The aes ciphers are supposed to use
aes-ni intrinsics when available.{quote}
JDK cipher doesn’t take advantage of Intel AES-NI very well until JDK 9
([JDK-8143925|https://bugs.openjdk.java.net/browse/JDK-8143925] Improve JDK 9
AES-CTR with 4~6x gain), so I think using Crypto Stream in Hadoop should be a
good choice.
{quote}Should use a custom sasl client/server that delegates to the actual sasl
instance. The ipc layer changes would be minimal and easier to maintain.{quote}
It is hard to separate current logic to independent sasl client/server with
minimal changes, I have move the logic to separate method for minimizing the
changes.
{quote}The cipher options appears to be present in every packet. If so, it
should only be in the negotiate/initiate messages.{quote}
The cipher option only appears in SASL negotiate/initiate.
Thanks.
> Optimize Hadoop RPC encryption performance
> --
>
> Key: HADOOP-10768
> URL: https://issues.apache.org/jira/browse/HADOOP-10768
> Project: Hadoop Common
> Issue Type: Improvement
> Components: performance, security
>Affects Versions: 3.0.0-alpha1
>Reporter: Yi Liu
>Assignee: Dapeng Sun
> Attachments: HADOOP-10768.001.patch, HADOOP-10768.002.patch,
> HADOOP-10768.003.patch, HADOOP-10768.004.patch, HADOOP-10768.005.patch,
> HADOOP-10768.006.patch, Optimize Hadoop RPC encryption performance.pdf
>
>
> Hadoop RPC encryption is enabled by setting {{hadoop.rpc.protection}} to
> "privacy". It utilized SASL {{GSSAPI}} and {{DIGEST-MD5}} mechanisms for
> secure authentication and data protection. Even {{GSSAPI}} supports using
> AES, but without AES-NI support by default, so the encryption is slow and
> will become bottleneck.
> After discuss with [~atm], [~tucu00] and [~umamaheswararao], we can do the
> same optimization as in HDFS-6606. Use AES-NI with more than *20x* speedup.
> On the other hand, RPC message is small, but RPC is frequent and there may be
> lots of RPC calls in one connection, we needs to setup benchmark to see real
[jira] [Commented] (HADOOP-10768) Optimize Hadoop RPC encryption performance
[
https://issues.apache.org/jira/browse/HADOOP-10768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252892#comment-16252892
]
Dapeng Sun commented on HADOOP-10768:
-
Hi, Daryn Sharp, Aaron T. Myers, I have updated the patch, please help to
review if you time allow.
{quote}Add some unit tests and results of measuring the performance improvement
{quote}
I have finished the initial Benchmark with {{RPC Call Benchmark}} and also
added two UTs in the latest patch, the benchmark result shows the patch improve
{{~4X}} on the throughput of RPC call, now the performance of PRIVACY is close
to INTEGRITY’s, here is the detail data of total calls per second:
||SASL||r:4 c:1 s:1 m:1024|| r:4 c:30 s:30 m:1024||
|NONE|17102|233726|
|INTEGRITY|11693|114303|
|PRIVACY (AES with HadoopOpensslCodec)|9194|112763|
|PRIVACY (AES with HadoopJceCodec)|7718|111807|
|PRIVACY (Original DES)|1872|34007|
{quote}Why not use javax cipher libraries? Any number of ciphers could be used
now and in the future w/o code change. The aes ciphers are supposed to use
aes-ni intrinsics when available.{quote}
JDK cipher doesn’t take advantage of Intel AES-NI very well until JDK 9
([JDK-8143925|https://bugs.openjdk.java.net/browse/JDK-8143925] Improve JDK 9
AES-CTR with 4~6x gain), so I think using Crypto Stream in Hadoop should be a
good choice.
{quote}Should use a custom sasl client/server that delegates to the actual sasl
instance. The ipc layer changes would be minimal and easier to maintain.{quote}
It is hard to separate current logic to independent sasl client/server with
minimal changes, I have move the logic to separate method for minimizing the
changes.
{quote}The cipher options appears to be present in every packet. If so, it
should only be in the negotiate/initiate messages.{quote}
The cipher option only appears in SASL negotiate/initiate.
Thanks.
> Optimize Hadoop RPC encryption performance
> --
>
> Key: HADOOP-10768
> URL: https://issues.apache.org/jira/browse/HADOOP-10768
> Project: Hadoop Common
> Issue Type: Improvement
> Components: performance, security
>Affects Versions: 3.0.0-alpha1
>Reporter: Yi Liu
>Assignee: Dapeng Sun
> Attachments: HADOOP-10768.001.patch, HADOOP-10768.002.patch,
> HADOOP-10768.003.patch, HADOOP-10768.004.patch, HADOOP-10768.005.patch,
> HADOOP-10768.006.patch, Optimize Hadoop RPC encryption performance.pdf
>
>
> Hadoop RPC encryption is enabled by setting {{hadoop.rpc.protection}} to
> "privacy". It utilized SASL {{GSSAPI}} and {{DIGEST-MD5}} mechanisms for
> secure authentication and data protection. Even {{GSSAPI}} supports using
> AES, but without AES-NI support by default, so the encryption is slow and
> will become bottleneck.
> After discuss with [~atm], [~tucu00] and [~umamaheswararao], we can do the
> same optimization as in HDFS-6606. Use AES-NI with more than *20x* speedup.
> On the other hand, RPC message is small, but RPC is frequent and there may be
> lots of RPC calls in one connection, we needs to setup benchmark to see real
> improvement and then make a trade-off.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252886#comment-16252886 ] Sean Mackrory commented on HADOOP-14475: Okay - thanks for the additional context. I think the fact that the .007. patch has the JIRA # slightly wrong my shell commands got messed up and resulted in a diff of identical files somehow. The .008. patch is looking pretty good to me and includes all the changes I had made to the test locally while messing with it. I'll review everything else about the latest patch and probably give you a +1 and commit tomorrow (and I'll address those 2 checkstyle issues then). > Metrics of S3A don't print out when enable it in Hadoop metrics property file > -- > > Key: HADOOP-14475 > URL: https://issues.apache.org/jira/browse/HADOOP-14475 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: uname -a > Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017 > x86_64 x86_64 x86_64 GNU/Linux > cat /etc/issue > Ubuntu 16.04.2 LTS \n \l >Reporter: Yonger >Assignee: Yonger > Attachments: HADOOP-14475-003.patch, HADOOP-14475.002.patch, > HADOOP-14475.005.patch, HADOOP-14475.006.patch, HADOOP-14475.008.patch, > HADOOP-14775.007.patch, failsafe-report-s3a-it.html, > failsafe-report-s3a-scale.html, failsafe-report-scale.html, > failsafe-report-scale.zip, s3a-metrics.patch1, stdout.zip > > > *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink > #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #*.sink.influxdb.url=http:/xx > #*.sink.influxdb.influxdb_port=8086 > #*.sink.influxdb.database=hadoop > #*.sink.influxdb.influxdb_username=hadoop > #*.sink.influxdb.influxdb_password=hadoop > #*.sink.ingluxdb.cluster=c1 > *.period=10 > #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out > I can't find the out put file even i run a MR job which should be used s3. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12725) RPC encryption benchmark and optimization prototypes
[ https://issues.apache.org/jira/browse/HADOOP-12725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252869#comment-16252869 ] Konstantin Shvachko commented on HADOOP-12725: -- Hey guys I actually ran some benchmarks to measure the performance of encryption for NameNode RPCs. I jsut used NNThoughputBenchmark, which you can now run against a real NameNode. So I would start a stand-alone NN with different security settings and run NNThroughput from a different machine. Would that be a good enough bechmark? Are you targeting other than NN RPCs here? BTW the ecnrypted RPC performance was not good, although not 25x either, but definitely worth optimizing. > RPC encryption benchmark and optimization prototypes > > > Key: HADOOP-12725 > URL: https://issues.apache.org/jira/browse/HADOOP-12725 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Kai Zheng >Assignee: Wei Zhou > > This would implement a benchmark tool to measure and compare the performance > of Hadoop IPC/RPC call when security is enabled and different SASL > QOP(Quality of Protection) is enforced. Given the data collected by this > benchmark, it would then be able to know if any performance concern when > considering to enforce privacy, integration, or authenticy protection level, > and do optimization accordingly. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[
https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252804#comment-16252804
]
Yonger edited comment on HADOOP-14475 at 11/15/17 1:44 AM:
---
For the # 2 point of my update
--
without this update,the metrics be flush into file like this(each record with
same URI mark the same "context+registry name" as begin):
{code:java}
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=dd721794-8269-44eb-911f-a39caa2e6b34-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=8fe27f60-8b3d-41c1-aa84-8bb812315d55-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=6bcf9a67-2dcd-4533-aa5b-2ded7a8ba038-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
{code}
Note, these three records written at the same time instead of different
intervals, but they should be consider three different kinds of
metrics(different fsid,different values in real logs).
(Additional, I don't know why the metrics be registered many times within a
process,like why we need to method "newMetricsSourceName")
with this update, each record will be easy to distinguish:
{code:java}
1510631110244 s3afs.S3aFileSystemMetrics: Context=s3afs,
FileSystemId=dd721794-8269-44eb-911f-a39caa2e6b34-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3aFileSystemMetrics2: Context=s3afs,
FileSystemId=8fe27f60-8b3d-41c1-aa84-8bb812315d55-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3aFileSystemMetrics3: Context=s3afs,
FileSystemId=6bcf9a67-2dcd-4533-aa5b-2ded7a8ba038-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
{code}
was (Author: iyonger):
For the # 2 point of my update
--
without this update,the metrics be flush into file like this(each record with
same URI mark the same "context+registry name" as begin):
{code:java}
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=dd721794-8269-44eb-911f-a39caa2e6b34-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=8fe27f60-8b3d-41c1-aa84-8bb812315d55-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=6bcf9a67-2dcd-4533-aa5b-2ded7a8ba038-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
{code}
Note, these three records written at the same time instead of different
intervals, so they should be consider three different kinds of metrics.
(Additional, I don't know why the metrics be registered many times within a
process,like why we need to method "newMetricsSourceName")
with this update, each record will be easy to distinguish:
{code:java}
1510631110244 s3afs.S3aFileSystemMetrics: Context=s3afs,
FileSystemId=dd721794-8269-44eb-911f-a39caa2e6b34-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3aFileSystemMetrics2: Context=s3afs,
FileSystemId=8fe27f60-8b3d-41c1-aa84-8bb812315d55-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3aFileSystemMetrics3: Context=s3afs,
FileSystemId=6bcf9a67-2dcd-4533-aa5b-2ded7a8ba038-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
{code}
> Metrics of S3A don't print out when enable it in Hadoop metrics property file
> --
>
> Key: HADOOP-14475
> URL: https://issues.apache.org/jira/browse/HADOOP-14475
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 2.8.0
> Environment: uname -a
> Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017
> x86_64 x86_64 x86_64 GNU/Linux
> cat /etc/issue
> Ubuntu 16.04.2 LTS \n \l
>Reporter: Yonger
>Assignee: Yonger
> Attachments: HADOOP-14475-003.patch, HADOOP-14475.002.patch,
> HADOOP-14475.005.patch, HADOOP-14475.006.patch, HADOOP-14475.008.patch,
> HADOOP-14775.007.patch, failsafe-report-s3a-it.html,
> failsafe-report-s3a-scale.html, failsafe-report-scale.html,
> failsafe-report-scale.zip, s3a-metrics.patch1, stdout.zip
>
>
> *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink
> #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink
> #*.sink.influxdb.url=http:/xx
> #*.sink.influxdb.influxdb_port=8086
> #*.sink.influxdb.database=hadoop
> #*.sink.influxdb.influxdb_username=hadoop
> #*.sink.influxdb.influxdb_password=hadoop
> #*.sink.ingluxdb.cluster=c1
> *.period=10
>
[jira] [Comment Edited] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[
https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252804#comment-16252804
]
Yonger edited comment on HADOOP-14475 at 11/15/17 1:42 AM:
---
For the # 2 point of my update
--
without this update,the metrics be flush into file like this(each record with
same URI mark the same "context+registry name" as begin):
{code:java}
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=dd721794-8269-44eb-911f-a39caa2e6b34-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=8fe27f60-8b3d-41c1-aa84-8bb812315d55-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=6bcf9a67-2dcd-4533-aa5b-2ded7a8ba038-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
{code}
Note, these three records written at the same time instead of different
intervals, so they should be consider three different kinds of metrics.
(Additional, I don't know why the metrics be registered many times within a
process,like why we need to method "newMetricsSourceName")
with this update, each record will be easy to distinguish:
{code:java}
1510631110244 s3afs.S3aFileSystemMetrics: Context=s3afs,
FileSystemId=dd721794-8269-44eb-911f-a39caa2e6b34-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3aFileSystemMetrics2: Context=s3afs,
FileSystemId=8fe27f60-8b3d-41c1-aa84-8bb812315d55-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3aFileSystemMetrics3: Context=s3afs,
FileSystemId=6bcf9a67-2dcd-4533-aa5b-2ded7a8ba038-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
{code}
was (Author: iyonger):
For the # 2 point of my update
--
without this update,the metrics be flush into file like this(each record with
same URI mark the same "context+registry name" as begin):
{code:java}
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=dd721794-8269-44eb-911f-a39caa2e6b34-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=8fe27f60-8b3d-41c1-aa84-8bb812315d55-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=6bcf9a67-2dcd-4533-aa5b-2ded7a8ba038-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
{code}
with this update, each record will be easy to distinguish:
{code:java}
1510631110244 s3afs.S3aFileSystemMetrics: Context=s3afs,
FileSystemId=dd721794-8269-44eb-911f-a39caa2e6b34-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3aFileSystemMetrics2: Context=s3afs,
FileSystemId=8fe27f60-8b3d-41c1-aa84-8bb812315d55-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3aFileSystemMetrics3: Context=s3afs,
FileSystemId=6bcf9a67-2dcd-4533-aa5b-2ded7a8ba038-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
{code}
> Metrics of S3A don't print out when enable it in Hadoop metrics property file
> --
>
> Key: HADOOP-14475
> URL: https://issues.apache.org/jira/browse/HADOOP-14475
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 2.8.0
> Environment: uname -a
> Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017
> x86_64 x86_64 x86_64 GNU/Linux
> cat /etc/issue
> Ubuntu 16.04.2 LTS \n \l
>Reporter: Yonger
>Assignee: Yonger
> Attachments: HADOOP-14475-003.patch, HADOOP-14475.002.patch,
> HADOOP-14475.005.patch, HADOOP-14475.006.patch, HADOOP-14475.008.patch,
> HADOOP-14775.007.patch, failsafe-report-s3a-it.html,
> failsafe-report-s3a-scale.html, failsafe-report-scale.html,
> failsafe-report-scale.zip, s3a-metrics.patch1, stdout.zip
>
>
> *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink
> #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink
> #*.sink.influxdb.url=http:/xx
> #*.sink.influxdb.influxdb_port=8086
> #*.sink.influxdb.database=hadoop
> #*.sink.influxdb.influxdb_username=hadoop
> #*.sink.influxdb.influxdb_password=hadoop
> #*.sink.ingluxdb.cluster=c1
> *.period=10
> #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink
> #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink
> S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out
> I can't find the out put file even i run a MR job which should be used s3.
--
This message was sent by Atlassian
[jira] [Comment Edited] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[
https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252804#comment-16252804
]
Yonger edited comment on HADOOP-14475 at 11/15/17 1:31 AM:
---
For the # 2 point of my update
--
without this update,the metrics be flush into file like this(each record with
same URI mark the same "context+registry name" as begin):
{code:java}
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=dd721794-8269-44eb-911f-a39caa2e6b34-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=8fe27f60-8b3d-41c1-aa84-8bb812315d55-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=6bcf9a67-2dcd-4533-aa5b-2ded7a8ba038-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
{code}
with this update, each record will be easy to distinguish:
{code:java}
1510631110244 s3afs.S3aFileSystemMetrics: Context=s3afs,
FileSystemId=dd721794-8269-44eb-911f-a39caa2e6b34-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3aFileSystemMetrics2: Context=s3afs,
FileSystemId=8fe27f60-8b3d-41c1-aa84-8bb812315d55-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3aFileSystemMetrics3: Context=s3afs,
FileSystemId=6bcf9a67-2dcd-4533-aa5b-2ded7a8ba038-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
{code}
was (Author: iyonger):
For the # 2 point of my update
--
without this update,the metrics be flush into file like this(each record mark
the same "context+registry name" as begin):
{code:java}
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=dd721794-8269-44eb-911f-a39caa2e6b34-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=8fe27f60-8b3d-41c1-aa84-8bb812315d55-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=6bcf9a67-2dcd-4533-aa5b-2ded7a8ba038-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
{code}
with this update, each record will be easy to distinguish:
{code:java}
1510631110244 s3afs.S3aFileSystemMetrics: Context=s3afs,
FileSystemId=dd721794-8269-44eb-911f-a39caa2e6b34-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3aFileSystemMetrics2: Context=s3afs,
FileSystemId=8fe27f60-8b3d-41c1-aa84-8bb812315d55-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3aFileSystemMetrics3: Context=s3afs,
FileSystemId=6bcf9a67-2dcd-4533-aa5b-2ded7a8ba038-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
{code}
> Metrics of S3A don't print out when enable it in Hadoop metrics property file
> --
>
> Key: HADOOP-14475
> URL: https://issues.apache.org/jira/browse/HADOOP-14475
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 2.8.0
> Environment: uname -a
> Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017
> x86_64 x86_64 x86_64 GNU/Linux
> cat /etc/issue
> Ubuntu 16.04.2 LTS \n \l
>Reporter: Yonger
>Assignee: Yonger
> Attachments: HADOOP-14475-003.patch, HADOOP-14475.002.patch,
> HADOOP-14475.005.patch, HADOOP-14475.006.patch, HADOOP-14475.008.patch,
> HADOOP-14775.007.patch, failsafe-report-s3a-it.html,
> failsafe-report-s3a-scale.html, failsafe-report-scale.html,
> failsafe-report-scale.zip, s3a-metrics.patch1, stdout.zip
>
>
> *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink
> #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink
> #*.sink.influxdb.url=http:/xx
> #*.sink.influxdb.influxdb_port=8086
> #*.sink.influxdb.database=hadoop
> #*.sink.influxdb.influxdb_username=hadoop
> #*.sink.influxdb.influxdb_password=hadoop
> #*.sink.ingluxdb.cluster=c1
> *.period=10
> #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink
> #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink
> S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out
> I can't find the out put file even i run a MR job which should be used s3.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[
https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252804#comment-16252804
]
Yonger commented on HADOOP-14475:
-
For the # 2 point of my update
--
without this update,the metrics be flush into file like this(each record mark
the same "context+registry name" as begin):
{code:java}
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=dd721794-8269-44eb-911f-a39caa2e6b34-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=8fe27f60-8b3d-41c1-aa84-8bb812315d55-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=6bcf9a67-2dcd-4533-aa5b-2ded7a8ba038-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
{code}
with this update, each record will be easy to distinguish:
{code:java}
1510631110244 s3afs.S3aFileSystemMetrics: Context=s3afs,
FileSystemId=dd721794-8269-44eb-911f-a39caa2e6b34-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3aFileSystemMetrics2: Context=s3afs,
FileSystemId=8fe27f60-8b3d-41c1-aa84-8bb812315d55-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3aFileSystemMetrics3: Context=s3afs,
FileSystemId=6bcf9a67-2dcd-4533-aa5b-2ded7a8ba038-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
{code}
> Metrics of S3A don't print out when enable it in Hadoop metrics property file
> --
>
> Key: HADOOP-14475
> URL: https://issues.apache.org/jira/browse/HADOOP-14475
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 2.8.0
> Environment: uname -a
> Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017
> x86_64 x86_64 x86_64 GNU/Linux
> cat /etc/issue
> Ubuntu 16.04.2 LTS \n \l
>Reporter: Yonger
>Assignee: Yonger
> Attachments: HADOOP-14475-003.patch, HADOOP-14475.002.patch,
> HADOOP-14475.005.patch, HADOOP-14475.006.patch, HADOOP-14475.008.patch,
> HADOOP-14775.007.patch, failsafe-report-s3a-it.html,
> failsafe-report-s3a-scale.html, failsafe-report-scale.html,
> failsafe-report-scale.zip, s3a-metrics.patch1, stdout.zip
>
>
> *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink
> #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink
> #*.sink.influxdb.url=http:/xx
> #*.sink.influxdb.influxdb_port=8086
> #*.sink.influxdb.database=hadoop
> #*.sink.influxdb.influxdb_username=hadoop
> #*.sink.influxdb.influxdb_password=hadoop
> #*.sink.ingluxdb.cluster=c1
> *.period=10
> #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink
> #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink
> S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out
> I can't find the out put file even i run a MR job which should be used s3.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252798#comment-16252798 ] Yonger commented on HADOOP-14475: - [~macdonsp]the .008 patch is newer than .007 patch, I think it's the correct code that I want to give you guys. > Metrics of S3A don't print out when enable it in Hadoop metrics property file > -- > > Key: HADOOP-14475 > URL: https://issues.apache.org/jira/browse/HADOOP-14475 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: uname -a > Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017 > x86_64 x86_64 x86_64 GNU/Linux > cat /etc/issue > Ubuntu 16.04.2 LTS \n \l >Reporter: Yonger >Assignee: Yonger > Attachments: HADOOP-14475-003.patch, HADOOP-14475.002.patch, > HADOOP-14475.005.patch, HADOOP-14475.006.patch, HADOOP-14475.008.patch, > HADOOP-14775.007.patch, failsafe-report-s3a-it.html, > failsafe-report-s3a-scale.html, failsafe-report-scale.html, > failsafe-report-scale.zip, s3a-metrics.patch1, stdout.zip > > > *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink > #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #*.sink.influxdb.url=http:/xx > #*.sink.influxdb.influxdb_port=8086 > #*.sink.influxdb.database=hadoop > #*.sink.influxdb.influxdb_username=hadoop > #*.sink.influxdb.influxdb_password=hadoop > #*.sink.ingluxdb.cluster=c1 > *.period=10 > #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out > I can't find the out put file even i run a MR job which should be used s3. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15040) AWS SDK NPE bug spams logs w/ Yarn Log Aggregation
Aaron Fabbri created HADOOP-15040:
-
Summary: AWS SDK NPE bug spams logs w/ Yarn Log Aggregation
Key: HADOOP-15040
URL: https://issues.apache.org/jira/browse/HADOOP-15040
Project: Hadoop Common
Issue Type: Bug
Components: fs/s3
Affects Versions: 3.0.0-beta1
Reporter: Aaron Fabbri
Assignee: Aaron Fabbri
My colleagues working with Yarn log aggregation found that they were getting
this message spammed in their logs when they used an s3a:// URI for logs
(yarn.nodemanager.remote-app-log-dir):
{noformat}
getting attribute Region of com.amazonaws.management:type=AwsSdkMetrics threw
an exception
javax.management.RuntimeMBeanException: java.lang.NullPointerException
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:839)
at
Caused by: java.lang.NullPointerException
at com.amazonaws.metrics.AwsSdkMetrics.getRegion(AwsSdkMetrics.java:729)
at com.amazonaws.metrics.MetricAdmin.getRegion(MetricAdmin.java:67)
at sun.reflect.GeneratedMethodAccessor132.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
{noformat}
This happens even though the aws sdk cloudwatch metrics reporting was disabled
(default), which is a bug.
I filed a [github issue|https://github.com/aws/aws-sdk-java/issues/1375|] and
it looks like a fix should be coming around SDK release 1.11.229 or so.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15038) Abstract MetadataStore in S3Guard into a common module.
[
https://issues.apache.org/jira/browse/HADOOP-15038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252646#comment-16252646
]
Aaron Fabbri commented on HADOOP-15038:
---
I am happy to move {{MetadataStore}} and related interfaces outside of
{{hadoop-aws}} as soon as another module needs it. I originally intended for
this to be possible, just didn't want to pollute public hadoop common until it
was needed by another client. I know some of the ADLS developers were
interested in using it so I worked to keep it separate.
Can you comment on when you will need this? It might be a good idea to use
HADOOP-14098 as a top-level "umbrella" JIRA for your effort, and create
subtasks so we can see more details on how you plan to do the work?
> Abstract MetadataStore in S3Guard into a common module.
> ---
>
> Key: HADOOP-15038
> URL: https://issues.apache.org/jira/browse/HADOOP-15038
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs
>Affects Versions: 3.0.0-beta1
>Reporter: Genmao Yu
>
> Open this JIRA to discuss if we should move {{MetadataStore}} in {{S3Guard}}
> into a common module.
> Based on this work, other filesystem or object store can implement their own
> metastore for optimization (known issues like consistency problem and
> metadata operation performance). [~ste...@apache.org] and other guys have
> done many base and great works in {{S3Guard}}. It is very helpful to start
> work. I did some perf test in HADOOP-14098, and started related work for
> Aliyun OSS. Indeed there are still works to do for {{S3Guard}}, like
> metadata cache inconsistent with S3 and so on. It also will be a problem for
> other object store. However, we can do these works in parallel.
> [~ste...@apache.org] [~fabbri] [~drankye] Any suggestion is appreciated.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14098) AliyunOSS: improve the performance of object metadata operation
[
https://issues.apache.org/jira/browse/HADOOP-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252610#comment-16252610
]
Aaron Fabbri commented on HADOOP-14098:
---
Thanks for sharing your POC work. For reasoning about maximum speedup, you
could also use the {{LocalMetadataStore}} implementation as an in-memory cache.
That should provide near-optimal speedup.
> AliyunOSS: improve the performance of object metadata operation
> ---
>
> Key: HADOOP-14098
> URL: https://issues.apache.org/jira/browse/HADOOP-14098
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Affects Versions: 3.0.0-alpha2
>Reporter: Genmao Yu
>Assignee: Genmao Yu
>
> Open this JIRA to research and address the potential request performance
> issue.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15003) Merge S3A committers into trunk: Yetus patch checker
[ https://issues.apache.org/jira/browse/HADOOP-15003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252559#comment-16252559 ] Aaron Fabbri commented on HADOOP-15003: --- I completed 20 runs of scale and integration tests last night on v48 patch. Only failures were related to HADOOP-14927. > Merge S3A committers into trunk: Yetus patch checker > > > Key: HADOOP-15003 > URL: https://issues.apache.org/jira/browse/HADOOP-15003 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13786-041.patch, HADOOP-13786-042.patch, > HADOOP-13786-043.patch, HADOOP-13786-044.patch, HADOOP-13786-045.patch, > HADOOP-13786-046.patch, HADOOP-13786-047.patch, HADOOP-13786-048.patch, > HADOOP-13786-049.patch > > > This is a Yetus only JIRA created to have Yetus review the > HADOOP-13786/HADOOP-14971 patch as a .patch file, as the review PR > [https://github.com/apache/hadoop/pull/282] is stopping this happening in > HADOOP-14971. > Reviews should go into the PR/other task -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15003) Merge S3A committers into trunk: Yetus patch checker
[ https://issues.apache.org/jira/browse/HADOOP-15003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252549#comment-16252549 ] Aaron Fabbri commented on HADOOP-15003: --- Javadoc: First one looks like an infra issue. Second one I cannot view. Third one appears to be existing code? Javac: deprecation warnings, unavoidable. Checkstyle: mostly combination of javadoc links that cannot be broken into multi-line (AFAIK), and existing warnings in adjacent code (that we should fix at some point). Unit: TestCommonConfigurationFields. Is there an existing JIRA for this? I don't recall you modifying this part of the code. > Merge S3A committers into trunk: Yetus patch checker > > > Key: HADOOP-15003 > URL: https://issues.apache.org/jira/browse/HADOOP-15003 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13786-041.patch, HADOOP-13786-042.patch, > HADOOP-13786-043.patch, HADOOP-13786-044.patch, HADOOP-13786-045.patch, > HADOOP-13786-046.patch, HADOOP-13786-047.patch, HADOOP-13786-048.patch, > HADOOP-13786-049.patch > > > This is a Yetus only JIRA created to have Yetus review the > HADOOP-13786/HADOOP-14971 patch as a .patch file, as the review PR > [https://github.com/apache/hadoop/pull/282] is stopping this happening in > HADOOP-14971. > Reviews should go into the PR/other task -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14876) Create downstream developer docs from the compatibility guidelines
[ https://issues.apache.org/jira/browse/HADOOP-14876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252186#comment-16252186 ] Anu Engineer commented on HADOOP-14876: --- [~dan...@cloudera.com] Thanks for updating the patch. I am a +1 on this. Thanks for all the hard work, writing good documentation it is very hard and often not appreciated enough. > Create downstream developer docs from the compatibility guidelines > -- > > Key: HADOOP-14876 > URL: https://issues.apache.org/jira/browse/HADOOP-14876 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation >Affects Versions: 3.0.0-beta1 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Attachments: Compatibility.pdf, DownstreamDev.pdf, > HADOOP-14876.001.patch, HADOOP-14876.002.patch, HADOOP-14876.003.patch, > HADOOP-14876.004.patch, HADOOP-14876.005.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12071) conftest is not documented
[
https://issues.apache.org/jira/browse/HADOOP-12071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252183#comment-16252183
]
Hadoop QA commented on HADOOP-12071:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 16m
35s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 45s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 58s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 14s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 97m 50s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.security.TestRaceWhenRelogin |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-12071 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12883090/HADOOP-12071-002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 0f7460178e84 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 18621af |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13679/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13679/testReport/ |
| Max. process+thread count | 1349 (vs. ulimit of 5000) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
[jira] [Commented] (HADOOP-14519) Client$Connection#waitForWork may suffer from spurious wakeups
[
https://issues.apache.org/jira/browse/HADOOP-14519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252175#comment-16252175
]
Hadoop QA commented on HADOOP-14519:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 54s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
48s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 36s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 1 new + 102 unchanged - 0 fixed = 103 total (was 102) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 51s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
48s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 81m 46s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-14519 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12872408/HADOOP-14519.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 2cccdc3cf189 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 18621af |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13684/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13684/testReport/ |
| Max. process+thread count | 1462 (vs. ulimit of 5000) |
| modules | C: hadoop-common-project/hadoop-common U:
[jira] [Commented] (HADOOP-14876) Create downstream developer docs from the compatibility guidelines
[ https://issues.apache.org/jira/browse/HADOOP-14876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252164#comment-16252164 ] Daniel Templeton commented on HADOOP-14876: --- [~anu], [~steve_l], [~rkanter], any other comments? > Create downstream developer docs from the compatibility guidelines > -- > > Key: HADOOP-14876 > URL: https://issues.apache.org/jira/browse/HADOOP-14876 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation >Affects Versions: 3.0.0-beta1 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Attachments: Compatibility.pdf, DownstreamDev.pdf, > HADOOP-14876.001.patch, HADOOP-14876.002.patch, HADOOP-14876.003.patch, > HADOOP-14876.004.patch, HADOOP-14876.005.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14807) should prevent the possibility of NPE about ReconfigurableBase.java
[
https://issues.apache.org/jira/browse/HADOOP-14807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252157#comment-16252157
]
Hadoop QA commented on HADOOP-14807:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 50s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 25s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
16s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 72m 10s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-14807 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12883655/HADOOP-14807.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 899b13d490f2 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 18621af |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13681/testReport/ |
| Max. process+thread count | 1421 (vs. ulimit of 5000) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13681/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> should prevent the possibility
[jira] [Commented] (HADOOP-15003) Merge S3A committers into trunk: Yetus patch checker
[
https://issues.apache.org/jira/browse/HADOOP-15003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252156#comment-16252156
]
Hadoop QA commented on HADOOP-15003:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 13m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 58 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
1s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 31m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 26m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 7m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
26m 33s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 10m
5s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
39s{color} | {color:red} hadoop-mapreduce-client-core in trunk failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
38s{color} | {color:red} hadoop-mapreduce-client-app in trunk failed. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
53s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 12m 53s{color}
| {color:red} root generated 2 new + 1234 unchanged - 0 fixed = 1236 total (was
1234) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 3s{color} | {color:orange} root: The patch generated 28 new + 117 unchanged
- 22 fixed = 145 total (was 139) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
23s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 75 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
1s{color} | {color:red} The patch 3 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
6s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 0s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
36s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
32s{color} | {color:red}
hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core
generated 19 new + 0 unchanged - 0 fixed = 19 total (was 0) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 50s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
0s{color} | {color:green} hadoop-yarn-registry in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
16s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
10s{color} | {color:green} hadoop-mapreduce-client-app in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} |
[jira] [Commented] (HADOOP-14791) SimpleKdcServer: Fail to delete krb5 conf
[
https://issues.apache.org/jira/browse/HADOOP-14791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252140#comment-16252140
]
Hadoop QA commented on HADOOP-14791:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 26s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
8m 56s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
34s{color} | {color:red} hadoop-common-project/hadoop-minikdc generated 1 new +
0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
25s{color} | {color:green} hadoop-minikdc in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 63m 13s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-common-project/hadoop-minikdc |
| | Exceptional return value of java.io.File.delete() ignored in
org.apache.hadoop.minikdc.MiniKdc.main(String[]) At MiniKdc.java:ignored in
org.apache.hadoop.minikdc.MiniKdc.main(String[]) At MiniKdc.java:[line 113] |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-14791 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12882765/HADOOP-14791.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 23bb7a3e0c68 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 18621af |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| findbugs |
[jira] [Commented] (HADOOP-13238) pid handling is failing on secure datanode
[
https://issues.apache.org/jira/browse/HADOOP-13238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252137#comment-16252137
]
Hadoop QA commented on HADOOP-13238:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 16m
59s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 28s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
4s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
8s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 8s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
6s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 61m 52s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-13238 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/1282/HADOOP-13238.02.patch
|
| Optional Tests | asflicense mvnsite unit shellcheck shelldocs |
| uname | Linux 5f32e8992cca 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 18621af |
| maven | version: Apache Maven 3.3.9 |
| shellcheck | v0.4.6 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13678/testReport/ |
| Max. process+thread count | 298 (vs. ulimit of 5000) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13678/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> pid handling is failing on secure datanode
> --
>
> Key: HADOOP-13238
> URL: https://issues.apache.org/jira/browse/HADOOP-13238
> Project: Hadoop Common
> Issue Type: Bug
> Components: scripts, security
>Reporter: Allen Wittenauer
>Assignee: Andras Bokor
> Attachments: HADOOP-13238.01.patch, HADOOP-13238.02.patch
>
>
> {code}
> hdfs --daemon stop datanode
> cat: /home/hadoop/H/pids/hadoop-hdfs-root-datanode.pid: No such file or
> directory
> WARNING: pid has changed for datanode, skip deleting pid file
> cat: /home/hadoop/H/pids/hadoop-hdfs-root-datanode.pid: No such file or
> directory
> WARNING: daemon pid has changed for datanode, skip deleting daemon pid file
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14776) clean up ITestS3AFileSystemContract
[
https://issues.apache.org/jira/browse/HADOOP-14776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252076#comment-16252076
]
Hadoop QA commented on HADOOP-14776:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 12m
0s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 1m
15s{color} | {color:red} root in trunk failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
20s{color} | {color:red} hadoop-aws in trunk failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
20s{color} | {color:red} hadoop-aws in trunk failed. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 3m
37s{color} | {color:red} branch has errors when building and testing our client
artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
20s{color} | {color:red} hadoop-aws in trunk failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
20s{color} | {color:red} hadoop-aws in trunk failed. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
20s{color} | {color:red} hadoop-aws in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
20s{color} | {color:red} hadoop-aws in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 20s{color}
| {color:red} hadoop-aws in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
21s{color} | {color:red} hadoop-aws in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
0m 13s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
6s{color} | {color:red} hadoop-aws in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
21s{color} | {color:red} hadoop-aws in the patch failed. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 20s{color}
| {color:red} hadoop-aws in the patch failed. {color} |
| {color:blue}0{color} | {color:blue} asflicense {color} | {color:blue} 0m
20s{color} | {color:blue} ASF License check generated no output? {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 39s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-14776 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12882007/HADOOP-14776.01.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 76f52f8038c1 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 18621af |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| mvninstall |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13682/artifact/out/branch-mvninstall-root.txt
|
| compile |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13682/artifact/out/branch-compile-hadoop-tools_hadoop-aws.txt
|
| mvnsite |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13682/artifact/out/branch-mvnsite-hadoop-tools_hadoop-aws.txt
|
| findbugs |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13682/artifact/out/branch-findbugs-hadoop-tools_hadoop-aws.txt
|
| javadoc |
[jira] [Updated] (HADOOP-14519) Client$Connection#waitForWork may suffer from spurious wakeups
[
https://issues.apache.org/jira/browse/HADOOP-14519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HADOOP-14519:
-
Target Version/s: 2.8.3, 3.1.0 (was: 2.8.3, 3.0.0)
> Client$Connection#waitForWork may suffer from spurious wakeups
> --
>
> Key: HADOOP-14519
> URL: https://issues.apache.org/jira/browse/HADOOP-14519
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 2.8.0
>Reporter: John Zhuge
>Assignee: John Zhuge
> Attachments: HADOOP-14519.001.patch
>
>
> {{Client$Connection#waitForWork}} may suffer spurious wakeup because the
> {{wait}} is not surrounded by a loop. See
> [https://docs.oracle.com/javase/7/docs/api/java/lang/Object.html#wait()].
> {code:title=Client$Connection#waitForWork}
> if (calls.isEmpty() && !shouldCloseConnection.get() && running.get()) {
> long timeout = maxIdleTime-
> (Time.now()-lastActivity.get());
> if (timeout>0) {
> try {
> wait(timeout); << spurious wakeup
> } catch (InterruptedException e) {}
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13327) Add OutputStream + Syncable to the Filesystem Specification
[
https://issues.apache.org/jira/browse/HADOOP-13327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252042#comment-16252042
]
Hadoop QA commented on HADOOP-13327:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} HADOOP-13327 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-13327 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12882859/HADOOP-13327-002.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13683/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add OutputStream + Syncable to the Filesystem Specification
> ---
>
> Key: HADOOP-13327
> URL: https://issues.apache.org/jira/browse/HADOOP-13327
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-13327-002.patch, HADOOP-13327-branch-2-001.patch
>
>
> Write down what a Filesystem output stream should do. While core the API is
> defined in Java, that doesn't say what's expected about visibility,
> durability, etc —and Hadoop Syncable interface is entirely ours to define.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14396) Add builder interface to FileContext
[
https://issues.apache.org/jira/browse/HADOOP-14396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HADOOP-14396:
-
Target Version/s: 3.1.0, 2.10.0 (was: 3.0.0, 2.10.0)
> Add builder interface to FileContext
>
>
> Key: HADOOP-14396
> URL: https://issues.apache.org/jira/browse/HADOOP-14396
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Affects Versions: 2.9.0, 3.0.0-alpha3
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Attachments: HADOOP-14396.00.patch
>
>
> Add builder interface for {{FileContext#create}} and {{FileContext#append}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14996) wasb: ReadFully occasionally fails when using page blobs
[
https://issues.apache.org/jira/browse/HADOOP-14996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HADOOP-14996:
-
Target Version/s: 3.1.0, 2.10.0 (was: 3.0.0, 2.10.0)
> wasb: ReadFully occasionally fails when using page blobs
>
>
> Key: HADOOP-14996
> URL: https://issues.apache.org/jira/browse/HADOOP-14996
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Reporter: Thomas Marquardt
>Assignee: Thomas Marquardt
>
> Looks like there is a functional bug or concurrency bug in the
> PageBlobInputStream implemenation of ReadFully.
> 1) Use 1 mapper to copy results in success:
> hadoop distcp -m 1
> wasb://hbt-lifet...@salsbx01sparkdata.blob.core.windows.net/hive_tables
> wasb://hbt-lifetime-...@supporttestl2.blob.core.windows.net/hdi_backup
> 2) Turn on DEBUG log by setting mapreduce.map.log.level=DEBUG in ambari. Then
> run with more than 1 mapper:
> Saw debug log like this:
> {code}
> 2017-10-27 06:18:53,545 DEBUG [main]
> org.apache.hadoop.fs.azure.NativeAzureFileSystem: Seek to position 136251.
> Bytes skipped 136210
> 2017-10-27 06:18:53,549 DEBUG [main]
> org.apache.hadoop.fs.azure.AzureNativeFileSystemStore: Closing page blob
> output stream.
> 2017-10-27 06:18:53,549 DEBUG [main]
> org.apache.hadoop.fs.azure.AzureNativeFileSystemStore:
> java.util.concurrent.ThreadPoolExecutor@73dce0e6[Terminated, pool size = 0,
> active threads = 0, queued tasks = 0, completed tasks = 0]
> 2017-10-27 06:18:53,549 DEBUG [main]
> org.apache.hadoop.security.UserGroupInformation: PrivilegedActionException
> as:mssupport (auth:SIMPLE) cause:java.io.EOFException
> 2017-10-27 06:18:53,553 WARN [main] org.apache.hadoop.mapred.YarnChild:
> Exception running child : java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:197)
> at java.io.DataInputStream.readFully(DataInputStream.java:169)
> at org.apache.hadoop.io.SequenceFile$Reader.sync(SequenceFile.java:2693)
> at
> org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordReader.initialize(SequenceFileRecordReader.java:58)
> at
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:548)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:786)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14157) FsUrlStreamHandlerFactory "Illegal character in path" parsing file URL on Windows
[
https://issues.apache.org/jira/browse/HADOOP-14157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HADOOP-14157:
-
Target Version/s: 2.6.5, 2.7.3, 3.1.0 (was: 2.7.3, 2.6.5, 3.0.0)
> FsUrlStreamHandlerFactory "Illegal character in path" parsing file URL on
> Windows
> -
>
> Key: HADOOP-14157
> URL: https://issues.apache.org/jira/browse/HADOOP-14157
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.7.3, 2.6.5, 3.0.0-alpha2
> Environment: Windows
>Reporter: Simon Scott
>Priority: Minor
> Attachments: HADOOP-14157.001.patch
>
>
> After registering the FsUrlStreamHandlerFactory with the JVM, subsequent
> calls to convert a "file" URL to a URI can fail with "Illegal character in
> path" where the illegal character is a backslash.
> For example:
> {code}
> URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory());
> File file = new File("C:/Users");
> final URL url = new URL("file:///" + file.getAbsolutePath());
> {code}
> gives stack trace:
> {noformat}
> java.net.URISyntaxException: Illegal character in path at index 8:
> file:/C:\Users
> at java.net.URI$Parser.fail(URI.java:2848)
> at java.net.URI$Parser.checkChars(URI.java:3021)
> at java.net.URI$Parser.parseHierarchical(URI.java:3105)
> at java.net.URI$Parser.parse(URI.java:3053)
> at java.net.URI.(URI.java:588)
> at java.net.URL.toURI(URL.java:946)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-10584) ActiveStandbyElector goes down if ZK quorum become unavailable
[ https://issues.apache.org/jira/browse/HADOOP-10584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HADOOP-10584: - Target Version/s: 3.1.0, 2.9.1 (was: 2.9.0, 3.0.0) > ActiveStandbyElector goes down if ZK quorum become unavailable > -- > > Key: HADOOP-10584 > URL: https://issues.apache.org/jira/browse/HADOOP-10584 > Project: Hadoop Common > Issue Type: Bug > Components: ha >Affects Versions: 2.4.0 >Reporter: Karthik Kambatla >Assignee: Daniel Templeton > Attachments: HADOOP-10584.prelim.patch, hadoop-10584-prelim.patch, > rm.log > > > ActiveStandbyElector retries operations for a few times. If the ZK quorum > itself is down, it goes down and the daemons will have to be brought up > again. > Instead, it should log the fact that it is unable to talk to ZK, call > becomeStandby on its client, and continue to attempt connecting to ZK. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13514) Upgrade maven surefire plugin to 2.19.1
[
https://issues.apache.org/jira/browse/HADOOP-13514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HADOOP-13514:
-
Target Version/s: 3.1.0, 2.10.0 (was: 3.0.0, 2.10.0)
> Upgrade maven surefire plugin to 2.19.1
> ---
>
> Key: HADOOP-13514
> URL: https://issues.apache.org/jira/browse/HADOOP-13514
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
>Affects Versions: 2.8.0
>Reporter: Ewan Higgs
>Assignee: Akira Ajisaka
> Attachments: HADOOP-13514-addendum.01.patch,
> HADOOP-13514-testing.001.patch, HADOOP-13514-testing.002.patch,
> HADOOP-13514-testing.003.patch, HADOOP-13514-testing.004.patch,
> HADOOP-13514-testing.005.patch, HADOOP-13514.002.patch,
> HADOOP-13514.003.patch, HADOOP-13514.004.patch, HADOOP-13514.005.patch,
> HADOOP-13514.006.patch, surefire-2.19.patch
>
>
> A lot of people working on Hadoop don't want to run all the tests when they
> develop; only the bits they're working on. Surefire 2.19 introduced more
> useful test filters which let us run a subset of the tests that brings the
> build time down from 'come back tomorrow' to 'grab a coffee'.
> For instance, if I only care about the S3 adaptor, I might run:
> {code}
> mvn test -Dmaven.javadoc.skip=true -Pdist,native -Djava.awt.headless=true
> \"-Dtest=org.apache.hadoop.fs.*, org.apache.hadoop.hdfs.*,
> org.apache.hadoop.fs.s3a.*\"
> {code}
> We can work around this by specifying the surefire version on the command
> line but it would be better, imo, to just update the default surefire used.
> {code}
> mvn test -Dmaven.javadoc.skip=true -Pdist,native -Djava.awt.headless=true
> \"-Dtest=org.apache.hadoop.fs.*, org.apache.hadoop.hdfs.*,
> org.apache.hadoop.fs.s3a.*\" -Dmaven-surefire-plugin.version=2.19.1
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14731) Update gitignore to exclude output of site build
[
https://issues.apache.org/jira/browse/HADOOP-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252033#comment-16252033
]
Hadoop QA commented on HADOOP-14731:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s{color}
| {color:red} HADOOP-14731 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-14731 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12880312/HADOOP-14731.001.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13680/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Update gitignore to exclude output of site build
>
>
> Key: HADOOP-14731
> URL: https://issues.apache.org/jira/browse/HADOOP-14731
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build, site
>Affects Versions: 3.0.0-alpha3
>Reporter: Andrew Wang
>Assignee: Andrew Wang
> Attachments: HADOOP-14731.001.patch
>
>
> Site build generates a bunch of files that aren't caught by gitignore, let's
> update.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14507) extend per-bucket secret key config with explicit getPassword() on fs.s3a.$bucket.secret,key
[
https://issues.apache.org/jira/browse/HADOOP-14507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252008#comment-16252008
]
Hadoop QA commented on HADOOP-14507:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 23s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} hadoop-tools/hadoop-aws: The patch generated 0 new +
6 unchanged - 2 fixed = 6 total (was 8) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 3s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
41s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 43m 36s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-14507 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12897604/HADOOP-14507-004.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 93d43c9880f5 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 18621af |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13676/testReport/ |
| Max. process+thread count | 300 (vs. ulimit of 5000) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13676/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> extend per-bucket secret key config with explicit getPassword() on
> fs.s3a.$bucket.secret,key
>
[jira] [Updated] (HADOOP-14507) extend per-bucket secret key config with explicit getPassword() on fs.s3a.$bucket.secret,key
[
https://issues.apache.org/jira/browse/HADOOP-14507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14507:
Attachment: HADOOP-14507-004.patch
tested: s3 ireland, ddb auth mode
> extend per-bucket secret key config with explicit getPassword() on
> fs.s3a.$bucket.secret,key
>
>
> Key: HADOOP-14507
> URL: https://issues.apache.org/jira/browse/HADOOP-14507
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-14507-001.patch, HADOOP-14507-002.patch,
> HADOOP-14507-003.patch, HADOOP-14507-004.patch
>
>
> Per-bucket jceks support turns out to be complex as you have to manage
> multiple jecks files & configure the client to ask for the right one. This is
> because we're calling {{Configuration.getPassword{"fs,s3a.secret.key"}}.
> If before that, we do a check for the explict id, key, session key in the
> properties {{fs.s3a.$bucket.secret}} ( & c), we could have a single JCEKs
> file with all the secrets for different bucket. You would only need to
> explicitly point the base config to the secrets file, and the right
> credentials would be picked up, if set
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14507) extend per-bucket secret key config with explicit getPassword() on fs.s3a.$bucket.secret,key
[
https://issues.apache.org/jira/browse/HADOOP-14507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14507:
Status: Open (was: Patch Available)
> extend per-bucket secret key config with explicit getPassword() on
> fs.s3a.$bucket.secret,key
>
>
> Key: HADOOP-14507
> URL: https://issues.apache.org/jira/browse/HADOOP-14507
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-14507-001.patch, HADOOP-14507-002.patch,
> HADOOP-14507-003.patch, HADOOP-14507-004.patch
>
>
> Per-bucket jceks support turns out to be complex as you have to manage
> multiple jecks files & configure the client to ask for the right one. This is
> because we're calling {{Configuration.getPassword{"fs,s3a.secret.key"}}.
> If before that, we do a check for the explict id, key, session key in the
> properties {{fs.s3a.$bucket.secret}} ( & c), we could have a single JCEKs
> file with all the secrets for different bucket. You would only need to
> explicitly point the base config to the secrets file, and the right
> credentials would be picked up, if set
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14507) extend per-bucket secret key config with explicit getPassword() on fs.s3a.$bucket.secret,key
[
https://issues.apache.org/jira/browse/HADOOP-14507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14507:
Status: Patch Available (was: Open)
> extend per-bucket secret key config with explicit getPassword() on
> fs.s3a.$bucket.secret,key
>
>
> Key: HADOOP-14507
> URL: https://issues.apache.org/jira/browse/HADOOP-14507
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-14507-001.patch, HADOOP-14507-002.patch,
> HADOOP-14507-003.patch, HADOOP-14507-004.patch
>
>
> Per-bucket jceks support turns out to be complex as you have to manage
> multiple jecks files & configure the client to ask for the right one. This is
> because we're calling {{Configuration.getPassword{"fs,s3a.secret.key"}}.
> If before that, we do a check for the explict id, key, session key in the
> properties {{fs.s3a.$bucket.secret}} ( & c), we could have a single JCEKs
> file with all the secrets for different bucket. You would only need to
> explicitly point the base config to the secrets file, and the right
> credentials would be picked up, if set
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14104) Client should always ask namenode for kms provider path.
[
https://issues.apache.org/jira/browse/HADOOP-14104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251874#comment-16251874
]
Xiao Chen commented on HADOOP-14104:
Right, 2 separate clusters, running with the same {{fs.defaultFS}}, e.g.
{{hdfs://nameservice1}}. That creative application is the only one that does
cross cluster work.
> Client should always ask namenode for kms provider path.
>
>
> Key: HADOOP-14104
> URL: https://issues.apache.org/jira/browse/HADOOP-14104
> Project: Hadoop Common
> Issue Type: Improvement
> Components: kms
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
> Fix For: 2.9.0, 3.0.0-alpha4, 2.8.2
>
> Attachments: HADOOP-14104-branch-2.8.patch,
> HADOOP-14104-branch-2.patch, HADOOP-14104-trunk-v1.patch,
> HADOOP-14104-trunk-v2.patch, HADOOP-14104-trunk-v3.patch,
> HADOOP-14104-trunk-v4.patch, HADOOP-14104-trunk-v5.patch,
> HADOOP-14104-trunk.patch
>
>
> According to current implementation of kms provider in client conf, there can
> only be one kms.
> In multi-cluster environment, if a client is reading encrypted data from
> multiple clusters it will only get kms token for local cluster.
> Not sure whether the target version is correct or not.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15003) Merge S3A committers into trunk: Yetus patch checker
[ https://issues.apache.org/jira/browse/HADOOP-15003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15003: Status: Open (was: Patch Available) > Merge S3A committers into trunk: Yetus patch checker > > > Key: HADOOP-15003 > URL: https://issues.apache.org/jira/browse/HADOOP-15003 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13786-041.patch, HADOOP-13786-042.patch, > HADOOP-13786-043.patch, HADOOP-13786-044.patch, HADOOP-13786-045.patch, > HADOOP-13786-046.patch, HADOOP-13786-047.patch, HADOOP-13786-048.patch, > HADOOP-13786-049.patch > > > This is a Yetus only JIRA created to have Yetus review the > HADOOP-13786/HADOOP-14971 patch as a .patch file, as the review PR > [https://github.com/apache/hadoop/pull/282] is stopping this happening in > HADOOP-14971. > Reviews should go into the PR/other task -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15003) Merge S3A committers into trunk: Yetus patch checker
[ https://issues.apache.org/jira/browse/HADOOP-15003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15003: Attachment: HADOOP-13786-049.patch Patch 049; this is just 048 with those addressable bits of checkstyle addressed. we appear to have crossed the 1MB of source milestone. oops > Merge S3A committers into trunk: Yetus patch checker > > > Key: HADOOP-15003 > URL: https://issues.apache.org/jira/browse/HADOOP-15003 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13786-041.patch, HADOOP-13786-042.patch, > HADOOP-13786-043.patch, HADOOP-13786-044.patch, HADOOP-13786-045.patch, > HADOOP-13786-046.patch, HADOOP-13786-047.patch, HADOOP-13786-048.patch, > HADOOP-13786-049.patch > > > This is a Yetus only JIRA created to have Yetus review the > HADOOP-13786/HADOOP-14971 patch as a .patch file, as the review PR > [https://github.com/apache/hadoop/pull/282] is stopping this happening in > HADOOP-14971. > Reviews should go into the PR/other task -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15003) Merge S3A committers into trunk: Yetus patch checker
[ https://issues.apache.org/jira/browse/HADOOP-15003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15003: Status: Patch Available (was: Open) > Merge S3A committers into trunk: Yetus patch checker > > > Key: HADOOP-15003 > URL: https://issues.apache.org/jira/browse/HADOOP-15003 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13786-041.patch, HADOOP-13786-042.patch, > HADOOP-13786-043.patch, HADOOP-13786-044.patch, HADOOP-13786-045.patch, > HADOOP-13786-046.patch, HADOOP-13786-047.patch, HADOOP-13786-048.patch, > HADOOP-13786-049.patch > > > This is a Yetus only JIRA created to have Yetus review the > HADOOP-13786/HADOOP-14971 patch as a .patch file, as the review PR > [https://github.com/apache/hadoop/pull/282] is stopping this happening in > HADOOP-14971. > Reviews should go into the PR/other task -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15027) Improvements for Hadoop read from AliyunOSS
[
https://issues.apache.org/jira/browse/HADOOP-15027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251109#comment-16251109
]
wujinhu edited comment on HADOOP-15027 at 11/14/17 4:27 PM:
[~uncleGen] [~ste...@apache.org]
I have set up hadoop env in Tencent cloud server, and read data from Aliyun OSS
via hadoop command
file size: 104MB,
When I tested aggressive random read, I divided this file into multipart(each
part is 1MB) and shuffle all the parts to read.
{code:java}
public static final String MULTIPART_DOWNLOAD_SIZE_KEY =
"fs.oss.multipart.download.size";
public static final long MULTIPART_DOWNLOAD_SIZE_DEFAULT = 1 * 1024 * 1024;
{code}
Current version
ubuntu@VM-0-11-ubuntu:~/hadoop-3.0.0-beta1$ date; hadoop fs -cat
oss://hadoop-on-oss/hadoop/wordcount/wordcount_bigdata_100M.txt > /dev/null;
date
Tue Nov 14 15:47:27 CST 2017
Tue Nov 14 15:48:47 CST 2017
80s
After optimized by multi-thread prefetch
---
fs.oss.multipart.download.ahead.part.max.number = 1
fs.oss.multipart.download.threads = 4
sequential IO: 77s
random IO: 74.51s
---
fs.oss.multipart.download.ahead.part.max.number = 8
fs.oss.multipart.download.threads = 4
sequential IO: 23s
random IO: 59.128
---
fs.oss.multipart.download.ahead.part.max.number = 16
fs.oss.multipart.download.threads = 4
sequential IO: 19s
random IO: 83.92
---
fs.oss.multipart.download.ahead.part.max.number = 16
fs.oss.multipart.download.threads = 8
sequential IO: 13s(continue to test this, already reach the upper limit of the
network)
random IO: 50.561
Summaries:
* if read ahead part number = 1, sequential IO and random IO are the same since
there is no pre-fetch
* sequential IO will be better if read ahead part number and thread number
increases
* worst case of random IO will be something like [sequential ... sequential,
random, sequential ... sequential, random.] so that some pre-fetch data
will be wasted.
was (Author: wujinhu):
[~uncleGen] [~ste...@apache.org]
I have set up hadoop env in Tencent cloud server, and read data from Aliyun OSS
via hadoop command
file size: 104MB,
When I tested aggressive random read, I divided this file into multipart(each
part is 1MB) and shuffle all the parts to read.
{code:java}
public static final String MULTIPART_DOWNLOAD_SIZE_KEY =
"fs.oss.multipart.download.size";
public static final long MULTIPART_DOWNLOAD_SIZE_DEFAULT = 1 * 1024 * 1024;
{code}
Current version
ubuntu@VM-0-11-ubuntu:~/hadoop-3.0.0-beta1$ date; hadoop fs -cat
oss://hadoop-on-oss/hadoop/wordcount/wordcount_bigdata_100M.txt > /dev/null;
date
Tue Nov 14 15:47:27 CST 2017
Tue Nov 14 15:48:47 CST 2017
80s
After optimized by multi-thread prefetch
---
fs.oss.multipart.download.ahead.part.max.number = 1
fs.oss.multipart.download.threads = 4
sequential IO: 77s
random IO: 74.51s
---
fs.oss.multipart.download.ahead.part.max.number = 8
fs.oss.multipart.download.threads = 4
sequential IO: 23s
random IO: 59.128
---
fs.oss.multipart.download.ahead.part.max.number = 16
fs.oss.multipart.download.threads = 4
sequential IO: 19s
random IO: 83.92
---
fs.oss.multipart.download.ahead.part.max.number = 16
fs.oss.multipart.download.threads = 8
sequential IO: 13s
random IO: 50.561
Summaries:
* if read ahead part number = 1, sequential IO and random IO are the same since
there is no pre-fetch
* sequential IO will be better if read ahead part number and thread number
increases
* worst case of random IO will be something like [sequential ... sequential,
random, sequential ... sequential, random.] so that some pre-fetch data
will be wasted.
> Improvements for Hadoop read from AliyunOSS
> ---
>
> Key: HADOOP-15027
> URL: https://issues.apache.org/jira/browse/HADOOP-15027
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/oss
>Affects Versions: 3.0.0
>Reporter: wujinhu
>Assignee: wujinhu
> Attachments: HADOOP-15027.001.patch, HADOOP-15027.002.patch,
> HADOOP-15027.003.patch
>
>
> Currently, read performance is poor when Hadoop reads from AliyunOSS. It
> needs about 1min to read 1GB from OSS.
> Class AliyunOSSInputStream uses single thread to read data from AliyunOSS,
> so we can refactor this by using multi-thread pre read to improve this.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15033) Use java.util.zip.CRC32C for Java 9 and above
[ https://issues.apache.org/jira/browse/HADOOP-15033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251631#comment-16251631 ] Dmitry Chuyko commented on HADOOP-15033: It worths mentioning here that java.util.zip pure Java implementation compiled by C2 is ~2x slower for len=512 than Hadoop's pure Java though they basically use the same approach to calculation. It is a subject for improvement in core library for platforms and JVMS that don't provide CRC intrinsics. But again in real life on Hotspot on x86 and aarch64 you always get an intrinsic implementation. > Use java.util.zip.CRC32C for Java 9 and above > - > > Key: HADOOP-15033 > URL: https://issues.apache.org/jira/browse/HADOOP-15033 > Project: Hadoop Common > Issue Type: Improvement > Components: performance, util >Affects Versions: 3.0.0 >Reporter: Dmitry Chuyko > > java.util.zip.CRC32C implementation is available since Java 9. > https://docs.oracle.com/javase/9/docs/api/java/util/zip/CRC32C.html > Platform specific assembler intrinsics make it more effective than any pure > Java implementation. > Hadoop is compiled against Java 8 but class constructor may be accessible > with method handle on 9 to instances implementing Checksum in runtime. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251577#comment-16251577 ] Sean Mackrory commented on HADOOP-14475: Also some minor checkstyle issues - but if you can upload the correct patch I can fix those before committing when I review. > Metrics of S3A don't print out when enable it in Hadoop metrics property file > -- > > Key: HADOOP-14475 > URL: https://issues.apache.org/jira/browse/HADOOP-14475 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: uname -a > Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017 > x86_64 x86_64 x86_64 GNU/Linux > cat /etc/issue > Ubuntu 16.04.2 LTS \n \l >Reporter: Yonger >Assignee: Yonger > Attachments: HADOOP-14475-003.patch, HADOOP-14475.002.patch, > HADOOP-14475.005.patch, HADOOP-14475.006.patch, HADOOP-14475.008.patch, > HADOOP-14775.007.patch, failsafe-report-s3a-it.html, > failsafe-report-s3a-scale.html, failsafe-report-scale.html, > failsafe-report-scale.zip, s3a-metrics.patch1, stdout.zip > > > *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink > #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #*.sink.influxdb.url=http:/xx > #*.sink.influxdb.influxdb_port=8086 > #*.sink.influxdb.database=hadoop > #*.sink.influxdb.influxdb_username=hadoop > #*.sink.influxdb.influxdb_password=hadoop > #*.sink.ingluxdb.cluster=c1 > *.period=10 > #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out > I can't find the out put file even i run a MR job which should be used s3. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15033) Use java.util.zip.CRC32C for Java 9 and above
[ https://issues.apache.org/jira/browse/HADOOP-15033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251538#comment-16251538 ] Dmitry Chuyko edited comment on HADOOP-15033 at 11/14/17 3:38 PM: -- [~raviprak] I compared partial checksums over random array and byte buffer. Some more details: * Over 1000 elements (crc of 0 element, crc of 1 element, crc of 2 elements...). * Zero and non-zero offset. * Interpreter, C1, C2 Hotspot compiler. * OpenJDK 10-internal * x86_64 and aarch64 CRC32C values required to be equal are equal. was (Author: dchuyko): [~raviprak] I compared partial checksums over partial random array and byte buffer. Some more details: * Over 1000 elements (crc of 0 element, crc of 1 element...). * Zero and non-zero offset. * Interpreter, C1, C2 Hotspot compiler. * OpenJDK 10-internal * x86_64 and aarch64 CRC32C values required to be equal are equal. > Use java.util.zip.CRC32C for Java 9 and above > - > > Key: HADOOP-15033 > URL: https://issues.apache.org/jira/browse/HADOOP-15033 > Project: Hadoop Common > Issue Type: Improvement > Components: performance, util >Affects Versions: 3.0.0 >Reporter: Dmitry Chuyko > > java.util.zip.CRC32C implementation is available since Java 9. > https://docs.oracle.com/javase/9/docs/api/java/util/zip/CRC32C.html > Platform specific assembler intrinsics make it more effective than any pure > Java implementation. > Hadoop is compiled against Java 8 but class constructor may be accessible > with method handle on 9 to instances implementing Checksum in runtime. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14104) Client should always ask namenode for kms provider path.
[
https://issues.apache.org/jira/browse/HADOOP-14104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251552#comment-16251552
]
Rushabh S Shah commented on HADOOP-14104:
-
bq. We have had a downstream application broken, due to the 'cache the
nameservice to provider mapping into UGI credentials' logic:
The key for the cache is {{"dfs-kms-hdfs://" + namenodeUri.getAuthority()}}. I
am confused about the usage of word {{nameservice}}.
The key differentiator is the {{namenodeUri.getAuthority()}}. Do you mean both
clusters has the same {{fs.defaultFS}} config ?
> Client should always ask namenode for kms provider path.
>
>
> Key: HADOOP-14104
> URL: https://issues.apache.org/jira/browse/HADOOP-14104
> Project: Hadoop Common
> Issue Type: Improvement
> Components: kms
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
> Fix For: 2.9.0, 3.0.0-alpha4, 2.8.2
>
> Attachments: HADOOP-14104-branch-2.8.patch,
> HADOOP-14104-branch-2.patch, HADOOP-14104-trunk-v1.patch,
> HADOOP-14104-trunk-v2.patch, HADOOP-14104-trunk-v3.patch,
> HADOOP-14104-trunk-v4.patch, HADOOP-14104-trunk-v5.patch,
> HADOOP-14104-trunk.patch
>
>
> According to current implementation of kms provider in client conf, there can
> only be one kms.
> In multi-cluster environment, if a client is reading encrypted data from
> multiple clusters it will only get kms token for local cluster.
> Not sure whether the target version is correct or not.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251543#comment-16251543 ] Sean Mackrory commented on HADOOP-14475: Thanks for the update [~iyonger]! Your .008 patch appears to be identical to your .007 patch - I think you uploaded the wrong thing. Sounds like you've addressed all the concerns I raised, and I like the changes you mention in 1. and 3. I'm not entirely sure what you mean in 2., but I'm sure I'll see once you update the patch. I've also looked into other metrics2 usage to address my other questions and I see that they behave the same (in that the metrics system will report metrics from seemingly unrelated conexts). I haven't yet got an answer on the thinking behind the difference between system and context, but I'm satisfied from what I've seen that that is at least intentional and consistent. So I expect that I'll be able to commit your patch very soon if you can the .008 patch and post the one with your updates. > Metrics of S3A don't print out when enable it in Hadoop metrics property file > -- > > Key: HADOOP-14475 > URL: https://issues.apache.org/jira/browse/HADOOP-14475 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: uname -a > Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017 > x86_64 x86_64 x86_64 GNU/Linux > cat /etc/issue > Ubuntu 16.04.2 LTS \n \l >Reporter: Yonger >Assignee: Yonger > Attachments: HADOOP-14475-003.patch, HADOOP-14475.002.patch, > HADOOP-14475.005.patch, HADOOP-14475.006.patch, HADOOP-14475.008.patch, > HADOOP-14775.007.patch, failsafe-report-s3a-it.html, > failsafe-report-s3a-scale.html, failsafe-report-scale.html, > failsafe-report-scale.zip, s3a-metrics.patch1, stdout.zip > > > *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink > #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #*.sink.influxdb.url=http:/xx > #*.sink.influxdb.influxdb_port=8086 > #*.sink.influxdb.database=hadoop > #*.sink.influxdb.influxdb_username=hadoop > #*.sink.influxdb.influxdb_password=hadoop > #*.sink.ingluxdb.cluster=c1 > *.period=10 > #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out > I can't find the out put file even i run a MR job which should be used s3. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15033) Use java.util.zip.CRC32C for Java 9 and above
[ https://issues.apache.org/jira/browse/HADOOP-15033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251538#comment-16251538 ] Dmitry Chuyko commented on HADOOP-15033: [~raviprak] I compared partial checksums over partial random array and byte buffer. Some more details: * Over 1000 elements (crc of 0 element, crc of 1 element...). * Zero and non-zero offset. * Interpreter, C1, C2 Hotspot compiler. * OpenJDK 10-internal * x86_64 and aarch64 CRC32C values required to be equal are equal. > Use java.util.zip.CRC32C for Java 9 and above > - > > Key: HADOOP-15033 > URL: https://issues.apache.org/jira/browse/HADOOP-15033 > Project: Hadoop Common > Issue Type: Improvement > Components: performance, util >Affects Versions: 3.0.0 >Reporter: Dmitry Chuyko > > java.util.zip.CRC32C implementation is available since Java 9. > https://docs.oracle.com/javase/9/docs/api/java/util/zip/CRC32C.html > Platform specific assembler intrinsics make it more effective than any pure > Java implementation. > Hadoop is compiled against Java 8 but class constructor may be accessible > with method handle on 9 to instances implementing Checksum in runtime. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[
https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251540#comment-16251540
]
Hadoop QA commented on HADOOP-14475:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 9m
45s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 30s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 12s{color} | {color:orange} hadoop-tools/hadoop-aws: The patch generated 2
new + 22 unchanged - 0 fixed = 24 total (was 22) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 59s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
40s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 52m 23s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-14475 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12897501/HADOOP-14475.008.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux ff5a32cb3dfc 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 18621af |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13674/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13674/testReport/ |
| Max. process+thread count | 336 (vs. ulimit of 5000) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13674/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This
[jira] [Commented] (HADOOP-14993) AliyunOSS: Override listFiles and listLocatedStatus
[ https://issues.apache.org/jira/browse/HADOOP-14993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251479#comment-16251479 ] Hudson commented on HADOOP-14993: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13236 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13236/]) HADOOP-14993. AliyunOSS: Override listFiles and listLocatedStatus. (kai.zheng: rev 18621af7ae8f8ed703245744f8f2a770d07bbfb9) * (edit) hadoop-tools/hadoop-aliyun/src/main/java/org/apache/hadoop/fs/aliyun/oss/AliyunOSSFileSystem.java * (edit) hadoop-tools/hadoop-aliyun/src/main/java/org/apache/hadoop/fs/aliyun/oss/AliyunOSSFileSystemStore.java * (add) hadoop-tools/hadoop-aliyun/src/main/java/org/apache/hadoop/fs/aliyun/oss/FileStatusAcceptor.java * (edit) hadoop-tools/hadoop-aliyun/src/main/java/org/apache/hadoop/fs/aliyun/oss/AliyunOSSUtils.java * (edit) hadoop-tools/hadoop-aliyun/src/site/markdown/tools/hadoop-aliyun/index.md > AliyunOSS: Override listFiles and listLocatedStatus > > > Key: HADOOP-14993 > URL: https://issues.apache.org/jira/browse/HADOOP-14993 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/oss >Affects Versions: 3.0.0-beta1 >Reporter: Genmao Yu >Assignee: Genmao Yu > Fix For: 3.1.0 > > Attachments: HADOOP-14993.001.patch, HADOOP-14993.002.patch, > HADOOP-14993.003.patch > > > Do a bulk listing off all entries under a path in one single operation, there > is no need to recursively walk the directory tree. > Updates: > - override listFiles and listLocatedStatus by using bulk listing > - some minor updates in hadoop-aliyun index.md -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15003) Merge S3A committers into trunk: Yetus patch checker
[ https://issues.apache.org/jira/browse/HADOOP-15003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251290#comment-16251290 ] Steve Loughran commented on HADOOP-15003: - thanks; just gone through the checkstyle and tweaked the new warnings. Still learning about the checkstyle-approved indentation of multiline lambda-exps. > Merge S3A committers into trunk: Yetus patch checker > > > Key: HADOOP-15003 > URL: https://issues.apache.org/jira/browse/HADOOP-15003 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13786-041.patch, HADOOP-13786-042.patch, > HADOOP-13786-043.patch, HADOOP-13786-044.patch, HADOOP-13786-045.patch, > HADOOP-13786-046.patch, HADOOP-13786-047.patch, HADOOP-13786-048.patch > > > This is a Yetus only JIRA created to have Yetus review the > HADOOP-13786/HADOOP-14971 patch as a .patch file, as the review PR > [https://github.com/apache/hadoop/pull/282] is stopping this happening in > HADOOP-14971. > Reviews should go into the PR/other task -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14936) S3Guard: remove "experimental" from documentation
[ https://issues.apache.org/jira/browse/HADOOP-14936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251267#comment-16251267 ] Steve Loughran commented on HADOOP-14936: - I think we should look through the list of things we consider essential & are still on the phase II list & make them the blockers for this. Or simply: plan for phase II & Hadoop 3.1 being the out-of-experimentation phase; anything which isn't essential for that can be postponed. # I consider HADOOP-15035 to be a blocker # We should be confident that either the DDB format is stable, or that we have an upgrade strategy which works # And we may want to consider auth mode to still be experimental. S3Guard hasn't been in people's hands as part of an ASF release, and while two commercial vendors of hadoop based products have offered S3Guard in some format over the summer, we haven't seen that much in terms of stack traces coming back. I usually view that as the metric of use. Certainly we didn't get anything in the 3.0 beta phase. This is similar in my mind to the S3A client in general: it shipped in 2.6, but it was only once it was out that some of the scale problems surfaced (many files, reading-to-EOF on stream close. HADOOP-11571 shows all that surfaced before we could really declare S3A ready for use. > S3Guard: remove "experimental" from documentation > - > > Key: HADOOP-14936 > URL: https://issues.apache.org/jira/browse/HADOOP-14936 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Aaron Fabbri >Priority: Minor > > I think it is time to remove the "experimental feature" designation in the > site docs for S3Guard. Discuss. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15027) Improvements for Hadoop read from AliyunOSS
[ https://issues.apache.org/jira/browse/HADOOP-15027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251254#comment-16251254 ] Genmao Yu commented on HADOOP-15027: pending on HADOOP-15039, refactor current patch after HADOOP-15039 completes. > Improvements for Hadoop read from AliyunOSS > --- > > Key: HADOOP-15027 > URL: https://issues.apache.org/jira/browse/HADOOP-15027 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/oss >Affects Versions: 3.0.0 >Reporter: wujinhu >Assignee: wujinhu > Attachments: HADOOP-15027.001.patch, HADOOP-15027.002.patch, > HADOOP-15027.003.patch > > > Currently, read performance is poor when Hadoop reads from AliyunOSS. It > needs about 1min to read 1GB from OSS. > Class AliyunOSSInputStream uses single thread to read data from AliyunOSS, > so we can refactor this by using multi-thread pre read to improve this. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15039) move SemaphoredDelegatingExecutor to hadoop-common
Genmao Yu created HADOOP-15039:
--
Summary: move SemaphoredDelegatingExecutor to hadoop-common
Key: HADOOP-15039
URL: https://issues.apache.org/jira/browse/HADOOP-15039
Project: Hadoop Common
Issue Type: Improvement
Components: fs, fs/oss, fs/s3
Affects Versions: 3.0.0-beta1
Reporter: Genmao Yu
Assignee: Genmao Yu
Priority: Minor
Detailed discussions in HADOOP-14999 and HADOOP-15027.
share {{SemaphoredDelegatingExecutor}} and move it to {{hadoop-common}}.
cc [~ste...@apache.org]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14999) AliyunOSS: provide one asynchronous multi-part based uploading mechanism
[
https://issues.apache.org/jira/browse/HADOOP-14999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16250906#comment-16250906
]
Genmao Yu edited comment on HADOOP-14999 at 11/14/17 11:09 AM:
---
pending on refactoring: use {{SemaphoredDelegatingExecutor}} instead of
{{TaskEngine}}.
Just as discussed in HADOOP-15027, I think
{{org.apache.hadoop.fs.s3a.SemaphoredDelegatingExecutor}} is a good common
class, and we may move it to hadoop-common. [~ste...@apache.org] Do you mind if
I open jira to do this work?
was (Author: unclegen):
pending on refactoring: move the {{TaskEngine}} from output stream to oss
filesystem .
Just as discussed in HADOOP-15027, I think
{{org.apache.hadoop.fs.s3a.SemaphoredDelegatingExecutor}} is a good common
class, and we may move it to hadoop-common.
Then, I will refactor the {{TaskEngine}} to use
{{SemaphoredDelegatingExecutor}}.
[~ste...@apache.org] Do you mind if I open jira to do this work?
> AliyunOSS: provide one asynchronous multi-part based uploading mechanism
>
>
> Key: HADOOP-14999
> URL: https://issues.apache.org/jira/browse/HADOOP-14999
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/oss
>Affects Versions: 3.0.0-beta1
>Reporter: Genmao Yu
>Assignee: Genmao Yu
> Attachments: HADOOP-14999.001.patch, HADOOP-14999.002.patch
>
>
> This mechanism is designed for uploading file in parallel and asynchronously:
> - improve the performance of uploading file to OSS server. Firstly, this
> mechanism splits result to multiple small blocks and upload them in parallel.
> Then, getting result and uploading blocks are asynchronous.
> - avoid buffering too large result into local disk. To cite an extreme
> example, there is a task which will output 100GB or even larger, we may need
> to output this 100GB to local disk and then upload it. Sometimes, it is
> inefficient and limited to disk space.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14993) AliyunOSS: Override listFiles and listLocatedStatus
[ https://issues.apache.org/jira/browse/HADOOP-14993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-14993: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.1.0 Status: Resolved (was: Patch Available) Committed to trunk. Thanks [~uncleGen] for the contribution! > AliyunOSS: Override listFiles and listLocatedStatus > > > Key: HADOOP-14993 > URL: https://issues.apache.org/jira/browse/HADOOP-14993 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/oss >Affects Versions: 3.0.0-beta1 >Reporter: Genmao Yu >Assignee: Genmao Yu > Fix For: 3.1.0 > > Attachments: HADOOP-14993.001.patch, HADOOP-14993.002.patch, > HADOOP-14993.003.patch > > > Do a bulk listing off all entries under a path in one single operation, there > is no need to recursively walk the directory tree. > Updates: > - override listFiles and listLocatedStatus by using bulk listing > - some minor updates in hadoop-aliyun index.md -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14993) AliyunOSS: Override listFiles and listLocatedStatus
[ https://issues.apache.org/jira/browse/HADOOP-14993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251174#comment-16251174 ] Kai Zheng commented on HADOOP-14993: The latest patch LGTM and +1. Will commit it shortly. > AliyunOSS: Override listFiles and listLocatedStatus > > > Key: HADOOP-14993 > URL: https://issues.apache.org/jira/browse/HADOOP-14993 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/oss >Affects Versions: 3.0.0-beta1 >Reporter: Genmao Yu >Assignee: Genmao Yu > Attachments: HADOOP-14993.001.patch, HADOOP-14993.002.patch, > HADOOP-14993.003.patch > > > Do a bulk listing off all entries under a path in one single operation, there > is no need to recursively walk the directory tree. > Updates: > - override listFiles and listLocatedStatus by using bulk listing > - some minor updates in hadoop-aliyun index.md -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15027) Improvements for Hadoop read from AliyunOSS
[
https://issues.apache.org/jira/browse/HADOOP-15027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251109#comment-16251109
]
wujinhu edited comment on HADOOP-15027 at 11/14/17 9:06 AM:
[~uncleGen] [~ste...@apache.org]
I have set up hadoop env in Tencent cloud server, and read data from Aliyun OSS
via hadoop command
file size: 104MB,
When I tested aggressive random read, I divided this file into multipart(each
part is 1MB) and shuffle all the parts to read.
{code:java}
public static final String MULTIPART_DOWNLOAD_SIZE_KEY =
"fs.oss.multipart.download.size";
public static final long MULTIPART_DOWNLOAD_SIZE_DEFAULT = 1 * 1024 * 1024;
{code}
Current version
ubuntu@VM-0-11-ubuntu:~/hadoop-3.0.0-beta1$ date; hadoop fs -cat
oss://hadoop-on-oss/hadoop/wordcount/wordcount_bigdata_100M.txt > /dev/null;
date
Tue Nov 14 15:47:27 CST 2017
Tue Nov 14 15:48:47 CST 2017
80s
After optimized by multi-thread prefetch
---
fs.oss.multipart.download.ahead.part.max.number = 1
fs.oss.multipart.download.threads = 4
sequential IO: 77s
random IO: 74.51s
---
fs.oss.multipart.download.ahead.part.max.number = 8
fs.oss.multipart.download.threads = 4
sequential IO: 23s
random IO: 59.128
---
fs.oss.multipart.download.ahead.part.max.number = 16
fs.oss.multipart.download.threads = 4
sequential IO: 19s
random IO: 83.92
---
fs.oss.multipart.download.ahead.part.max.number = 16
fs.oss.multipart.download.threads = 8
sequential IO: 13s
random IO: 50.561
Summaries:
* if read ahead part number = 1, sequential IO and random IO are the same since
there is no pre-fetch
* sequential IO will be better if read ahead part number and thread number
increases
* worst case of random IO will be something like [sequential ... sequential,
random, sequential ... sequential, random.] so that some pre-fetch data
will be wasted.
was (Author: wujinhu):
[~uncleGen]
I have set up hadoop env in Tencent cloud server, and read data from Aliyun OSS
via hadoop command
file size: 104MB,
When I tested aggressive random read, I divided this file into multipart(each
part is 1MB) and shuffle all the parts to read.
{code:java}
public static final String MULTIPART_DOWNLOAD_SIZE_KEY =
"fs.oss.multipart.download.size";
public static final long MULTIPART_DOWNLOAD_SIZE_DEFAULT = 1 * 1024 * 1024;
{code}
Current version
ubuntu@VM-0-11-ubuntu:~/hadoop-3.0.0-beta1$ date; hadoop fs -cat
oss://hadoop-on-oss/hadoop/wordcount/wordcount_bigdata_100M.txt > /dev/null;
date
Tue Nov 14 15:47:27 CST 2017
Tue Nov 14 15:48:47 CST 2017
80s
After optimized by multi-thread prefetch
---
fs.oss.multipart.download.ahead.part.max.number = 1
fs.oss.multipart.download.threads = 4
sequential IO: 77s
random IO: 74.51s
---
fs.oss.multipart.download.ahead.part.max.number = 8
fs.oss.multipart.download.threads = 4
sequential IO: 23s
random IO: 59.128
---
fs.oss.multipart.download.ahead.part.max.number = 16
fs.oss.multipart.download.threads = 4
sequential IO: 19s
random IO: 83.92
---
fs.oss.multipart.download.ahead.part.max.number = 16
fs.oss.multipart.download.threads = 8
sequential IO: 13s
random IO: 50.561
Summaries:
* if read ahead part number = 1, sequential IO and random IO are the same since
there is no pre-fetch
* sequential IO will be better if read ahead part number and thread number
increases
* worst case of random IO will be something like [sequential ... sequential,
random, sequential ... sequential, random.] so that some pre-fetch data
will be wasted.
> Improvements for Hadoop read from AliyunOSS
> ---
>
> Key: HADOOP-15027
> URL: https://issues.apache.org/jira/browse/HADOOP-15027
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/oss
>Affects Versions: 3.0.0
>Reporter: wujinhu
>Assignee: wujinhu
> Attachments: HADOOP-15027.001.patch, HADOOP-15027.002.patch,
> HADOOP-15027.003.patch
>
>
> Currently, read performance is poor when Hadoop reads from AliyunOSS. It
> needs about 1min to read 1GB from OSS.
> Class AliyunOSSInputStream uses single thread to read data from AliyunOSS,
> so we can refactor this by using multi-thread pre read to improve this.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15027) Improvements for Hadoop read from AliyunOSS
[
https://issues.apache.org/jira/browse/HADOOP-15027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251109#comment-16251109
]
wujinhu commented on HADOOP-15027:
--
[~uncleGen]
I have set up hadoop env in Tencent cloud server, and read data from Aliyun OSS
via hadoop command
file size: 104MB,
When I tested aggressive random read, I divided this file into multipart(each
part is 1MB) and shuffle all the parts to read.
{code:java}
public static final String MULTIPART_DOWNLOAD_SIZE_KEY =
"fs.oss.multipart.download.size";
public static final long MULTIPART_DOWNLOAD_SIZE_DEFAULT = 1 * 1024 * 1024;
{code}
Current version
ubuntu@VM-0-11-ubuntu:~/hadoop-3.0.0-beta1$ date; hadoop fs -cat
oss://hadoop-on-oss/hadoop/wordcount/wordcount_bigdata_100M.txt > /dev/null;
date
Tue Nov 14 15:47:27 CST 2017
Tue Nov 14 15:48:47 CST 2017
80s
After optimized by multi-thread prefetch
---
fs.oss.multipart.download.ahead.part.max.number = 1
fs.oss.multipart.download.threads = 4
sequential IO: 77s
random IO: 74.51s
---
fs.oss.multipart.download.ahead.part.max.number = 8
fs.oss.multipart.download.threads = 4
sequential IO: 23s
random IO: 59.128
---
fs.oss.multipart.download.ahead.part.max.number = 16
fs.oss.multipart.download.threads = 4
sequential IO: 19s
random IO: 83.92
---
fs.oss.multipart.download.ahead.part.max.number = 16
fs.oss.multipart.download.threads = 8
sequential IO: 13s
random IO: 50.561
Summaries:
* if read ahead part number = 1, sequential IO and random IO are the same since
there is no pre-fetch
* sequential IO will be better if read ahead part number and thread number
increases
* worst case of random IO will be something like [sequential ... sequential,
random, sequential ... sequential, random.] so that some pre-fetch data
will be wasted.
> Improvements for Hadoop read from AliyunOSS
> ---
>
> Key: HADOOP-15027
> URL: https://issues.apache.org/jira/browse/HADOOP-15027
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/oss
>Affects Versions: 3.0.0
>Reporter: wujinhu
>Assignee: wujinhu
> Attachments: HADOOP-15027.001.patch, HADOOP-15027.002.patch,
> HADOOP-15027.003.patch
>
>
> Currently, read performance is poor when Hadoop reads from AliyunOSS. It
> needs about 1min to read 1GB from OSS.
> Class AliyunOSSInputStream uses single thread to read data from AliyunOSS,
> so we can refactor this by using multi-thread pre read to improve this.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15038) Abstract MetadataStore in S3Guard into a common module.
[
https://issues.apache.org/jira/browse/HADOOP-15038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Genmao Yu updated HADOOP-15038:
---
Description:
Open this JIRA to discuss if we should move {{MetadataStore}} in {{S3Guard}}
into a common module.
Based on this work, other filesystem or object store can implement their own
metastore for optimization (known issues like consistency problem and metadata
operation performance). [~ste...@apache.org] and other guys have done many base
and great works in {{S3Guard}}. It is very helpful to start work. I did some
perf test in HADOOP-14098, and started related work for Aliyun OSS. Indeed
there are still works to do for {{S3Guard}}, like metadata cache inconsistent
with S3 and so on. It also will be a problem for other object store. However,
we can do these works in parallel.
[~ste...@apache.org] [~fabbri] [~drankye] Any suggestion is appreciated.
was:
Open this JIRA to discuss if we should move {{MetadataStore}} in {{S3Guard}}
into a common module.
Based on this work, other filesystem or object store can implement their own
metastore for optimization (known issues like consistency problem and metadata
operation performance). [~ste...@apache.org] and other guys have done many base
and great works in {{S3Guard}}. It is very helpful to start work. I did some
perf test in HADOOP-14098, and started related work for Aliyun OSS. Indeed
there are still works to do for {{S3Guard}}, like metadata cache inconsistent
with S3 and so on. It also will be a problem for other object store. However,
we can do these works in parallel.
cc [~drankye]
> Abstract MetadataStore in S3Guard into a common module.
> ---
>
> Key: HADOOP-15038
> URL: https://issues.apache.org/jira/browse/HADOOP-15038
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs
>Affects Versions: 3.0.0-beta1
>Reporter: Genmao Yu
>
> Open this JIRA to discuss if we should move {{MetadataStore}} in {{S3Guard}}
> into a common module.
> Based on this work, other filesystem or object store can implement their own
> metastore for optimization (known issues like consistency problem and
> metadata operation performance). [~ste...@apache.org] and other guys have
> done many base and great works in {{S3Guard}}. It is very helpful to start
> work. I did some perf test in HADOOP-14098, and started related work for
> Aliyun OSS. Indeed there are still works to do for {{S3Guard}}, like
> metadata cache inconsistent with S3 and so on. It also will be a problem for
> other object store. However, we can do these works in parallel.
> [~ste...@apache.org] [~fabbri] [~drankye] Any suggestion is appreciated.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
