[jira] [Commented] (HADOOP-14188) Remove the usage of org.mockito.internal.util.reflection.Whitebox
[
https://issues.apache.org/jira/browse/HADOOP-14188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15939858#comment-15939858
]
Tsuyoshi Ozawa commented on HADOOP-14188:
-
{quote}
Getter/setter method can be used instead of this hack.
{quote}
Personally, I don't prefer to add setter/getter to actual code, (I mean not
test code), to upgrade a test tool since this solution breaks encapsulation.
Instead of that, how about porting Whitebox class over to Hadoop common as a
utility tool? The choice is worth thinking, since your patch just proved that
the Whitebox class is useful and widely used. Fortunately, mockito is
distributed under MIT license.
> Remove the usage of org.mockito.internal.util.reflection.Whitebox
> -
>
> Key: HADOOP-14188
> URL: https://issues.apache.org/jira/browse/HADOOP-14188
> Project: Hadoop Common
> Issue Type: Test
> Components: test
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Minor
> Attachments: HADOOP-14188.01.patch, HADOOP-14188.02.patch
>
>
> org.mockito.internal.util.reflection.Whitebox was removed in Mockito 2.1, so
> we need to remove the usage to upgrade Mockito. Getter/setter method can be
> used instead of this hack.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13973) S3A GET/HEAD requests failing: java.lang.IllegalStateException: Connection is not open/Connection pool shut down
[ https://issues.apache.org/jira/browse/HADOOP-13973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinayakumar B updated HADOOP-13973: --- Target Version/s: 2.8.1 Fix Version/s: (was: 2.8.0) > S3A GET/HEAD requests failing: java.lang.IllegalStateException: Connection is > not open/Connection pool shut down > > > Key: HADOOP-13973 > URL: https://issues.apache.org/jira/browse/HADOOP-13973 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: EC2 cluster >Reporter: Rajesh Balamohan >Assignee: Steve Loughran > > S3 requests failing with an error coming from Http client, > "java.lang.IllegalStateException: Connection is not open" > Some online discussion implies that this is related to shared connection pool > shutdown & fixed in http client 4.4+. Hadoop & AWS SDK use v 4.5.2 so the fix > is in, we just need to make sure the pool is being set up right. > There's a problem here of course: it may require moving to a later version of > the AWS SDK, with the consequences on jackson , as seen in HADOOP-13050. > And that's if there is a patched version out there -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14188) Remove the usage of org.mockito.internal.util.reflection.Whitebox
[ https://issues.apache.org/jira/browse/HADOOP-14188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HADOOP-14188: --- Attachment: HADOOP-14188.02.patch 02 patch * Removed trailing whitespaces * Fixed findbugs warnings * Fixed TestFileWithSnapshotFeature * Fixed checkstyle warnings > Remove the usage of org.mockito.internal.util.reflection.Whitebox > - > > Key: HADOOP-14188 > URL: https://issues.apache.org/jira/browse/HADOOP-14188 > Project: Hadoop Common > Issue Type: Test > Components: test >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Minor > Attachments: HADOOP-14188.01.patch, HADOOP-14188.02.patch > > > org.mockito.internal.util.reflection.Whitebox was removed in Mockito 2.1, so > we need to remove the usage to upgrade Mockito. Getter/setter method can be > used instead of this hack. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-10101) Update guava dependency to the latest version
[
https://issues.apache.org/jira/browse/HADOOP-10101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15939795#comment-15939795
]
Hadoop QA commented on HADOOP-10101:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
58s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
36s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
59s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
46s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 13m 46s{color}
| {color:red} root generated 8 new + 776 unchanged - 1 fixed = 784 total (was
777) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
0s{color} | {color:green} root: The patch generated 0 new + 220 unchanged - 1
fixed = 220 total (was 221) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
16s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 12s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
15s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 69m 4s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 40m 40s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}230m 32s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ha.TestZKFailoverController |
| | hadoop.security.TestKDiag |
| |
[jira] [Commented] (HADOOP-14176) distcp reports beyond physical memory limits on 2.X
[
https://issues.apache.org/jira/browse/HADOOP-14176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15939630#comment-15939630
]
Fei Hui commented on HADOOP-14176:
--
[~aw] What is the way you suggest on branch-2 for fix this issue? shoud we
remove {{mapred.job.[map/reduce].memory.mb}} like MAPREDUCE-5653? I think we
should resolve it. If there is a problem using default setting, users can be
troubled.
> distcp reports beyond physical memory limits on 2.X
> ---
>
> Key: HADOOP-14176
> URL: https://issues.apache.org/jira/browse/HADOOP-14176
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools/distcp
>Affects Versions: 2.9.0
>Reporter: Fei Hui
>Assignee: Fei Hui
> Attachments: HADOOP-14176-branch-2.001.patch,
> HADOOP-14176-branch-2.002.patch, HADOOP-14176-branch-2.003.patch,
> HADOOP-14176-branch-2.004.patch
>
>
> When i run distcp, i get some errors as follow
> {quote}
> 17/02/21 15:31:18 INFO mapreduce.Job: Task Id :
> attempt_1487645941615_0037_m_03_0, Status : FAILED
> Container [pid=24661,containerID=container_1487645941615_0037_01_05] is
> running beyond physical memory limits. Current usage: 1.1 GB of 1 GB physical
> memory used; 4.0 GB of 5 GB virtual memory used. Killing container.
> Dump of the process-tree for container_1487645941615_0037_01_05 :
> |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
> |- 24661 24659 24661 24661 (bash) 0 0 108650496 301 /bin/bash -c
> /usr/lib/jvm/java/bin/java -Djava.net.preferIPv4Stack=true
> -Dhadoop.metrics.log.level=WARN -Xmx2120m
> -Djava.io.tmpdir=/mnt/disk4/yarn/usercache/hadoop/appcache/application_1487645941615_0037/container_1487645941615_0037_01_05/tmp
> -Dlog4j.configuration=container-log4j.properties
> -Dyarn.app.container.log.dir=/mnt/disk2/log/hadoop-yarn/containers/application_1487645941615_0037/container_1487645941615_0037_01_05
> -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
> -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 192.168.1.208
> 44048 attempt_1487645941615_0037_m_03_0 5
> 1>/mnt/disk2/log/hadoop-yarn/containers/application_1487645941615_0037/container_1487645941615_0037_01_05/stdout
>
> 2>/mnt/disk2/log/hadoop-yarn/containers/application_1487645941615_0037/container_1487645941615_0037_01_05/stderr
> |- 24665 24661 24661 24661 (java) 1766 336 4235558912 280699
> /usr/lib/jvm/java/bin/java -Djava.net.preferIPv4Stack=true
> -Dhadoop.metrics.log.level=WARN -Xmx2120m
> -Djava.io.tmpdir=/mnt/disk4/yarn/usercache/hadoop/appcache/application_1487645941615_0037/container_1487645941615_0037_01_05/tmp
> -Dlog4j.configuration=container-log4j.properties
> -Dyarn.app.container.log.dir=/mnt/disk2/log/hadoop-yarn/containers/application_1487645941615_0037/container_1487645941615_0037_01_05
> -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
> -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 192.168.1.208
> 44048 attempt_1487645941615_0037_m_03_0 5
> Container killed on request. Exit code is 143
> Container exited with a non-zero exit code 143
> {quote}
> Deep into the code , i find that because distcp configuration covers
> mapred-site.xml
> {code}
>
> mapred.job.map.memory.mb
> 1024
>
>
> mapred.job.reduce.memory.mb
> 1024
>
> {code}
> When mapreduce.map.java.opts and mapreduce.map.memory.mb is setting in
> mapred-default.xml, and the value is larger than setted in
> distcp-default.xml, the error maybe occur.
> we should remove those two configurations in distcp-default.xml
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-10101) Update guava dependency to the latest version
[ https://issues.apache.org/jira/browse/HADOOP-10101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated HADOOP-10101: Status: Patch Available (was: Open) > Update guava dependency to the latest version > - > > Key: HADOOP-10101 > URL: https://issues.apache.org/jira/browse/HADOOP-10101 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.0.0-alpha2 >Reporter: Rakesh R >Assignee: Tsuyoshi Ozawa > Labels: BB2015-05-TBR > Attachments: HADOOP-10101-002.patch, HADOOP-10101-004.patch, > HADOOP-10101-005.patch, HADOOP-10101-006.patch, HADOOP-10101-007.patch, > HADOOP-10101-008.patch, HADOOP-10101-009.patch, HADOOP-10101-009.patch, > HADOOP-10101-010.patch, HADOOP-10101-010.patch, HADOOP-10101-011.patch, > HADOOP-10101.012.patch, HADOOP-10101.013.patch, HADOOP-10101.014.patch, > HADOOP-10101.015.patch, HADOOP-10101.016.patch, HADOOP-10101.017.patch, > HADOOP-10101.patch, HADOOP-10101.patch > > > The existing guava version is 11.0.2 which is quite old. This issue tries to > update the version to as latest version as possible. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14194) Aliyun OSS should not use empty endpoint as default
[
https://issues.apache.org/jira/browse/HADOOP-14194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15939587#comment-15939587
]
Genmao Yu commented on HADOOP-14194:
Good catch! We should expose the real reason. [~xiaobingo] Could you please
give a patch?
> Aliyun OSS should not use empty endpoint as default
> ---
>
> Key: HADOOP-14194
> URL: https://issues.apache.org/jira/browse/HADOOP-14194
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/oss
>Reporter: Mingliang Liu
>Assignee: Xiaobing Zhou
>
> In {{AliyunOSSFileSystemStore::initialize()}}, it retrieves the endPoint and
> using empty string as a default value.
> {code}
> String endPoint = conf.getTrimmed(ENDPOINT_KEY, "");
> {code}
> The plain value without validation is passed to OSSClient. If the endPoint is
> not provided (empty string) or the endPoint is not valid, users will get
> exception from Aliyun OSS sdk with raw exception message like:
> {code}
> java.lang.IllegalArgumentException: java.net.URISyntaxException: Expected
> authority at index 8: https://
> at com.aliyun.oss.OSSClient.toURI(OSSClient.java:359)
> at com.aliyun.oss.OSSClient.setEndpoint(OSSClient.java:313)
> at com.aliyun.oss.OSSClient.(OSSClient.java:297)
> at
> org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystemStore.initialize(AliyunOSSFileSystemStore.java:134)
> at
> org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem.initialize(AliyunOSSFileSystem.java:272)
> at
> org.apache.hadoop.fs.aliyun.oss.AliyunOSSTestUtils.createTestFileSystem(AliyunOSSTestUtils.java:63)
> at
> org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemContract.setUp(TestAliyunOSSFileSystemContract.java:47)
> at junit.framework.TestCase.runBare(TestCase.java:139)
> at junit.framework.TestResult$1.protect(TestResult.java:122)
> at junit.framework.TestResult.runProtected(TestResult.java:142)
> at junit.framework.TestResult.run(TestResult.java:125)
> at junit.framework.TestCase.run(TestCase.java:129)
> at junit.framework.TestSuite.runTest(TestSuite.java:255)
> at junit.framework.TestSuite.run(TestSuite.java:250)
> at
> org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
> at
> com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51)
> at
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:237)
> at
> com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
> Caused by: java.net.URISyntaxException: Expected authority at index 8:
> https://
> at java.net.URI$Parser.fail(URI.java:2848)
> at java.net.URI$Parser.failExpecting(URI.java:2854)
> at java.net.URI$Parser.parseHierarchical(URI.java:3102)
> at java.net.URI$Parser.parse(URI.java:3053)
> at java.net.URI.(URI.java:588)
> at com.aliyun.oss.OSSClient.toURI(OSSClient.java:357)
> {code}
> Let's check endPoint is not null or empty, catch the IllegalArgumentException
> and log it, wrapping the exception with clearer message stating the
> misconfiguration in endpoint or credentials.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-10101) Update guava dependency to the latest version
[ https://issues.apache.org/jira/browse/HADOOP-10101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated HADOOP-10101: Attachment: HADOOP-10101.017.patch Fixing javac warnings. > Update guava dependency to the latest version > - > > Key: HADOOP-10101 > URL: https://issues.apache.org/jira/browse/HADOOP-10101 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.0.0-alpha2 >Reporter: Rakesh R >Assignee: Tsuyoshi Ozawa > Labels: BB2015-05-TBR > Attachments: HADOOP-10101-002.patch, HADOOP-10101-004.patch, > HADOOP-10101-005.patch, HADOOP-10101-006.patch, HADOOP-10101-007.patch, > HADOOP-10101-008.patch, HADOOP-10101-009.patch, HADOOP-10101-009.patch, > HADOOP-10101-010.patch, HADOOP-10101-010.patch, HADOOP-10101-011.patch, > HADOOP-10101.012.patch, HADOOP-10101.013.patch, HADOOP-10101.014.patch, > HADOOP-10101.015.patch, HADOOP-10101.016.patch, HADOOP-10101.017.patch, > HADOOP-10101.patch, HADOOP-10101.patch > > > The existing guava version is 11.0.2 which is quite old. This issue tries to > update the version to as latest version as possible. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14217) Object Storage: support colon in object path
[ https://issues.apache.org/jira/browse/HADOOP-14217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15939579#comment-15939579 ] Genmao Yu commented on HADOOP-14217: Also this doc: http://labs.totango.com/spark-read-file-with-colon/ > Object Storage: support colon in object path > > > Key: HADOOP-14217 > URL: https://issues.apache.org/jira/browse/HADOOP-14217 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs, fs/oss >Reporter: Genmao Yu > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14217) Object Storage: support colon in object path
[ https://issues.apache.org/jira/browse/HADOOP-14217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15939573#comment-15939573 ] Genmao Yu edited comment on HADOOP-14217 at 3/24/17 2:22 AM: - [~drankye] and [~ste...@apache.org] Please take a look at SPARK-20061. I think it is a good case. It is valid to create object whose key name contains colon. We will fail to read them in some scenes. was (Author: unclegen): [~drankye] and [~ste...@apache.org] Please take a look at SPARK-20061. I think it is a good case. It is valid to create object which key contains colon. We will fail to read them in some scenes. > Object Storage: support colon in object path > > > Key: HADOOP-14217 > URL: https://issues.apache.org/jira/browse/HADOOP-14217 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs, fs/oss >Reporter: Genmao Yu > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14217) Object Storage: support colon in object path
[ https://issues.apache.org/jira/browse/HADOOP-14217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15939573#comment-15939573 ] Genmao Yu edited comment on HADOOP-14217 at 3/24/17 2:22 AM: - [~drankye] and [~ste...@apache.org] Please take a look at SPARK-20061. I think it is a good case. It is valid to create object which key contains colon. We will fail to read them in some scenes. was (Author: unclegen): [~drankye] and [~ste...@apache.org] Please take a look at SPARK-20061. I think it is a good case. > Object Storage: support colon in object path > > > Key: HADOOP-14217 > URL: https://issues.apache.org/jira/browse/HADOOP-14217 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs, fs/oss >Reporter: Genmao Yu > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14217) Object Storage: support colon in object path
[ https://issues.apache.org/jira/browse/HADOOP-14217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15939573#comment-15939573 ] Genmao Yu commented on HADOOP-14217: [~drankye] and [~ste...@apache.org] Please take a look at SPARK-20061. I think it is a good case. > Object Storage: support colon in object path > > > Key: HADOOP-14217 > URL: https://issues.apache.org/jira/browse/HADOOP-14217 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs, fs/oss >Reporter: Genmao Yu > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-10101) Update guava dependency to the latest version
[ https://issues.apache.org/jira/browse/HADOOP-10101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15939562#comment-15939562 ] Tsuyoshi Ozawa commented on HADOOP-10101: - [~szetszwo] thanks for taking a look! More taking a look at deeper, javac warnings were caused by deprecated warnings after upgrading Guava, as you mentioned.I will update it soon. Canceling a patch. > Update guava dependency to the latest version > - > > Key: HADOOP-10101 > URL: https://issues.apache.org/jira/browse/HADOOP-10101 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.0.0-alpha2 >Reporter: Rakesh R >Assignee: Tsuyoshi Ozawa > Labels: BB2015-05-TBR > Attachments: HADOOP-10101-002.patch, HADOOP-10101-004.patch, > HADOOP-10101-005.patch, HADOOP-10101-006.patch, HADOOP-10101-007.patch, > HADOOP-10101-008.patch, HADOOP-10101-009.patch, HADOOP-10101-009.patch, > HADOOP-10101-010.patch, HADOOP-10101-010.patch, HADOOP-10101-011.patch, > HADOOP-10101.012.patch, HADOOP-10101.013.patch, HADOOP-10101.014.patch, > HADOOP-10101.015.patch, HADOOP-10101.016.patch, HADOOP-10101.patch, > HADOOP-10101.patch > > > The existing guava version is 11.0.2 which is quite old. This issue tries to > update the version to as latest version as possible. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-10101) Update guava dependency to the latest version
[ https://issues.apache.org/jira/browse/HADOOP-10101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated HADOOP-10101: Status: Open (was: Patch Available) > Update guava dependency to the latest version > - > > Key: HADOOP-10101 > URL: https://issues.apache.org/jira/browse/HADOOP-10101 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.0.0-alpha2 >Reporter: Rakesh R >Assignee: Tsuyoshi Ozawa > Labels: BB2015-05-TBR > Attachments: HADOOP-10101-002.patch, HADOOP-10101-004.patch, > HADOOP-10101-005.patch, HADOOP-10101-006.patch, HADOOP-10101-007.patch, > HADOOP-10101-008.patch, HADOOP-10101-009.patch, HADOOP-10101-009.patch, > HADOOP-10101-010.patch, HADOOP-10101-010.patch, HADOOP-10101-011.patch, > HADOOP-10101.012.patch, HADOOP-10101.013.patch, HADOOP-10101.014.patch, > HADOOP-10101.015.patch, HADOOP-10101.016.patch, HADOOP-10101.patch, > HADOOP-10101.patch > > > The existing guava version is 11.0.2 which is quite old. This issue tries to > update the version to as latest version as possible. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14211) ChRootedFs is too aggressive about enforcing "authorityNeeded"
[
https://issues.apache.org/jira/browse/HADOOP-14211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15939558#comment-15939558
]
Andrew Wang commented on HADOOP-14211:
--
Interesting find here Erik! I notice that both {{ChRootedFs}} and {{FilterFs}}
have this special, and probably both should be updated. Filtering a ViewFs is
also perhaps a more likely real-world usecase.
> ChRootedFs is too aggressive about enforcing "authorityNeeded"
> --
>
> Key: HADOOP-14211
> URL: https://issues.apache.org/jira/browse/HADOOP-14211
> Project: Hadoop Common
> Issue Type: Bug
> Components: viewfs
>Affects Versions: 2.6.0
>Reporter: Erik Krogen
>Assignee: Erik Krogen
> Attachments: HADOOP-14211.000.patch
>
>
> Right now {{ChRootedFs}} passes the following up to the
> {{AbstractFileSystem}} superconstructor:
> {code}
> super(fs.getUri(), fs.getUri().getScheme(),
> fs.getUri().getAuthority() != null, fs.getUriDefaultPort());
> {code}
> This passes a value of {{authorityNeeded==true}} for any {{fs}} which has an
> authority, but this isn't necessarily the case - ViewFS itself is an example
> of this. In fact you will encounter this issue if you try to nest one ViewFS
> within another - I can't think of any reason why you would want to do that
> but there's no reason why you shouldn't be able to and in general ViewFS is
> making an assumption that it then proves invalid by its own behavior. The
> {{authorityNeeded}} check isn't necessary in this case anyway; {{fs}} is
> already an instantiated {{AbstractFileSystem}} which means it has already
> used the same constructor with the value of {{authorityNeeded}} (and
> corresponding validation) that it actually requires.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13377) AliyunOSS: improvements for stabilization
[ https://issues.apache.org/jira/browse/HADOOP-13377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15939545#comment-15939545 ] Kai Zheng commented on HADOOP-13377: Thanks [~liuml07] for the fixes. Looks like the AliyunOSS integration work is stable now and we may consider backport it to branch-2 sometime. > AliyunOSS: improvements for stabilization > - > > Key: HADOOP-13377 > URL: https://issues.apache.org/jira/browse/HADOOP-13377 > Project: Hadoop Common > Issue Type: Improvement > Components: fs, fs/oss >Affects Versions: 3.0.0-alpha2 >Reporter: Genmao Yu >Assignee: Genmao Yu > > This work is to handle some stabilization issues. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14227) S3Guard: ITestS3AConcurrentOps is not cleaning up test data
[
https://issues.apache.org/jira/browse/HADOOP-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15939536#comment-15939536
]
Hadoop QA commented on HADOOP-14227:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
53s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
22s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
29s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
43s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 58s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-14227 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12860262/HADOOP-14227-HADOOP-13345.000.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 911fa2207e11 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HADOOP-13345 / 9521c96 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11903/testReport/ |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11903/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> S3Guard: ITestS3AConcurrentOps is not cleaning up test data
> ---
>
> Key: HADOOP-14227
> URL: https://issues.apache.org/jira/browse/HADOOP-14227
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HADOOP-14227-HADOOP-13345.000.patch
>
>
> After running {{ITestS3AConcurrentOps}}, the test data is not cleanup in
> DynamoDB. There are two reasons:
> #
[jira] [Commented] (HADOOP-14210) Directories are not listed recursively when fs.defaultFs is viewFs
[
https://issues.apache.org/jira/browse/HADOOP-14210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15939527#comment-15939527
]
Andrew Wang commented on HADOOP-14210:
--
I don't have a fully educated opinion on this topic, but I normally look to
Unixy behavior to determine user expectations of HDFS. "ls -R /" lists
everything. When I stat a mount point, it's a directory, not a symlink.
Symlinks are currently not enabled in HDFS, so no downstream apps have ever
encountered a symlink. Again looking at Unix as a guide, applications still
don't handle symlinks correctly from a correctness or security POV after
decades of existence. So, if we can roll out VFS without requiring apps to
understand symlinks, I'd be happy.
A lot of our apps use HDFS APIs like the shell or {{FileSystem#listFiles}}.
We'd want these to work the same way regardless of how the VFS is sharded, for
transparency.
Can we dredge up any of the initial motivations from the VFS design docs or
JIRA?
Given that existing FSs can already span to 100s of millions of files, clients
need to handle high scale no matter what. For the CLI ls command, I'm hoping
we're using the iterator-based listing methods, as well as providing a mode
that doesn't try to align columns (which requires buffering all the output).
> Directories are not listed recursively when fs.defaultFs is viewFs
> --
>
> Key: HADOOP-14210
> URL: https://issues.apache.org/jira/browse/HADOOP-14210
> Project: Hadoop Common
> Issue Type: Bug
> Components: viewfs
>Affects Versions: 2.7.0
>Reporter: Ajith S
> Labels: viewfs
> Attachments: HDFS-8413.patch
>
>
> Mount a cluster on client throught viewFs mount table
> Example:
> {quote}
>
> fs.defaultFS
> viewfs:///
>
>
> fs.viewfs.mounttable.default.link./nn1
> hdfs://ns1/
>
>
> fs.viewfs.mounttable.default.link./user
> hdfs://host-72:8020/
>
>
> {quote}
> Try to list the files recursively *(hdfs dfs -ls -R / or hadoop fs -ls -R /)*
> only the parent folders are listed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14226) S3Guard: ITestDynamoDBMetadataStoreScale is not cleaning up test data
[
https://issues.apache.org/jira/browse/HADOOP-14226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15939488#comment-15939488
]
Hadoop QA commented on HADOOP-14226:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
29s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
36s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
23s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
31s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
38s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 28s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-14226 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12860249/HADOOP-14226-HADOOP-13345.000.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux bbd40d1c3e63 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24
10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HADOOP-13345 / 9521c96 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11902/testReport/ |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11902/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> S3Guard: ITestDynamoDBMetadataStoreScale is not cleaning up test data
> -
>
> Key: HADOOP-14226
> URL: https://issues.apache.org/jira/browse/HADOOP-14226
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: HADOOP-13345
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HADOOP-14226-HADOOP-13345.000.patch
>
>
> After running {{ITestDynamoDBMetadataStoreScale}},
[jira] [Updated] (HADOOP-14227) S3Guard: ITestS3AConcurrentOps is not cleaning up test data
[
https://issues.apache.org/jira/browse/HADOOP-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HADOOP-14227:
---
Attachment: HADOOP-14227-HADOOP-13345.000.patch
> S3Guard: ITestS3AConcurrentOps is not cleaning up test data
> ---
>
> Key: HADOOP-14227
> URL: https://issues.apache.org/jira/browse/HADOOP-14227
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HADOOP-14227-HADOOP-13345.000.patch
>
>
> After running {{ITestS3AConcurrentOps}}, the test data is not cleanup in
> DynamoDB. There are two reasons:
> # The {{ITestS3AConcurrentOps::teardown()}} method is not calling super
> teardown() method to clean up the default test directory.
> # The {{auxFs}} is not S3Guard aware even though the {{fs}} to test is.
> That's because the {{auxFs}} is creating a new Configuration object without
> patching in S3Guard options (via {{maybeEnableS3Guard(conf);}}).
> This JIRA is to clean up the data after test.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14227) S3Guard: ITestS3AConcurrentOps is not cleaning up test data
[
https://issues.apache.org/jira/browse/HADOOP-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HADOOP-14227:
---
Status: Patch Available (was: Open)
> S3Guard: ITestS3AConcurrentOps is not cleaning up test data
> ---
>
> Key: HADOOP-14227
> URL: https://issues.apache.org/jira/browse/HADOOP-14227
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HADOOP-14227-HADOOP-13345.000.patch
>
>
> After running {{ITestS3AConcurrentOps}}, the test data is not cleanup in
> DynamoDB. There are two reasons:
> # The {{ITestS3AConcurrentOps::teardown()}} method is not calling super
> teardown() method to clean up the default test directory.
> # The {{auxFs}} is not S3Guard aware even though the {{fs}} to test is.
> That's because the {{auxFs}} is creating a new Configuration object without
> patching in S3Guard options (via {{maybeEnableS3Guard(conf);}}).
> This JIRA is to clean up the data after test.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14227) S3Guard: ITestS3AConcurrentOps is not cleaning up test data
Mingliang Liu created HADOOP-14227:
--
Summary: S3Guard: ITestS3AConcurrentOps is not cleaning up test
data
Key: HADOOP-14227
URL: https://issues.apache.org/jira/browse/HADOOP-14227
Project: Hadoop Common
Issue Type: Bug
Components: fs/s3
Reporter: Mingliang Liu
Assignee: Mingliang Liu
After running {{ITestS3AConcurrentOps}}, the test data is not cleanup in
DynamoDB. There are two reasons:
# The {{ITestS3AConcurrentOps::teardown()}} method is not calling super
teardown() method to clean up the default test directory.
# The {{auxFs}} is not S3Guard aware even though the {{fs}} to test is. That's
because the {{auxFs}} is creating a new Configuration object without patching
in S3Guard options (via {{maybeEnableS3Guard(conf);}}).
This JIRA is to clean up the data after test.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-10101) Update guava dependency to the latest version
[ https://issues.apache.org/jira/browse/HADOOP-10101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated HADOOP-10101: Attachment: HADOOP-10101.016.patch Fixing checkstyle warning again. > Update guava dependency to the latest version > - > > Key: HADOOP-10101 > URL: https://issues.apache.org/jira/browse/HADOOP-10101 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.0.0-alpha2 >Reporter: Rakesh R >Assignee: Tsuyoshi Ozawa > Labels: BB2015-05-TBR > Attachments: HADOOP-10101-002.patch, HADOOP-10101-004.patch, > HADOOP-10101-005.patch, HADOOP-10101-006.patch, HADOOP-10101-007.patch, > HADOOP-10101-008.patch, HADOOP-10101-009.patch, HADOOP-10101-009.patch, > HADOOP-10101-010.patch, HADOOP-10101-010.patch, HADOOP-10101-011.patch, > HADOOP-10101.012.patch, HADOOP-10101.013.patch, HADOOP-10101.014.patch, > HADOOP-10101.015.patch, HADOOP-10101.016.patch, HADOOP-10101.patch, > HADOOP-10101.patch > > > The existing guava version is 11.0.2 which is quite old. This issue tries to > update the version to as latest version as possible. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14226) S3Guard: ITestDynamoDBMetadataStoreScale is not cleaning up test data
[
https://issues.apache.org/jira/browse/HADOOP-14226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15939462#comment-15939462
]
Mingliang Liu commented on HADOOP-14226:
I manually checked the test and the data was cleaned up successfully w/ this
patch.
> S3Guard: ITestDynamoDBMetadataStoreScale is not cleaning up test data
> -
>
> Key: HADOOP-14226
> URL: https://issues.apache.org/jira/browse/HADOOP-14226
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: HADOOP-13345
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HADOOP-14226-HADOOP-13345.000.patch
>
>
> After running {{ITestDynamoDBMetadataStoreScale}}, the test data is not
> cleaned up. There is a call to {{clearMetadataStore(ms, count);}} in the
> finally clause though. The reason is that, the internally called method
> {{DynamoDBMetadataStore::deleteSubtree()}} is assuming there should be an
> item for the parent dest path:
> {code}
> parent=/fake-bucket, child=moved-here, is_dir=true
> {code}
> In DynamoDBMetadataStore implementation, we assume that _if a path exists,
> all its ancestors will also exist in the table_. We need to pre-create dest
> path to maintain this invariant so that test data can be cleaned up
> successfully.
> I think there may be other tests with the same problem. Let's
> identify/address them separately.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14226) S3Guard: ITestDynamoDBMetadataStoreScale is not cleaning up test data
[
https://issues.apache.org/jira/browse/HADOOP-14226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HADOOP-14226:
---
Status: Patch Available (was: Open)
Ping [~ste...@apache.org] and [~rajesh.balamohan]. This may have something to
do with failing deletion in Hive.
> S3Guard: ITestDynamoDBMetadataStoreScale is not cleaning up test data
> -
>
> Key: HADOOP-14226
> URL: https://issues.apache.org/jira/browse/HADOOP-14226
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: HADOOP-13345
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HADOOP-14226-HADOOP-13345.000.patch
>
>
> After running {{ITestDynamoDBMetadataStoreScale}}, the test data is not
> cleaned up. There is a call to {{clearMetadataStore(ms, count);}} in the
> finally clause though. The reason is that, the internally called method
> {{DynamoDBMetadataStore::deleteSubtree()}} is assuming there should be an
> item for the parent dest path:
> {code}
> parent=/fake-bucket, child=moved-here, is_dir=true
> {code}
> In DynamoDBMetadataStore implementation, we assume that _if a path exists,
> all its ancestors will also exist in the table_. We need to pre-create dest
> path to maintain this invariant so that test data can be cleaned up
> successfully.
> I think there may be other tests with the same problem. Let's
> identify/address them separately.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14226) S3Guard: ITestDynamoDBMetadataStoreScale is not cleaning up test data
[
https://issues.apache.org/jira/browse/HADOOP-14226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HADOOP-14226:
---
Attachment: HADOOP-14226-HADOOP-13345.000.patch
> S3Guard: ITestDynamoDBMetadataStoreScale is not cleaning up test data
> -
>
> Key: HADOOP-14226
> URL: https://issues.apache.org/jira/browse/HADOOP-14226
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: HADOOP-13345
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HADOOP-14226-HADOOP-13345.000.patch
>
>
> After running {{ITestDynamoDBMetadataStoreScale}}, the test data is not
> cleaned up. There is a call to {{clearMetadataStore(ms, count);}} in the
> finally clause though. The reason is that, the internally called method
> {{DynamoDBMetadataStore::deleteSubtree()}} is assuming there should be an
> item for the parent dest path:
> {code}
> parent=/fake-bucket, child=moved-here, is_dir=true
> {code}
> In DynamoDBMetadataStore implementation, we assume that _if a path exists,
> all its ancestors will also exist in the table_. We need to pre-create dest
> path to maintain this invariant so that test data can be cleaned up
> successfully.
> I think there may be other tests with the same problem. Let's
> identify/address them separately.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14226) S3Guard: ITestDynamoDBMetadataStoreScale is not cleaning up test data
Mingliang Liu created HADOOP-14226:
--
Summary: S3Guard: ITestDynamoDBMetadataStoreScale is not cleaning
up test data
Key: HADOOP-14226
URL: https://issues.apache.org/jira/browse/HADOOP-14226
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: HADOOP-13345
Reporter: Mingliang Liu
Assignee: Mingliang Liu
After running {{ITestDynamoDBMetadataStoreScale}}, the test data is not cleaned
up. There is a call to {{clearMetadataStore(ms, count);}} in the finally clause
though. The reason is that, the internally called method
{{DynamoDBMetadataStore::deleteSubtree()}} is assuming there should be an item
for the parent dest path:
{code}
parent=/fake-bucket, child=moved-here, is_dir=true
{code}
In DynamoDBMetadataStore implementation, we assume that _if a path exists, all
its ancestors will also exist in the table_. We need to pre-create dest path to
maintain this invariant so that test data can be cleaned up successfully.
I think there may be other tests with the same problem. Let's identify/address
them separately.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13715) Add isErasureCoded() API to FileStatus class
[
https://issues.apache.org/jira/browse/HADOOP-13715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15939451#comment-15939451
]
Andrew Wang commented on HADOOP-13715:
--
Cool, glad that we found this via unit test :) Took a deeper look, just one
last comment, my bad for not mentioning this earlier:
* Could we rename the JSON field "erasureBit" to "ecBit" for conciseness? This
will make the wire representation a little more compact. Also rename
{{ERASURE_CODED_BIT_JSON}} to {{EC_BIT_JSON}} to match as well.
> Add isErasureCoded() API to FileStatus class
>
>
> Key: HADOOP-13715
> URL: https://issues.apache.org/jira/browse/HADOOP-13715
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Affects Versions: 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Manoj Govindassamy
>Priority: Blocker
> Labels: hdfs-ec-3.0-must-do
> Attachments: HADOOP-13715.01.patch, HADOOP-13715.02.patch,
> HADOOP-13715.03.patch, HADOOP-13715.04.patch, HADOOP-13715.05.patch
>
>
> Per the discussion in
> [HDFS-10971|https://issues.apache.org/jira/browse/HDFS-10971?focusedCommentId=15567108=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15567108]
> I would like to add a new API {{isErasureCoded()}} to {{FileStatus}} so that
> tools and downstream applications can tell if it needs to treat a file
> differently.
> Hadoop tools that can benefit from this effort include: distcp and
> teragen/terasort.
> Downstream applications such as flume or hbase may also benefit from it.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14216) Improve Configuration XML Parsing Performance
[ https://issues.apache.org/jira/browse/HADOOP-14216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15939398#comment-15939398 ] Jonathan Eagles commented on HADOOP-14216: -- Regarding xi:include xi:fallback, this was never a documented feature that I could find and no tests proved continued support. Through using the underlying xerces2 xml parser, the xml extension https://www.w3.org/TR/xinclude/ was however available and examples were found to be using this feature. Stax parser, by design or maturity, don't support Xinclude. It was mostly trivial to implement. The essential feature is that xi:include statements xml inine statements. Thus, the source remains the same. xi:fallback statements are not used unless the xi:include fetch fails. Finals in xi:include statements preclude being overridden in the remaining document, but not final configuration can be overridden in the remaining document. If a fetch fails on an xi:include statement and no fallback is found, then that is a fatal error. > Improve Configuration XML Parsing Performance > - > > Key: HADOOP-14216 > URL: https://issues.apache.org/jira/browse/HADOOP-14216 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: HADOOP-14216.1.patch > > > JIRA is to improve XML parsing performance through reuse and a change in XML > parser (STAX) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14216) Improve Configuration XML Parsing Performance
[
https://issues.apache.org/jira/browse/HADOOP-14216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15939367#comment-15939367
]
Jonathan Eagles commented on HADOOP-14216:
--
Client-Side Performance Tests:
Setup: Essentially run normal user commands and see the performance gains with
only the client hadoop-common.jar replaced with a patch version
*Eyeball test*:
1. _hadoop fs -ls_
{code}
# baseline - ran dozens of times, this is a typical results
$ time hadoop fs -ls /
real0m2.694s
user0m6.633s
sys 0m0.303s
# patched version - ran dozens of times, this is a typical result
$ time HADOOP_USER_CLASSPATH_FIRST=true
HADOOP_CLASSPATH="./hadoop-common-2.8.1-HADOOP-14216.jar:./stax2-api-3.1.4.jar:./aalto-xml-1.0.0.jar"
hadoop fs -ls /
real0m2.335s
user0m4.963s
sys 0m0.292s
{code}
===
Result on a real cluster is roughly 300 ms real 1700 ms user faster per hadoop
fs -ls command
2. _yarn application -list_
{code}
$ time yarn application -list
real0m1.867s
user0m5.178s
sys 0m0.288s
$ time
YARN_USER_CLASSPATH="./hadoop-common-2.8.1-HADOOP-14216.jar:./stax2-api-3.1.4.jar:./aalto-xml-1.0.0.jar"
YARN_USER_CLASSPATH_FIRST=true yarn application -list
real0m1.607s
user0m3.911s
sys 0m0.225s
{code}
===
Result on a real cluster is roughly 250ms real and 1200 user faster per yarn
application -list command
*Performance Numbers at scale*
{code:title=ConfPerf.java}
import org.apache.hadoop.conf.Configuration;
public class ConfPerf {
public static void main(String[] args) throws Exception {
long start = System.currentTimeMillis();
long count = 0;
Configuration.addDefaultResource("core-default.xml");
Configuration.addDefaultResource("core-site.xml");
Configuration.addDefaultResource("yarn-default.xml");
Configuration.addDefaultResource("yarn-site.xml");
Configuration.addDefaultResource("mapred-default.xml");
Configuration.addDefaultResource("mapred-site.xml");
Configuration.addDefaultResource("hdfs-default.xml");
Configuration.addDefaultResource("hdfs-site.xml");
for (int i = 0; i < 3000; i++) {
Configuration conf = new Configuration();
conf.get("trigger.loading");
count += conf.size();
}
long end = System.currentTimeMillis();
System.out.println("duration: " + (end - start) + " count: " + count);
}
}
{code}
{code}
# setup performance tests
$ javac -cp ./:`hadoop classpath` ConfPerf.java
# baseline performance numbers
$ time java -cp ./:`hadoop classpath` ConfPerf
real0m52.456s
user1m2.209s
sys 0m3.601s
# performance numbers with patch
$ time java -cp
./:./hadoop-common-2.8.1-HADOOP-14216.jar:./stax2-api-3.1.4.jar:./aalto-xml-1.0.0.jar:`hadoop
classpath` ConfPerf
real0m23.108s
user0m27.434s
sys 0m1.816s
{code}
===
Result in a real cluster are roughly 29300 ms real and 34800 ms user faster
*Equality Test*
{code:title=ConfEquality.java}
import org.apache.hadoop.conf.Configuration;
public class ConfEquality {
public static void main(String[] args) throws Exception {
Configuration.addDefaultResource("core-default.xml");
Configuration.addDefaultResource("core-site.xml");
Configuration.addDefaultResource("yarn-default.xml");
Configuration.addDefaultResource("yarn-site.xml");
Configuration.addDefaultResource("mapred-default.xml");
Configuration.addDefaultResource("mapred-site.xml");
Configuration.addDefaultResource("hdfs-default.xml");
Configuration.addDefaultResource("hdfs-site.xml");
Configuration conf = new Configuration();
conf.get("trigger.loading");
conf.writeXml(System.out);
}
}
{code}
{code}
# prepare the equality test
$ javac -cp ./:`hadoop classpath` ConfEquality.java
# run the equality test
$ diff <(java -cp ./:`hadoop classpath` ConfEquality) <(java -cp
./:./hadoop-common-2.8.1-HADOOP-14216.jar:./stax2-api-3.1.4.jar:./aalto-xml-1.0.0.jar:`hadoop
classpath` ConfEquality)
{code}
> Improve Configuration XML Parsing Performance
> -
>
> Key: HADOOP-14216
> URL: https://issues.apache.org/jira/browse/HADOOP-14216
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
> Attachments: HADOOP-14216.1.patch
>
>
> JIRA is to improve XML parsing performance through reuse and a change in XML
> parser (STAX)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14197) Fix ADLS doc for credential provider
[
https://issues.apache.org/jira/browse/HADOOP-14197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15939359#comment-15939359
]
John Zhuge commented on HADOOP-14197:
-
Thanks [~chris.douglas] for the review and commit!
> Fix ADLS doc for credential provider
>
>
> Key: HADOOP-14197
> URL: https://issues.apache.org/jira/browse/HADOOP-14197
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: documentation, fs/adl
>Affects Versions: 2.8.0
>Reporter: John Zhuge
>Assignee: John Zhuge
> Fix For: 2.8.1, 3.0.0-alpha3
>
> Attachments: HADOOP-14197.001.patch
>
>
> There are a few errors in section {{Protecting the Credentials with
> Credential Providers}} of {{index.md}}:
> * Should add {{dfs.adls.oauth2.client.id}} instead of
> {{dfs.adls.oauth2.credential}} to the cred store
> * Should add {{dfs.adls.oauth2.access.token.provider.type}} to core-site.xml
> or DistCp command line
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11794) distcp can copy blocks in parallel
[ https://issues.apache.org/jira/browse/HADOOP-11794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15939337#comment-15939337 ] Yongjun Zhang commented on HADOOP-11794: The failed tests succeeded in local run. They look flaky and are not relevant to the change here. > distcp can copy blocks in parallel > -- > > Key: HADOOP-11794 > URL: https://issues.apache.org/jira/browse/HADOOP-11794 > Project: Hadoop Common > Issue Type: Improvement > Components: tools/distcp >Affects Versions: 0.21.0 >Reporter: dhruba borthakur >Assignee: Yongjun Zhang > Attachments: HADOOP-11794.001.patch, HADOOP-11794.002.patch, > HADOOP-11794.003.patch, HADOOP-11794.004.patch, HADOOP-11794.005.patch, > HADOOP-11794.006.patch, HADOOP-11794.007.patch, HADOOP-11794.008.patch, > HADOOP-11794.009.patch, HADOOP-11794.010.patch, MAPREDUCE-2257.patch > > > The minimum unit of work for a distcp task is a file. We have files that are > greater than 1 TB with a block size of 1 GB. If we use distcp to copy these > files, the tasks either take a long long long time or finally fails. A better > way for distcp would be to copy all the source blocks in parallel, and then > stich the blocks back to files at the destination via the HDFS Concat API > (HDFS-222) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14210) Directories are not listed recursively when fs.defaultFs is viewFs
[
https://issues.apache.org/jira/browse/HADOOP-14210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15939330#comment-15939330
]
Manoj Govindassamy commented on HADOOP-14210:
-
[~ajithshetty]
bq. {code}
-
- result[i++] = new FileStatus(0, false, 0, 0,
+ boolean isDir=link.isMergeLink;
+ if (link.targetDirLinkList.length == 1) {
+try {
+ isDir =
+ link.targetFileSystem.getFileLinkStatus(link.getTargetLink())
+ .isDirectory();
{code}
LinkMerge / MergeMounts are not supported yet. But, I see your point of
reaching out to the target filesystem to find if the linked item is indeed a
directory or not.
[~xkrogen]
In the context of ViewFileSystem, all the linked entities - be it be Dir or
File in target filesystem are of type INodeLink. Only the internal directories
in the ViewFileSystem mount tree are of type INodeDir. So, [~ajithshetty]
pointed out, ViewFileSystem treats only its internal directories as Dirs and
all others as linked files. So, the FileStatus[] returned by ViewFileSystem has
the Dir flag turned off for all the linked Directories in the target filesystem
making the LS command stop the file tree traversal.
Given the scale ViewFileSystem could be, returning millions of FileStatus[]
across all namenodes could be a problem for Clients as well. So, I was assuming
the intention for {{listStatus}} on ViewFileSystem is to only list the mount
tree and not the entire world. But, this doesn't go well "ls -R" expectation
from Clients. [~andrew.wang], any thoughts on the expectation for "ls -R" on
ViewFileSystem root ?
> Directories are not listed recursively when fs.defaultFs is viewFs
> --
>
> Key: HADOOP-14210
> URL: https://issues.apache.org/jira/browse/HADOOP-14210
> Project: Hadoop Common
> Issue Type: Bug
> Components: viewfs
>Affects Versions: 2.7.0
>Reporter: Ajith S
> Labels: viewfs
> Attachments: HDFS-8413.patch
>
>
> Mount a cluster on client throught viewFs mount table
> Example:
> {quote}
>
> fs.defaultFS
> viewfs:///
>
>
> fs.viewfs.mounttable.default.link./nn1
> hdfs://ns1/
>
>
> fs.viewfs.mounttable.default.link./user
> hdfs://host-72:8020/
>
>
> {quote}
> Try to list the files recursively *(hdfs dfs -ls -R / or hadoop fs -ls -R /)*
> only the parent folders are listed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13715) Add isErasureCoded() API to FileStatus class
[
https://issues.apache.org/jira/browse/HADOOP-13715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15939279#comment-15939279
]
Manoj Govindassamy commented on HADOOP-13715:
-
Test failures are not related to the patch.
> Add isErasureCoded() API to FileStatus class
>
>
> Key: HADOOP-13715
> URL: https://issues.apache.org/jira/browse/HADOOP-13715
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Affects Versions: 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Manoj Govindassamy
>Priority: Blocker
> Labels: hdfs-ec-3.0-must-do
> Attachments: HADOOP-13715.01.patch, HADOOP-13715.02.patch,
> HADOOP-13715.03.patch, HADOOP-13715.04.patch, HADOOP-13715.05.patch
>
>
> Per the discussion in
> [HDFS-10971|https://issues.apache.org/jira/browse/HDFS-10971?focusedCommentId=15567108=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15567108]
> I would like to add a new API {{isErasureCoded()}} to {{FileStatus}} so that
> tools and downstream applications can tell if it needs to treat a file
> differently.
> Hadoop tools that can benefit from this effort include: distcp and
> teragen/terasort.
> Downstream applications such as flume or hbase may also benefit from it.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13715) Add isErasureCoded() API to FileStatus class
[
https://issues.apache.org/jira/browse/HADOOP-13715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15939274#comment-15939274
]
Hadoop QA commented on HADOOP-13715:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 11 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
22s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 17m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
5s{color} | {color:green} root: The patch generated 0 new + 638 unchanged - 4
fixed = 638 total (was 642) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
36s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 59s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
11s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 65m 44s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
27s{color} | {color:green} hadoop-hdfs-httpfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
35s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
44s{color} | {color:green} hadoop-azure-datalake in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}205m 20s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.security.TestKDiag |
| | hadoop.hdfs.TestMaintenanceState |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-13715 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12860201/HADOOP-13715.05.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux ab381bd505f7 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
|
[jira] [Commented] (HADOOP-13887) Support for client-side encryption in S3A file system
[
https://issues.apache.org/jira/browse/HADOOP-13887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15939273#comment-15939273
]
Hadoop QA commented on HADOOP-13887:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 18m
54s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 14 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
40s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
25s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
36s{color} | {color:green} branch-2 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
27s{color} | {color:green} branch-2 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
29s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
31s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
20s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
27s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} branch-2 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
20s{color} | {color:green} branch-2 passed with JDK v1.7.0_121 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
12s{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 7m 12s{color}
| {color:red} root-jdk1.8.0_121 with JDK v1.8.0_121 generated 5 new + 884
unchanged - 3 fixed = 889 total (was 887) {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
45s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 7m 45s{color}
| {color:red} root-jdk1.7.0_121 with JDK v1.7.0_121 generated 4 new + 981
unchanged - 2 fixed = 985 total (was 983) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 44s{color} | {color:orange} root: The patch generated 2 new + 21 unchanged -
2 fixed = 23 total (was 23) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
0s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
32s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 10m 0s{color}
| {color:red} hadoop-common in the patch failed with JDK v1.7.0_121. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
32s{color} | {color:green} hadoop-aws in the patch passed with JDK v1.7.0_121.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}127m 13s{color} |
[jira] [Commented] (HADOOP-14195) CredentialProviderFactory$getProviders is not thread-safe
[
https://issues.apache.org/jira/browse/HADOOP-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15939244#comment-15939244
]
Hudson commented on HADOOP-14195:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11451 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/11451/])
HADOOP-14195. CredentialProviderFactory$getProviders is not thread-safe.
(jzhuge: rev 128015584d69492806fd1700c8f840d78aa9c729)
* (edit)
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/CredentialProviderFactory.java
> CredentialProviderFactory$getProviders is not thread-safe
> -
>
> Key: HADOOP-14195
> URL: https://issues.apache.org/jira/browse/HADOOP-14195
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.7.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Fix For: 2.9.0, 2.8.1, 3.0.0-alpha3
>
> Attachments: HADOOP-14195.001.patch, HADOOP-14195.002.patch,
> HADOOP-14195.003.patch, TestCredentialProvider.java
>
>
> Multi-threaded access to CredentialProviderFactory is not thread-safe because
> {{java.util.ServiceLoader}} is not thread-safe (as noted in its Java doc).
> Thanks to [~jzhuge] I was able to reproduce this issue but creating a simple
> multi-threaded application which executes the following code in parallel.
> {code:java}
> for (int i = 0; i < ITEMS; i++) {
> futures.add(executor.submit(new Callable() {
> @Override
> public Void call() throws Exception {
> boolean found = false;
> for (CredentialProviderFactory factory : serviceLoader) {
> CredentialProvider kp = factory.createProvider(uri,
> conf);
> if (kp != null) {
> result.add(kp);
> found = true;
> break;
> }
> }
> if (!found) {
> throw new IOException(Thread.currentThread() + "No
> CredentialProviderFactory for " + uri);
> } else {
> System.out.println(Thread.currentThread().getName() + "
> found credentialProvider for " + path);
> }
> return null;
> }
> }));
> }
> {code}
> I see the following exception trace when I execute the above code.
> {code:java}
> java.util.concurrent.ExecutionException: java.util.NoSuchElementException
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at TestCredentialProvider.main(TestCredentialProvider.java:58)
> Caused by: java.util.NoSuchElementException
> at java.net.URLClassLoader$3.nextElement(URLClassLoader.java:615)
> at java.net.URLClassLoader$3.nextElement(URLClassLoader.java:590)
> at sun.misc.CompoundEnumeration.nextElement(CompoundEnumeration.java:61)
> at
> java.util.ServiceLoader$LazyIterator.hasNextService(ServiceLoader.java:357)
> at java.util.ServiceLoader$LazyIterator.hasNext(ServiceLoader.java:393)
> at java.util.ServiceLoader$1.hasNext(ServiceLoader.java:474)
> at TestCredentialProvider$1.call(TestCredentialProvider.java:38)
> at TestCredentialProvider$1.call(TestCredentialProvider.java:1)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> I also see a NPE sometimes
> {code:java}
> java.util.concurrent.ExecutionException: java.lang.NullPointerException
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at TestCredentialProvider.main(TestCredentialProvider.java:58)
> Caused by: java.lang.NullPointerException
> at java.util.ServiceLoader.parse(ServiceLoader.java:304)
> at java.util.ServiceLoader.access$200(ServiceLoader.java:185)
> at
> java.util.ServiceLoader$LazyIterator.hasNextService(ServiceLoader.java:357)
> at java.util.ServiceLoader$LazyIterator.hasNext(ServiceLoader.java:393)
> at java.util.ServiceLoader$1.hasNext(ServiceLoader.java:474)
> at TestCredentialProvider$1.call(TestCredentialProvider.java:38)
> at TestCredentialProvider$1.call(TestCredentialProvider.java:1)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
>
[jira] [Commented] (HADOOP-13377) AliyunOSS: improvements for stabilization
[ https://issues.apache.org/jira/browse/HADOOP-13377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15939238#comment-15939238 ] Mingliang Liu commented on HADOOP-13377: I moved two recent changes of fs/oss module as subtasks of this uber JIRA. Let's track changes here. > AliyunOSS: improvements for stabilization > - > > Key: HADOOP-13377 > URL: https://issues.apache.org/jira/browse/HADOOP-13377 > Project: Hadoop Common > Issue Type: Improvement > Components: fs, fs/oss >Affects Versions: 3.0.0-alpha2 >Reporter: Genmao Yu >Assignee: Genmao Yu > > This work is to handle some stabilization issues. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14194) Aliyun OSS should not use empty endpoint as default
[
https://issues.apache.org/jira/browse/HADOOP-14194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HADOOP-14194:
---
Issue Type: Sub-task (was: Bug)
Parent: HADOOP-13377
> Aliyun OSS should not use empty endpoint as default
> ---
>
> Key: HADOOP-14194
> URL: https://issues.apache.org/jira/browse/HADOOP-14194
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/oss
>Reporter: Mingliang Liu
>Assignee: Xiaobing Zhou
>
> In {{AliyunOSSFileSystemStore::initialize()}}, it retrieves the endPoint and
> using empty string as a default value.
> {code}
> String endPoint = conf.getTrimmed(ENDPOINT_KEY, "");
> {code}
> The plain value without validation is passed to OSSClient. If the endPoint is
> not provided (empty string) or the endPoint is not valid, users will get
> exception from Aliyun OSS sdk with raw exception message like:
> {code}
> java.lang.IllegalArgumentException: java.net.URISyntaxException: Expected
> authority at index 8: https://
> at com.aliyun.oss.OSSClient.toURI(OSSClient.java:359)
> at com.aliyun.oss.OSSClient.setEndpoint(OSSClient.java:313)
> at com.aliyun.oss.OSSClient.(OSSClient.java:297)
> at
> org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystemStore.initialize(AliyunOSSFileSystemStore.java:134)
> at
> org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem.initialize(AliyunOSSFileSystem.java:272)
> at
> org.apache.hadoop.fs.aliyun.oss.AliyunOSSTestUtils.createTestFileSystem(AliyunOSSTestUtils.java:63)
> at
> org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemContract.setUp(TestAliyunOSSFileSystemContract.java:47)
> at junit.framework.TestCase.runBare(TestCase.java:139)
> at junit.framework.TestResult$1.protect(TestResult.java:122)
> at junit.framework.TestResult.runProtected(TestResult.java:142)
> at junit.framework.TestResult.run(TestResult.java:125)
> at junit.framework.TestCase.run(TestCase.java:129)
> at junit.framework.TestSuite.runTest(TestSuite.java:255)
> at junit.framework.TestSuite.run(TestSuite.java:250)
> at
> org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
> at
> com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51)
> at
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:237)
> at
> com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
> Caused by: java.net.URISyntaxException: Expected authority at index 8:
> https://
> at java.net.URI$Parser.fail(URI.java:2848)
> at java.net.URI$Parser.failExpecting(URI.java:2854)
> at java.net.URI$Parser.parseHierarchical(URI.java:3102)
> at java.net.URI$Parser.parse(URI.java:3053)
> at java.net.URI.(URI.java:588)
> at com.aliyun.oss.OSSClient.toURI(OSSClient.java:357)
> {code}
> Let's check endPoint is not null or empty, catch the IllegalArgumentException
> and log it, wrapping the exception with clearer message stating the
> misconfiguration in endpoint or credentials.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14192) Aliyun OSS FileSystem contract test should implement getTestBaseDir()
[
https://issues.apache.org/jira/browse/HADOOP-14192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HADOOP-14192:
---
Issue Type: Sub-task (was: Bug)
Parent: HADOOP-13377
> Aliyun OSS FileSystem contract test should implement getTestBaseDir()
> -
>
> Key: HADOOP-14192
> URL: https://issues.apache.org/jira/browse/HADOOP-14192
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/oss
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Fix For: 3.0.0-alpha3
>
> Attachments: HADOOP-14192.000.patch
>
>
> [HADOOP-14170] is the recent effort of improving the file system contract
> tests {{FileSystemContractBaseTest}}, which make {{path()}} method final and
> add a new method {{getTestBaseDir()}} for subclasses to implement. Aliyun OSS
> should override that as it uses unique directory (naming with fork id) for
> supporting parallel tests. Plus, the current {{testWorkingDirectory}} needs
> not override per changes in {{FileSystemContractBaseTest}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14195) CredentialProviderFactory$getProviders is not thread-safe
[
https://issues.apache.org/jira/browse/HADOOP-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HADOOP-14195:
Target Version/s: 2.8.1 (was: 2.7.3)
> CredentialProviderFactory$getProviders is not thread-safe
> -
>
> Key: HADOOP-14195
> URL: https://issues.apache.org/jira/browse/HADOOP-14195
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.7.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Fix For: 2.9.0, 2.8.1, 3.0.0-alpha3
>
> Attachments: HADOOP-14195.001.patch, HADOOP-14195.002.patch,
> HADOOP-14195.003.patch, TestCredentialProvider.java
>
>
> Multi-threaded access to CredentialProviderFactory is not thread-safe because
> {{java.util.ServiceLoader}} is not thread-safe (as noted in its Java doc).
> Thanks to [~jzhuge] I was able to reproduce this issue but creating a simple
> multi-threaded application which executes the following code in parallel.
> {code:java}
> for (int i = 0; i < ITEMS; i++) {
> futures.add(executor.submit(new Callable() {
> @Override
> public Void call() throws Exception {
> boolean found = false;
> for (CredentialProviderFactory factory : serviceLoader) {
> CredentialProvider kp = factory.createProvider(uri,
> conf);
> if (kp != null) {
> result.add(kp);
> found = true;
> break;
> }
> }
> if (!found) {
> throw new IOException(Thread.currentThread() + "No
> CredentialProviderFactory for " + uri);
> } else {
> System.out.println(Thread.currentThread().getName() + "
> found credentialProvider for " + path);
> }
> return null;
> }
> }));
> }
> {code}
> I see the following exception trace when I execute the above code.
> {code:java}
> java.util.concurrent.ExecutionException: java.util.NoSuchElementException
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at TestCredentialProvider.main(TestCredentialProvider.java:58)
> Caused by: java.util.NoSuchElementException
> at java.net.URLClassLoader$3.nextElement(URLClassLoader.java:615)
> at java.net.URLClassLoader$3.nextElement(URLClassLoader.java:590)
> at sun.misc.CompoundEnumeration.nextElement(CompoundEnumeration.java:61)
> at
> java.util.ServiceLoader$LazyIterator.hasNextService(ServiceLoader.java:357)
> at java.util.ServiceLoader$LazyIterator.hasNext(ServiceLoader.java:393)
> at java.util.ServiceLoader$1.hasNext(ServiceLoader.java:474)
> at TestCredentialProvider$1.call(TestCredentialProvider.java:38)
> at TestCredentialProvider$1.call(TestCredentialProvider.java:1)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> I also see a NPE sometimes
> {code:java}
> java.util.concurrent.ExecutionException: java.lang.NullPointerException
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at TestCredentialProvider.main(TestCredentialProvider.java:58)
> Caused by: java.lang.NullPointerException
> at java.util.ServiceLoader.parse(ServiceLoader.java:304)
> at java.util.ServiceLoader.access$200(ServiceLoader.java:185)
> at
> java.util.ServiceLoader$LazyIterator.hasNextService(ServiceLoader.java:357)
> at java.util.ServiceLoader$LazyIterator.hasNext(ServiceLoader.java:393)
> at java.util.ServiceLoader$1.hasNext(ServiceLoader.java:474)
> at TestCredentialProvider$1.call(TestCredentialProvider.java:38)
> at TestCredentialProvider$1.call(TestCredentialProvider.java:1)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail:
[jira] [Comment Edited] (HADOOP-14216) Improve Configuration XML Parsing Performance
[
https://issues.apache.org/jira/browse/HADOOP-14216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15939231#comment-15939231
]
Chris Douglas edited comment on HADOOP-14216 at 3/23/17 9:07 PM:
-
I only skimmed the patch, but this looks good. Just a few questions:
* I have no trouble believing this shows up in perf traces, but out of
curiosity, do you have any data on the improvement this effects?
* Filed HADOOP-14225; can we use the dependency added here to replace xmlenc?
* Can you update/add/link docs on the {{xi:fallback}} and {{xi:include}}
semantics? How do these interact with {{final}} config values?
was (Author: chris.douglas):
I only skimmed the patch, but this looks good. Just a few questions:
* I have no trouble believing this shows up in perf traces, but out of
curiosity, do you have any data on the improvement this effects?
* Filed HADOOP-14255; can we use the dependency added here to replace xmlenc?
* Can you update/add/link docs on the {{xi:fallback}} and {{xi:include}}
semantics? How do these interact with {{final}} config values?
> Improve Configuration XML Parsing Performance
> -
>
> Key: HADOOP-14216
> URL: https://issues.apache.org/jira/browse/HADOOP-14216
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
> Attachments: HADOOP-14216.1.patch
>
>
> JIRA is to improve XML parsing performance through reuse and a change in XML
> parser (STAX)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14216) Improve Configuration XML Parsing Performance
[
https://issues.apache.org/jira/browse/HADOOP-14216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15939231#comment-15939231
]
Chris Douglas commented on HADOOP-14216:
I only skimmed the patch, but this looks good. Just a few questions:
* I have no trouble believing this shows up in perf traces, but out of
curiosity, do you have any data on the improvement this effects?
* Filed HADOOP-14255; can we use the dependency added here to replace xmlenc?
* Can you update/add/link docs on the {{xi:fallback}} and {{xi:include}}
semantics? How do these interact with {{final}} config values?
> Improve Configuration XML Parsing Performance
> -
>
> Key: HADOOP-14216
> URL: https://issues.apache.org/jira/browse/HADOOP-14216
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
> Attachments: HADOOP-14216.1.patch
>
>
> JIRA is to improve XML parsing performance through reuse and a change in XML
> parser (STAX)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14225) Remove xmlenc dependency
Chris Douglas created HADOOP-14225:
--
Summary: Remove xmlenc dependency
Key: HADOOP-14225
URL: https://issues.apache.org/jira/browse/HADOOP-14225
Project: Hadoop Common
Issue Type: Improvement
Reporter: Chris Douglas
Priority: Minor
The xmlenc library is used only in the following two classes:
{noformat}
o.a.h.fs.MD5MD5CRC32FileChecksum
o.a.h.hdfs.server.namenode.DfsServlet
{noformat}
Given that Hadoop already includes other fast XML encoders as dependencies, we
can lose this one.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14195) CredentialProviderFactory$getProviders is not thread-safe
[
https://issues.apache.org/jira/browse/HADOOP-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HADOOP-14195:
Resolution: Fixed
Fix Version/s: 3.0.0-alpha3
2.8.1
2.9.0
Status: Resolved (was: Patch Available)
> CredentialProviderFactory$getProviders is not thread-safe
> -
>
> Key: HADOOP-14195
> URL: https://issues.apache.org/jira/browse/HADOOP-14195
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.7.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Fix For: 2.9.0, 2.8.1, 3.0.0-alpha3
>
> Attachments: HADOOP-14195.001.patch, HADOOP-14195.002.patch,
> HADOOP-14195.003.patch, TestCredentialProvider.java
>
>
> Multi-threaded access to CredentialProviderFactory is not thread-safe because
> {{java.util.ServiceLoader}} is not thread-safe (as noted in its Java doc).
> Thanks to [~jzhuge] I was able to reproduce this issue but creating a simple
> multi-threaded application which executes the following code in parallel.
> {code:java}
> for (int i = 0; i < ITEMS; i++) {
> futures.add(executor.submit(new Callable() {
> @Override
> public Void call() throws Exception {
> boolean found = false;
> for (CredentialProviderFactory factory : serviceLoader) {
> CredentialProvider kp = factory.createProvider(uri,
> conf);
> if (kp != null) {
> result.add(kp);
> found = true;
> break;
> }
> }
> if (!found) {
> throw new IOException(Thread.currentThread() + "No
> CredentialProviderFactory for " + uri);
> } else {
> System.out.println(Thread.currentThread().getName() + "
> found credentialProvider for " + path);
> }
> return null;
> }
> }));
> }
> {code}
> I see the following exception trace when I execute the above code.
> {code:java}
> java.util.concurrent.ExecutionException: java.util.NoSuchElementException
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at TestCredentialProvider.main(TestCredentialProvider.java:58)
> Caused by: java.util.NoSuchElementException
> at java.net.URLClassLoader$3.nextElement(URLClassLoader.java:615)
> at java.net.URLClassLoader$3.nextElement(URLClassLoader.java:590)
> at sun.misc.CompoundEnumeration.nextElement(CompoundEnumeration.java:61)
> at
> java.util.ServiceLoader$LazyIterator.hasNextService(ServiceLoader.java:357)
> at java.util.ServiceLoader$LazyIterator.hasNext(ServiceLoader.java:393)
> at java.util.ServiceLoader$1.hasNext(ServiceLoader.java:474)
> at TestCredentialProvider$1.call(TestCredentialProvider.java:38)
> at TestCredentialProvider$1.call(TestCredentialProvider.java:1)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> I also see a NPE sometimes
> {code:java}
> java.util.concurrent.ExecutionException: java.lang.NullPointerException
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at TestCredentialProvider.main(TestCredentialProvider.java:58)
> Caused by: java.lang.NullPointerException
> at java.util.ServiceLoader.parse(ServiceLoader.java:304)
> at java.util.ServiceLoader.access$200(ServiceLoader.java:185)
> at
> java.util.ServiceLoader$LazyIterator.hasNextService(ServiceLoader.java:357)
> at java.util.ServiceLoader$LazyIterator.hasNext(ServiceLoader.java:393)
> at java.util.ServiceLoader$1.hasNext(ServiceLoader.java:474)
> at TestCredentialProvider$1.call(TestCredentialProvider.java:38)
> at TestCredentialProvider$1.call(TestCredentialProvider.java:1)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To
[jira] [Commented] (HADOOP-14195) CredentialProviderFactory$getProviders is not thread-safe
[
https://issues.apache.org/jira/browse/HADOOP-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15939226#comment-15939226
]
John Zhuge commented on HADOOP-14195:
-
Committed to trunk, branch-2, and branch-2.8.
Thanks [~vihangk1] for the contribution!
> CredentialProviderFactory$getProviders is not thread-safe
> -
>
> Key: HADOOP-14195
> URL: https://issues.apache.org/jira/browse/HADOOP-14195
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.7.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HADOOP-14195.001.patch, HADOOP-14195.002.patch,
> HADOOP-14195.003.patch, TestCredentialProvider.java
>
>
> Multi-threaded access to CredentialProviderFactory is not thread-safe because
> {{java.util.ServiceLoader}} is not thread-safe (as noted in its Java doc).
> Thanks to [~jzhuge] I was able to reproduce this issue but creating a simple
> multi-threaded application which executes the following code in parallel.
> {code:java}
> for (int i = 0; i < ITEMS; i++) {
> futures.add(executor.submit(new Callable() {
> @Override
> public Void call() throws Exception {
> boolean found = false;
> for (CredentialProviderFactory factory : serviceLoader) {
> CredentialProvider kp = factory.createProvider(uri,
> conf);
> if (kp != null) {
> result.add(kp);
> found = true;
> break;
> }
> }
> if (!found) {
> throw new IOException(Thread.currentThread() + "No
> CredentialProviderFactory for " + uri);
> } else {
> System.out.println(Thread.currentThread().getName() + "
> found credentialProvider for " + path);
> }
> return null;
> }
> }));
> }
> {code}
> I see the following exception trace when I execute the above code.
> {code:java}
> java.util.concurrent.ExecutionException: java.util.NoSuchElementException
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at TestCredentialProvider.main(TestCredentialProvider.java:58)
> Caused by: java.util.NoSuchElementException
> at java.net.URLClassLoader$3.nextElement(URLClassLoader.java:615)
> at java.net.URLClassLoader$3.nextElement(URLClassLoader.java:590)
> at sun.misc.CompoundEnumeration.nextElement(CompoundEnumeration.java:61)
> at
> java.util.ServiceLoader$LazyIterator.hasNextService(ServiceLoader.java:357)
> at java.util.ServiceLoader$LazyIterator.hasNext(ServiceLoader.java:393)
> at java.util.ServiceLoader$1.hasNext(ServiceLoader.java:474)
> at TestCredentialProvider$1.call(TestCredentialProvider.java:38)
> at TestCredentialProvider$1.call(TestCredentialProvider.java:1)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> I also see a NPE sometimes
> {code:java}
> java.util.concurrent.ExecutionException: java.lang.NullPointerException
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at TestCredentialProvider.main(TestCredentialProvider.java:58)
> Caused by: java.lang.NullPointerException
> at java.util.ServiceLoader.parse(ServiceLoader.java:304)
> at java.util.ServiceLoader.access$200(ServiceLoader.java:185)
> at
> java.util.ServiceLoader$LazyIterator.hasNextService(ServiceLoader.java:357)
> at java.util.ServiceLoader$LazyIterator.hasNext(ServiceLoader.java:393)
> at java.util.ServiceLoader$1.hasNext(ServiceLoader.java:474)
> at TestCredentialProvider$1.call(TestCredentialProvider.java:38)
> at TestCredentialProvider$1.call(TestCredentialProvider.java:1)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional
[jira] [Commented] (HADOOP-14214) DomainSocketWatcher::add()/delete() should not self interrupt while looping await()
[
https://issues.apache.org/jira/browse/HADOOP-14214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15939220#comment-15939220
]
Hudson commented on HADOOP-14214:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11450 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/11450/])
HADOOP-14214. DomainSocketWatcher::add()/delete() should not self (liuml07: rev
d35e79abc2fee7153a6168e6088f100de59d8c81)
* (edit)
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/unix/DomainSocketWatcher.java
> DomainSocketWatcher::add()/delete() should not self interrupt while looping
> await()
> ---
>
> Key: HADOOP-14214
> URL: https://issues.apache.org/jira/browse/HADOOP-14214
> Project: Hadoop Common
> Issue Type: Bug
> Components: hdfs-client
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>Priority: Critical
> Fix For: 2.8.1, 3.0.0-alpha3
>
> Attachments: HADOOP-14214.000.patch
>
>
> Our hive team found a TPCDS job whose queries running on LLAP seem to be
> getting stuck. Dozens of threads were waiting for the
> {{DfsClientShmManager::lock}}, as following jstack:
> {code}
> Thread 251 (IO-Elevator-Thread-5):
> State: WAITING
> Blocked count: 3871
> Wtaited count: 4565
> Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@16ead198
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUninterruptibly(AbstractQueuedSynchronizer.java:1976)
>
> org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager$EndpointShmManager.allocSlot(DfsClientShmManager.java:255)
>
> org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager.allocSlot(DfsClientShmManager.java:434)
>
> org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.allocShmSlot(ShortCircuitCache.java:1017)
>
> org.apache.hadoop.hdfs.BlockReaderFactory.createShortCircuitReplicaInfo(BlockReaderFactory.java:476)
>
> org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.create(ShortCircuitCache.java:784)
>
> org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.fetchOrCreate(ShortCircuitCache.java:718)
>
> org.apache.hadoop.hdfs.BlockReaderFactory.getBlockReaderLocal(BlockReaderFactory.java:422)
>
> org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:333)
>
> org.apache.hadoop.hdfs.DFSInputStream.actualGetFromOneDataNode(DFSInputStream.java:1181)
>
> org.apache.hadoop.hdfs.DFSInputStream.fetchBlockByteRange(DFSInputStream.java:1118)
> org.apache.hadoop.hdfs.DFSInputStream.pread(DFSInputStream.java:1478)
> org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:1441)
> org.apache.hadoop.fs.FSInputStream.readFully(FSInputStream.java:121)
>
> org.apache.hadoop.fs.FSDataInputStream.readFully(FSDataInputStream.java:111)
>
> org.apache.orc.impl.RecordReaderUtils$DefaultDataReader.readStripeFooter(RecordReaderUtils.java:166)
>
> org.apache.hadoop.hive.llap.io.metadata.OrcStripeMetadata.(OrcStripeMetadata.java:64)
>
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.readStripesMetadata(OrcEncodedDataReader.java:622)
> {code}
> The thread that is expected to signal those threads is calling
> {{DomainSocketWatcher::add()}} method, but it gets stuck there dealing with
> InterruptedException infinitely. The jstack is like:
> {code}
> Thread 44417 (TezTR-257387_2840_12_10_52_0):
> State: RUNNABLE
> Blocked count: 3
> Wtaited count: 5
> Stack:
> java.lang.Throwable.fillInStackTrace(Native Method)
> java.lang.Throwable.fillInStackTrace(Throwable.java:783)
> java.lang.Throwable.(Throwable.java:250)
> java.lang.Exception.(Exception.java:54)
> java.lang.InterruptedException.(InterruptedException.java:57)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2034)
>
> org.apache.hadoop.net.unix.DomainSocketWatcher.add(DomainSocketWatcher.java:325)
>
> org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager$EndpointShmManager.allocSlot(DfsClientShmManager.java:266)
>
> org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager.allocSlot(DfsClientShmManager.java:434)
>
> org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.allocShmSlot(ShortCircuitCache.java:1017)
>
> org.apache.hadoop.hdfs.BlockReaderFactory.createShortCircuitReplicaInfo(BlockReaderFactory.java:476)
>
> org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.create(ShortCircuitCache.java:784)
>
> org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.fetchOrCreate(ShortCircuitCache.java:718)
>
>
[jira] [Updated] (HADOOP-14214) DomainSocketWatcher::add()/delete() should not self interrupt while looping await()
[
https://issues.apache.org/jira/browse/HADOOP-14214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HADOOP-14214:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha3
2.8.1
Status: Resolved (was: Patch Available)
Committed to {{trunk}} through {{branch-2.8}} branches. Thanks [~sershe] for
discussion. Thanks [~jnp] and [~szetszwo] for review.
> DomainSocketWatcher::add()/delete() should not self interrupt while looping
> await()
> ---
>
> Key: HADOOP-14214
> URL: https://issues.apache.org/jira/browse/HADOOP-14214
> Project: Hadoop Common
> Issue Type: Bug
> Components: hdfs-client
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>Priority: Critical
> Fix For: 2.8.1, 3.0.0-alpha3
>
> Attachments: HADOOP-14214.000.patch
>
>
> Our hive team found a TPCDS job whose queries running on LLAP seem to be
> getting stuck. Dozens of threads were waiting for the
> {{DfsClientShmManager::lock}}, as following jstack:
> {code}
> Thread 251 (IO-Elevator-Thread-5):
> State: WAITING
> Blocked count: 3871
> Wtaited count: 4565
> Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@16ead198
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUninterruptibly(AbstractQueuedSynchronizer.java:1976)
>
> org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager$EndpointShmManager.allocSlot(DfsClientShmManager.java:255)
>
> org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager.allocSlot(DfsClientShmManager.java:434)
>
> org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.allocShmSlot(ShortCircuitCache.java:1017)
>
> org.apache.hadoop.hdfs.BlockReaderFactory.createShortCircuitReplicaInfo(BlockReaderFactory.java:476)
>
> org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.create(ShortCircuitCache.java:784)
>
> org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.fetchOrCreate(ShortCircuitCache.java:718)
>
> org.apache.hadoop.hdfs.BlockReaderFactory.getBlockReaderLocal(BlockReaderFactory.java:422)
>
> org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:333)
>
> org.apache.hadoop.hdfs.DFSInputStream.actualGetFromOneDataNode(DFSInputStream.java:1181)
>
> org.apache.hadoop.hdfs.DFSInputStream.fetchBlockByteRange(DFSInputStream.java:1118)
> org.apache.hadoop.hdfs.DFSInputStream.pread(DFSInputStream.java:1478)
> org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:1441)
> org.apache.hadoop.fs.FSInputStream.readFully(FSInputStream.java:121)
>
> org.apache.hadoop.fs.FSDataInputStream.readFully(FSDataInputStream.java:111)
>
> org.apache.orc.impl.RecordReaderUtils$DefaultDataReader.readStripeFooter(RecordReaderUtils.java:166)
>
> org.apache.hadoop.hive.llap.io.metadata.OrcStripeMetadata.(OrcStripeMetadata.java:64)
>
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.readStripesMetadata(OrcEncodedDataReader.java:622)
> {code}
> The thread that is expected to signal those threads is calling
> {{DomainSocketWatcher::add()}} method, but it gets stuck there dealing with
> InterruptedException infinitely. The jstack is like:
> {code}
> Thread 44417 (TezTR-257387_2840_12_10_52_0):
> State: RUNNABLE
> Blocked count: 3
> Wtaited count: 5
> Stack:
> java.lang.Throwable.fillInStackTrace(Native Method)
> java.lang.Throwable.fillInStackTrace(Throwable.java:783)
> java.lang.Throwable.(Throwable.java:250)
> java.lang.Exception.(Exception.java:54)
> java.lang.InterruptedException.(InterruptedException.java:57)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2034)
>
> org.apache.hadoop.net.unix.DomainSocketWatcher.add(DomainSocketWatcher.java:325)
>
> org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager$EndpointShmManager.allocSlot(DfsClientShmManager.java:266)
>
> org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager.allocSlot(DfsClientShmManager.java:434)
>
> org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.allocShmSlot(ShortCircuitCache.java:1017)
>
> org.apache.hadoop.hdfs.BlockReaderFactory.createShortCircuitReplicaInfo(BlockReaderFactory.java:476)
>
> org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.create(ShortCircuitCache.java:784)
>
> org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.fetchOrCreate(ShortCircuitCache.java:718)
>
> org.apache.hadoop.hdfs.BlockReaderFactory.getBlockReaderLocal(BlockReaderFactory.java:422)
>
>
[jira] [Commented] (HADOOP-11794) distcp can copy blocks in parallel
[
https://issues.apache.org/jira/browse/HADOOP-11794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15939192#comment-15939192
]
Hadoop QA commented on HADOOP-11794:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 22m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 17m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
23s{color} | {color:green} root: The patch generated 0 new + 377 unchanged - 13
fixed = 377 total (was 390) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} hadoop-tools_hadoop-distcp generated 0 new + 48
unchanged - 1 fixed = 48 total (was 49) {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 76m 13s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m
46s{color} | {color:green} hadoop-distcp in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
44s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}192m 12s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000 |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-11794 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12860188/HADOOP-11794.010.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 5de3607cd1d0 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 595f62e |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit |
[jira] [Commented] (HADOOP-13786) Add S3Guard committer for zero-rename commits to consistent S3 endpoints
[
https://issues.apache.org/jira/browse/HADOOP-13786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15939186#comment-15939186
]
Hadoop QA commented on HADOOP-13786:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 30 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
54s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
42s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
51s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
4s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
40s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
15s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
20s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
57s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
8s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 11m 8s{color}
| {color:red} root generated 9 new + 761 unchanged - 2 fixed = 770 total (was
763) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 11s{color} | {color:orange} root: The patch generated 118 new + 98 unchanged
- 14 fixed = 216 total (was 112) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 16 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
16s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
29s{color} | {color:red} hadoop-aws in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 25s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
13s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 1s{color}
| {color:red} hadoop-aws in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
57s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}105m 29s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ha.TestZKFailoverController |
| | hadoop.security.TestKDiag |
| | hadoop.net.TestDNS |
| | hadoop.fs.s3a.commit.staging.TestStagingMRJob |
| | hadoop.fs.s3a.commit.staging.TestStagingCommitter |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-13786 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12860204/HADOOP-13786-HADOOP-13345-019.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 8982abcea441 3.13.0-106-generic #153-Ubuntu SMP Tue
[jira] [Commented] (HADOOP-14214) DomainSocketWatcher::add()/delete() should not self interrupt while looping await()
[
https://issues.apache.org/jira/browse/HADOOP-14214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15939179#comment-15939179
]
Tsz Wo Nicholas Sze commented on HADOOP-14214:
--
The try-await-catch-InterruptedException-interrupt is clearly a bug. Using
awaitUninterruptibly sounds good.
+1 on the patch.
> DomainSocketWatcher::add()/delete() should not self interrupt while looping
> await()
> ---
>
> Key: HADOOP-14214
> URL: https://issues.apache.org/jira/browse/HADOOP-14214
> Project: Hadoop Common
> Issue Type: Bug
> Components: hdfs-client
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>Priority: Critical
> Attachments: HADOOP-14214.000.patch
>
>
> Our hive team found a TPCDS job whose queries running on LLAP seem to be
> getting stuck. Dozens of threads were waiting for the
> {{DfsClientShmManager::lock}}, as following jstack:
> {code}
> Thread 251 (IO-Elevator-Thread-5):
> State: WAITING
> Blocked count: 3871
> Wtaited count: 4565
> Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@16ead198
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUninterruptibly(AbstractQueuedSynchronizer.java:1976)
>
> org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager$EndpointShmManager.allocSlot(DfsClientShmManager.java:255)
>
> org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager.allocSlot(DfsClientShmManager.java:434)
>
> org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.allocShmSlot(ShortCircuitCache.java:1017)
>
> org.apache.hadoop.hdfs.BlockReaderFactory.createShortCircuitReplicaInfo(BlockReaderFactory.java:476)
>
> org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.create(ShortCircuitCache.java:784)
>
> org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.fetchOrCreate(ShortCircuitCache.java:718)
>
> org.apache.hadoop.hdfs.BlockReaderFactory.getBlockReaderLocal(BlockReaderFactory.java:422)
>
> org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:333)
>
> org.apache.hadoop.hdfs.DFSInputStream.actualGetFromOneDataNode(DFSInputStream.java:1181)
>
> org.apache.hadoop.hdfs.DFSInputStream.fetchBlockByteRange(DFSInputStream.java:1118)
> org.apache.hadoop.hdfs.DFSInputStream.pread(DFSInputStream.java:1478)
> org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:1441)
> org.apache.hadoop.fs.FSInputStream.readFully(FSInputStream.java:121)
>
> org.apache.hadoop.fs.FSDataInputStream.readFully(FSDataInputStream.java:111)
>
> org.apache.orc.impl.RecordReaderUtils$DefaultDataReader.readStripeFooter(RecordReaderUtils.java:166)
>
> org.apache.hadoop.hive.llap.io.metadata.OrcStripeMetadata.(OrcStripeMetadata.java:64)
>
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.readStripesMetadata(OrcEncodedDataReader.java:622)
> {code}
> The thread that is expected to signal those threads is calling
> {{DomainSocketWatcher::add()}} method, but it gets stuck there dealing with
> InterruptedException infinitely. The jstack is like:
> {code}
> Thread 44417 (TezTR-257387_2840_12_10_52_0):
> State: RUNNABLE
> Blocked count: 3
> Wtaited count: 5
> Stack:
> java.lang.Throwable.fillInStackTrace(Native Method)
> java.lang.Throwable.fillInStackTrace(Throwable.java:783)
> java.lang.Throwable.(Throwable.java:250)
> java.lang.Exception.(Exception.java:54)
> java.lang.InterruptedException.(InterruptedException.java:57)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2034)
>
> org.apache.hadoop.net.unix.DomainSocketWatcher.add(DomainSocketWatcher.java:325)
>
> org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager$EndpointShmManager.allocSlot(DfsClientShmManager.java:266)
>
> org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager.allocSlot(DfsClientShmManager.java:434)
>
> org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.allocShmSlot(ShortCircuitCache.java:1017)
>
> org.apache.hadoop.hdfs.BlockReaderFactory.createShortCircuitReplicaInfo(BlockReaderFactory.java:476)
>
> org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.create(ShortCircuitCache.java:784)
>
> org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.fetchOrCreate(ShortCircuitCache.java:718)
>
> org.apache.hadoop.hdfs.BlockReaderFactory.getBlockReaderLocal(BlockReaderFactory.java:422)
>
> org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:333)
>
>
[jira] [Commented] (HADOOP-14197) Fix ADLS doc for credential provider
[
https://issues.apache.org/jira/browse/HADOOP-14197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15939121#comment-15939121
]
Hudson commented on HADOOP-14197:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11448 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/11448/])
HADOOP-14197. Fix ADLS doc for credential provider. Contributed by John
(cdouglas: rev c55195588f5853b16d34cb6389cc6f84acc3edea)
* (edit) hadoop-tools/hadoop-azure-datalake/src/site/markdown/index.md
> Fix ADLS doc for credential provider
>
>
> Key: HADOOP-14197
> URL: https://issues.apache.org/jira/browse/HADOOP-14197
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: documentation, fs/adl
>Affects Versions: 2.8.0
>Reporter: John Zhuge
>Assignee: John Zhuge
> Fix For: 2.8.1, 3.0.0-alpha3
>
> Attachments: HADOOP-14197.001.patch
>
>
> There are a few errors in section {{Protecting the Credentials with
> Credential Providers}} of {{index.md}}:
> * Should add {{dfs.adls.oauth2.client.id}} instead of
> {{dfs.adls.oauth2.credential}} to the cred store
> * Should add {{dfs.adls.oauth2.access.token.provider.type}} to core-site.xml
> or DistCp command line
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14196) Azure Data Lake doc is missing required config entry
[ https://issues.apache.org/jira/browse/HADOOP-14196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15939120#comment-15939120 ] Hudson commented on HADOOP-14196: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11448 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/11448/]) HADOOP-14196. Azure Data Lake doc is missing required config entry. (cdouglas: rev 03955612b77f250d53b8f7e4268338e5b78a3e66) * (edit) hadoop-tools/hadoop-azure-datalake/src/site/markdown/index.md > Azure Data Lake doc is missing required config entry > > > Key: HADOOP-14196 > URL: https://issues.apache.org/jira/browse/HADOOP-14196 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/adl >Affects Versions: 3.0.0-alpha2 >Reporter: Atul Sikaria >Assignee: Atul Sikaria > Fix For: 2.8.1, 3.0.0-alpha3 > > Attachments: HADOOP-14196-001.patch > > > The index.md for adl file system is missing one of the config entries needed > for setting up OAuth with client credentials. Users need to set the key > dfs.adls.oauth2.access.token.provider.type = ClientCredential, but the > instructions do not say that. > This has led to people not being able to connect to the backend after setting > up a cluster with ADL. > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14224) add dnsutils to Dockerfile to aid in debugging maven repo failures
Allen Wittenauer created HADOOP-14224: - Summary: add dnsutils to Dockerfile to aid in debugging maven repo failures Key: HADOOP-14224 URL: https://issues.apache.org/jira/browse/HADOOP-14224 Project: Hadoop Common Issue Type: Improvement Components: build Affects Versions: 3.0.0-alpha3 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Priority: Trivial It would be useful to have dig to troubleshoot if the Docker-local resolver is working correctly. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14197) Fix ADLS doc for credential provider
[
https://issues.apache.org/jira/browse/HADOOP-14197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas updated HADOOP-14197:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha3
2.8.1
Status: Resolved (was: Patch Available)
+1 I committed this. Thanks John
> Fix ADLS doc for credential provider
>
>
> Key: HADOOP-14197
> URL: https://issues.apache.org/jira/browse/HADOOP-14197
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: documentation, fs/adl
>Affects Versions: 2.8.0
>Reporter: John Zhuge
>Assignee: John Zhuge
> Fix For: 2.8.1, 3.0.0-alpha3
>
> Attachments: HADOOP-14197.001.patch
>
>
> There are a few errors in section {{Protecting the Credentials with
> Credential Providers}} of {{index.md}}:
> * Should add {{dfs.adls.oauth2.client.id}} instead of
> {{dfs.adls.oauth2.credential}} to the cred store
> * Should add {{dfs.adls.oauth2.access.token.provider.type}} to core-site.xml
> or DistCp command line
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14196) Azure Data Lake doc is missing required config entry
[ https://issues.apache.org/jira/browse/HADOOP-14196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated HADOOP-14196: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0-alpha3 2.8.1 Status: Resolved (was: Patch Available) +1 I committed this. Thanks, Atul > Azure Data Lake doc is missing required config entry > > > Key: HADOOP-14196 > URL: https://issues.apache.org/jira/browse/HADOOP-14196 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/adl >Affects Versions: 3.0.0-alpha2 >Reporter: Atul Sikaria >Assignee: Atul Sikaria > Fix For: 2.8.1, 3.0.0-alpha3 > > Attachments: HADOOP-14196-001.patch > > > The index.md for adl file system is missing one of the config entries needed > for setting up OAuth with client credentials. Users need to set the key > dfs.adls.oauth2.access.token.provider.type = ClientCredential, but the > instructions do not say that. > This has led to people not being able to connect to the backend after setting > up a cluster with ADL. > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13887) Support for client-side encryption in S3A file system
[ https://issues.apache.org/jira/browse/HADOOP-13887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Mazur updated HADOOP-13887: Attachment: HADOOP-13897-branch-2-010.patch Style, whitespaces, and license > Support for client-side encryption in S3A file system > - > > Key: HADOOP-13887 > URL: https://issues.apache.org/jira/browse/HADOOP-13887 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Jeeyoung Kim >Assignee: Igor Mazur >Priority: Minor > Attachments: HADOOP-13887-002.patch, HADOOP-13887-007.patch, > HADOOP-13887-branch-2-003.patch, HADOOP-13897-branch-2-004.patch, > HADOOP-13897-branch-2-005.patch, HADOOP-13897-branch-2-006.patch, > HADOOP-13897-branch-2-008.patch, HADOOP-13897-branch-2-009.patch, > HADOOP-13897-branch-2-010.patch, HADOOP-14171-001.patch > > > Expose the client-side encryption option documented in Amazon S3 > documentation - > http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingClientSideEncryption.html > Currently this is not exposed in Hadoop but it is exposed as an option in AWS > Java SDK, which Hadoop currently includes. It should be trivial to propagate > this as a parameter passed to the S3client used in S3AFileSystem.java -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14135) Remove URI parameter in AWSCredentialProvider constructors
[
https://issues.apache.org/jira/browse/HADOOP-14135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15939038#comment-15939038
]
Hudson commented on HADOOP-14135:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11447 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/11447/])
HADOOP-14135. Remove URI parameter in AWSCredentialProvider (liuml07: rev
2e30aa72e01de7b5774fcb312406a393221e0908)
* (edit)
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AAWSCredentialsProvider.java
* (edit)
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/MockS3ClientFactory.java
* (edit)
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ClientFactory.java
* (edit)
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
* (edit)
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java
* (edit)
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ATemporaryCredentials.java
* (edit)
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/SimpleAWSCredentialsProvider.java
* (edit)
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AAWSCredentialsProvider.java
* (edit)
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/TemporaryAWSCredentialsProvider.java
> Remove URI parameter in AWSCredentialProvider constructors
> --
>
> Key: HADOOP-14135
> URL: https://issues.apache.org/jira/browse/HADOOP-14135
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Fix For: 2.8.1, 3.0.0-alpha3
>
> Attachments: HADOOP-14135.000.patch, HADOOP-14135.001.patch,
> HADOOP-14135.002.patch, HADOOP-14135.003.patch
>
>
> This was from comment in [HADOOP-13252].
> It looks like the URI parameter is not needed for our AWSCredentialProvider
> constructors. This was useful because we relied on URI parameter for
> retrieving user:pass. Now in binding URIs, we have
> {code}
> 279 S3xLoginHelper.Login creds = getAWSAccessKeys(binding, conf);
> 280 credentials.add(new BasicAWSCredentialsProvider(
> 281 creds.getUser(), creds.getPassword()));
> {code}
> This way, we only need configuration object (if necessary) for all
> AWSCredentialProvider implementations. The benefit is that, if we create
> AWSCredentialProvider list for DynamoDB, we don't have to pass down the
> associated file system URI. This might be useful to S3Guard tools.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14222) Create specialized IOException subclass to represent closed filesystem
[ https://issues.apache.org/jira/browse/HADOOP-14222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15939026#comment-15939026 ] Steve Loughran commented on HADOOP-14222: - + to be precise, this is about a closed connection. We can also raise it in all checks for input/output streams being open > Create specialized IOException subclass to represent closed filesystem > -- > > Key: HADOOP-14222 > URL: https://issues.apache.org/jira/browse/HADOOP-14222 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Ted Yu > Labels: newbie > > I was working on HBASE-17287 where hbase master didn't recognize that file > system had closed due to extended unavailability of hdfs. > Chatting with [~steve_l], he suggested creating IOException subclass to > represent closed filesystem so that downstream projects don't have to rely on > the specific exception message. > The string in existing exception message can't be changed. > We should add clear comment around that part to avoid breakage. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13786) Add S3Guard committer for zero-rename commits to consistent S3 endpoints
[
https://issues.apache.org/jira/browse/HADOOP-13786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15939015#comment-15939015
]
Steve Loughran commented on HADOOP-13786:
-
And for the very curious, here's a trace of a test run of the latest code doing
some simple {{sparkContact.makeRDD(1 to
count).saveAsTextFile("s3a:///something")}} with the directory committer.
There's a job committer and task committer in the logs, hence the role log info
to work out whats happening. fs.s3a mostly at debug, except for
instrumentation. FS stats exclude all the info on the uploads because they
(currently) go straight through the s3a client.
{code}
---
2017-03-23 18:42:41,455 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerGetFileStatus(1782)) - Getting
path status for
s3a://hwdev-steve-new/spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api
(spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api)
2017-03-23 18:42:42,024 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG s3a.S3AFileSystem (S3AFileSystem.java:s3GetFileStatus(1905)) - Not Found:
s3a://hwdev-steve-new/spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api
2017-03-23 18:42:42,024 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG s3a.S3AFileSystem (S3AFileSystem.java:delete(1418)) - Couldn't delete
s3a://hwdev-steve-new/spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api
- does not exist
2017-03-23 18:42:42,024 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
s3.S3ANumbersSuiteV2APISuite (CloudLogging.scala:logInfo(56)) - Switching to
local file:// fs for default FS
2017-03-23 18:42:42,098 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
spark.SparkContext (Logging.scala:logInfo(54)) - Running Spark version
2.2.0-SNAPSHOT
2017-03-23 18:42:42,133 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
spark.SparkContext (Logging.scala:logInfo(54)) - Submitted application: test
2017-03-23 18:42:42,147 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
spark.SecurityManager (Logging.scala:logInfo(54)) - Changing view acls to:
stevel
2017-03-23 18:42:42,147 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
spark.SecurityManager (Logging.scala:logInfo(54)) - Changing modify acls to:
stevel
2017-03-23 18:42:42,148 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
spark.SecurityManager (Logging.scala:logInfo(54)) - Changing view acls groups
to:
2017-03-23 18:42:42,149 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
spark.SecurityManager (Logging.scala:logInfo(54)) - Changing modify acls
groups to:
2017-03-23 18:42:42,150 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
spark.SecurityManager (Logging.scala:logInfo(54)) - SecurityManager:
authentication disabled; ui acls disabled; users with view permissions:
Set(stevel); groups with view permissions: Set(); users with modify
permissions: Set(stevel); groups with modify permissions: Set()
2017-03-23 18:42:42,369 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
util.Utils (Logging.scala:logInfo(54)) - Successfully started service
'sparkDriver' on port 54392.
2017-03-23 18:42:42,389 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
spark.SparkEnv (Logging.scala:logInfo(54)) - Registering MapOutputTracker
2017-03-23 18:42:42,402 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
spark.SparkEnv (Logging.scala:logInfo(54)) - Registering BlockManagerMaster
2017-03-23 18:42:42,405 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
storage.BlockManagerMasterEndpoint (Logging.scala:logInfo(54)) - Using
org.apache.spark.storage.DefaultTopologyMapper for getting topology information
2017-03-23 18:42:42,405 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
storage.BlockManagerMasterEndpoint (Logging.scala:logInfo(54)) -
BlockManagerMasterEndpoint up
2017-03-23 18:42:42,412 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
storage.DiskBlockManager (Logging.scala:logInfo(54)) - Created local directory
at
/Users/stevel/Projects/sparkwork/spark-cloud-examples/cloud-examples/target/tmp/blockmgr-9ab575c8-964f-4cb7-96cb-e5c6efc04d60
2017-03-23 18:42:42,429 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
memory.MemoryStore (Logging.scala:logInfo(54)) - MemoryStore started with
capacity 2004.6 MB
2017-03-23 18:42:42,466 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
spark.SparkEnv (Logging.scala:logInfo(54)) - Registering
OutputCommitCoordinator
2017-03-23 18:42:42,579 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
util.Utils (Logging.scala:logInfo(54)) - Successfully started service
'SparkUI' on port 4040.
2017-03-23 18:42:42,606 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
ui.SparkUI (Logging.scala:logInfo(54)) - Bound SparkUI to
[jira] [Updated] (HADOOP-14135) Remove URI parameter in AWSCredentialProvider constructors
[
https://issues.apache.org/jira/browse/HADOOP-14135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HADOOP-14135:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha3
2.8.1
Status: Resolved (was: Patch Available)
Committed to {{branch-2.8}}, {{branch-2}} and {{trunk}} branches. Thanks for
your review, [~mackrorysd] and [~ste...@apache.org].
> Remove URI parameter in AWSCredentialProvider constructors
> --
>
> Key: HADOOP-14135
> URL: https://issues.apache.org/jira/browse/HADOOP-14135
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Fix For: 2.8.1, 3.0.0-alpha3
>
> Attachments: HADOOP-14135.000.patch, HADOOP-14135.001.patch,
> HADOOP-14135.002.patch, HADOOP-14135.003.patch
>
>
> This was from comment in [HADOOP-13252].
> It looks like the URI parameter is not needed for our AWSCredentialProvider
> constructors. This was useful because we relied on URI parameter for
> retrieving user:pass. Now in binding URIs, we have
> {code}
> 279 S3xLoginHelper.Login creds = getAWSAccessKeys(binding, conf);
> 280 credentials.add(new BasicAWSCredentialsProvider(
> 281 creds.getUser(), creds.getPassword()));
> {code}
> This way, we only need configuration object (if necessary) for all
> AWSCredentialProvider implementations. The benefit is that, if we create
> AWSCredentialProvider list for DynamoDB, we don't have to pass down the
> associated file system URI. This might be useful to S3Guard tools.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13786) Add S3Guard committer for zero-rename commits to consistent S3 endpoints
[ https://issues.apache.org/jira/browse/HADOOP-13786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13786: Status: Patch Available (was: Open) > Add S3Guard committer for zero-rename commits to consistent S3 endpoints > > > Key: HADOOP-13786 > URL: https://issues.apache.org/jira/browse/HADOOP-13786 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3 >Affects Versions: HADOOP-13345 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13786-HADOOP-13345-001.patch, > HADOOP-13786-HADOOP-13345-002.patch, HADOOP-13786-HADOOP-13345-003.patch, > HADOOP-13786-HADOOP-13345-004.patch, HADOOP-13786-HADOOP-13345-005.patch, > HADOOP-13786-HADOOP-13345-006.patch, HADOOP-13786-HADOOP-13345-006.patch, > HADOOP-13786-HADOOP-13345-007.patch, HADOOP-13786-HADOOP-13345-009.patch, > HADOOP-13786-HADOOP-13345-010.patch, HADOOP-13786-HADOOP-13345-011.patch, > HADOOP-13786-HADOOP-13345-012.patch, HADOOP-13786-HADOOP-13345-013.patch, > HADOOP-13786-HADOOP-13345-015.patch, HADOOP-13786-HADOOP-13345-016.patch, > HADOOP-13786-HADOOP-13345-017.patch, HADOOP-13786-HADOOP-13345-018.patch, > HADOOP-13786-HADOOP-13345-019.patch, s3committer-master.zip > > > A goal of this code is "support O(1) commits to S3 repositories in the > presence of failures". Implement it, including whatever is needed to > demonstrate the correctness of the algorithm. (that is, assuming that s3guard > provides a consistent view of the presence/absence of blobs, show that we can > commit directly). > I consider ourselves free to expose the blobstore-ness of the s3 output > streams (ie. not visible until the close()), if we need to use that to allow > us to abort commit operations. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13786) Add S3Guard committer for zero-rename commits to consistent S3 endpoints
[ https://issues.apache.org/jira/browse/HADOOP-13786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13786: Attachment: HADOOP-13786-HADOOP-13345-019.patch HADOOP-13786 patch 019: cleanup of jobs & tasks * Lots of work on cleanup, including abortjob purging all pending uploads against a dest. Without that, there's actually a problem in staging aborts. But what? testRecoveryAndCleanup was finding the problem * this is starting to use commitActions in Staging tests, complicating mocking a bit (more methods need to be handled/stubbed). * a bit more docs on lifecycle, but not enough. * still confusion about when to use (deprecated) cleanup() calls vs abort? * track "role" of an instance, included in logs and exceptions. This makes a big difference in IT tests downstream where there is >1 role/instance per process and you want to identify which is doing work. Sort term todos: * IT test on commit contention, verify that the committed output completes * an IT test of a real MR job * more testing in my downstream module * better specification of what's going on with the commit protocol > Add S3Guard committer for zero-rename commits to consistent S3 endpoints > > > Key: HADOOP-13786 > URL: https://issues.apache.org/jira/browse/HADOOP-13786 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3 >Affects Versions: HADOOP-13345 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13786-HADOOP-13345-001.patch, > HADOOP-13786-HADOOP-13345-002.patch, HADOOP-13786-HADOOP-13345-003.patch, > HADOOP-13786-HADOOP-13345-004.patch, HADOOP-13786-HADOOP-13345-005.patch, > HADOOP-13786-HADOOP-13345-006.patch, HADOOP-13786-HADOOP-13345-006.patch, > HADOOP-13786-HADOOP-13345-007.patch, HADOOP-13786-HADOOP-13345-009.patch, > HADOOP-13786-HADOOP-13345-010.patch, HADOOP-13786-HADOOP-13345-011.patch, > HADOOP-13786-HADOOP-13345-012.patch, HADOOP-13786-HADOOP-13345-013.patch, > HADOOP-13786-HADOOP-13345-015.patch, HADOOP-13786-HADOOP-13345-016.patch, > HADOOP-13786-HADOOP-13345-017.patch, HADOOP-13786-HADOOP-13345-018.patch, > HADOOP-13786-HADOOP-13345-019.patch, s3committer-master.zip > > > A goal of this code is "support O(1) commits to S3 repositories in the > presence of failures". Implement it, including whatever is needed to > demonstrate the correctness of the algorithm. (that is, assuming that s3guard > provides a consistent view of the presence/absence of blobs, show that we can > commit directly). > I consider ourselves free to expose the blobstore-ness of the s3 output > streams (ie. not visible until the close()), if we need to use that to allow > us to abort commit operations. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13786) Add S3Guard committer for zero-rename commits to consistent S3 endpoints
[
https://issues.apache.org/jira/browse/HADOOP-13786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15938978#comment-15938978
]
Aaron Fabbri commented on HADOOP-13786:
---
{quote}
How would a webex at 09:00 PST work next week
{quote}
Works for me.. 10 PST is good as well.
> Add S3Guard committer for zero-rename commits to consistent S3 endpoints
>
>
> Key: HADOOP-13786
> URL: https://issues.apache.org/jira/browse/HADOOP-13786
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/s3
>Affects Versions: HADOOP-13345
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-13786-HADOOP-13345-001.patch,
> HADOOP-13786-HADOOP-13345-002.patch, HADOOP-13786-HADOOP-13345-003.patch,
> HADOOP-13786-HADOOP-13345-004.patch, HADOOP-13786-HADOOP-13345-005.patch,
> HADOOP-13786-HADOOP-13345-006.patch, HADOOP-13786-HADOOP-13345-006.patch,
> HADOOP-13786-HADOOP-13345-007.patch, HADOOP-13786-HADOOP-13345-009.patch,
> HADOOP-13786-HADOOP-13345-010.patch, HADOOP-13786-HADOOP-13345-011.patch,
> HADOOP-13786-HADOOP-13345-012.patch, HADOOP-13786-HADOOP-13345-013.patch,
> HADOOP-13786-HADOOP-13345-015.patch, HADOOP-13786-HADOOP-13345-016.patch,
> HADOOP-13786-HADOOP-13345-017.patch, HADOOP-13786-HADOOP-13345-018.patch,
> s3committer-master.zip
>
>
> A goal of this code is "support O(1) commits to S3 repositories in the
> presence of failures". Implement it, including whatever is needed to
> demonstrate the correctness of the algorithm. (that is, assuming that s3guard
> provides a consistent view of the presence/absence of blobs, show that we can
> commit directly).
> I consider ourselves free to expose the blobstore-ness of the s3 output
> streams (ie. not visible until the close()), if we need to use that to allow
> us to abort commit operations.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13786) Add S3Guard committer for zero-rename commits to consistent S3 endpoints
[ https://issues.apache.org/jira/browse/HADOOP-13786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13786: Status: Open (was: Patch Available) > Add S3Guard committer for zero-rename commits to consistent S3 endpoints > > > Key: HADOOP-13786 > URL: https://issues.apache.org/jira/browse/HADOOP-13786 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3 >Affects Versions: HADOOP-13345 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13786-HADOOP-13345-001.patch, > HADOOP-13786-HADOOP-13345-002.patch, HADOOP-13786-HADOOP-13345-003.patch, > HADOOP-13786-HADOOP-13345-004.patch, HADOOP-13786-HADOOP-13345-005.patch, > HADOOP-13786-HADOOP-13345-006.patch, HADOOP-13786-HADOOP-13345-006.patch, > HADOOP-13786-HADOOP-13345-007.patch, HADOOP-13786-HADOOP-13345-009.patch, > HADOOP-13786-HADOOP-13345-010.patch, HADOOP-13786-HADOOP-13345-011.patch, > HADOOP-13786-HADOOP-13345-012.patch, HADOOP-13786-HADOOP-13345-013.patch, > HADOOP-13786-HADOOP-13345-015.patch, HADOOP-13786-HADOOP-13345-016.patch, > HADOOP-13786-HADOOP-13345-017.patch, HADOOP-13786-HADOOP-13345-018.patch, > s3committer-master.zip > > > A goal of this code is "support O(1) commits to S3 repositories in the > presence of failures". Implement it, including whatever is needed to > demonstrate the correctness of the algorithm. (that is, assuming that s3guard > provides a consistent view of the presence/absence of blobs, show that we can > commit directly). > I consider ourselves free to expose the blobstore-ness of the s3 output > streams (ie. not visible until the close()), if we need to use that to allow > us to abort commit operations. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14135) Remove URI parameter in AWSCredentialProvider constructors
[
https://issues.apache.org/jira/browse/HADOOP-14135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HADOOP-14135:
---
Target Version/s: 2.8.1, 3.0.0-alpha3 (was: 2.9.0, 3.0.0-alpha3)
> Remove URI parameter in AWSCredentialProvider constructors
> --
>
> Key: HADOOP-14135
> URL: https://issues.apache.org/jira/browse/HADOOP-14135
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HADOOP-14135.000.patch, HADOOP-14135.001.patch,
> HADOOP-14135.002.patch, HADOOP-14135.003.patch
>
>
> This was from comment in [HADOOP-13252].
> It looks like the URI parameter is not needed for our AWSCredentialProvider
> constructors. This was useful because we relied on URI parameter for
> retrieving user:pass. Now in binding URIs, we have
> {code}
> 279 S3xLoginHelper.Login creds = getAWSAccessKeys(binding, conf);
> 280 credentials.add(new BasicAWSCredentialsProvider(
> 281 creds.getUser(), creds.getPassword()));
> {code}
> This way, we only need configuration object (if necessary) for all
> AWSCredentialProvider implementations. The benefit is that, if we create
> AWSCredentialProvider list for DynamoDB, we don't have to pass down the
> associated file system URI. This might be useful to S3Guard tools.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-10101) Update guava dependency to the latest version
[ https://issues.apache.org/jira/browse/HADOOP-10101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15938962#comment-15938962 ] Tsz Wo Nicholas Sze commented on HADOOP-10101: -- Patch looks good. Thanks for working on this well known hard issue -- Hadoop is stuck on Guava 11. Please take a look the javac/checkstyle warnings. The test failures seem unrelated. Please take a look as well. Thanks. > Update guava dependency to the latest version > - > > Key: HADOOP-10101 > URL: https://issues.apache.org/jira/browse/HADOOP-10101 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.0.0-alpha2 >Reporter: Rakesh R >Assignee: Tsuyoshi Ozawa > Labels: BB2015-05-TBR > Attachments: HADOOP-10101-002.patch, HADOOP-10101-004.patch, > HADOOP-10101-005.patch, HADOOP-10101-006.patch, HADOOP-10101-007.patch, > HADOOP-10101-008.patch, HADOOP-10101-009.patch, HADOOP-10101-009.patch, > HADOOP-10101-010.patch, HADOOP-10101-010.patch, HADOOP-10101-011.patch, > HADOOP-10101.012.patch, HADOOP-10101.013.patch, HADOOP-10101.014.patch, > HADOOP-10101.015.patch, HADOOP-10101.patch, HADOOP-10101.patch > > > The existing guava version is 11.0.2 which is quite old. This issue tries to > update the version to as latest version as possible. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13715) Add isErasureCoded() API to FileStatus class
[
https://issues.apache.org/jira/browse/HADOOP-13715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manoj Govindassamy updated HADOOP-13715:
Attachment: HADOOP-13715.05.patch
Good catch [~andrew.wang]. My apologies, copy paste error. But, this uncovered
a missing fix in {{FsOperations}} which is needed for httpfs. Thanks. Attached
the v05 with FsOperations and BaseTestHttpFSWith fix. Please take a look.
Filed HADOOP-14223 to track FileStatus#toString() enhancements. Thanks
[~ste...@apache.org]
> Add isErasureCoded() API to FileStatus class
>
>
> Key: HADOOP-13715
> URL: https://issues.apache.org/jira/browse/HADOOP-13715
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Affects Versions: 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Manoj Govindassamy
>Priority: Blocker
> Labels: hdfs-ec-3.0-must-do
> Attachments: HADOOP-13715.01.patch, HADOOP-13715.02.patch,
> HADOOP-13715.03.patch, HADOOP-13715.04.patch, HADOOP-13715.05.patch
>
>
> Per the discussion in
> [HDFS-10971|https://issues.apache.org/jira/browse/HDFS-10971?focusedCommentId=15567108=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15567108]
> I would like to add a new API {{isErasureCoded()}} to {{FileStatus}} so that
> tools and downstream applications can tell if it needs to treat a file
> differently.
> Hadoop tools that can benefit from this effort include: distcp and
> teragen/terasort.
> Downstream applications such as flume or hbase may also benefit from it.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14223) Extend FileStatus#toString() to include details like Erasure Coding and Encryption
Manoj Govindassamy created HADOOP-14223:
---
Summary: Extend FileStatus#toString() to include details like
Erasure Coding and Encryption
Key: HADOOP-14223
URL: https://issues.apache.org/jira/browse/HADOOP-14223
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 3.0.0-alpha1
Reporter: Manoj Govindassamy
Assignee: Manoj Govindassamy
HDFS-6843 and HADOOP-13715 have enhanced {{FileStatus}} to include details on
whether the underlying path is Encrypted and Erasure Coded. The additional
details are embedded in the FsPermission high order bits. It would be really
helpful for debugging if FileStatus#toString() returns these new bits details
along with already existing one.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13887) Support for client-side encryption in S3A file system
[
https://issues.apache.org/jira/browse/HADOOP-13887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15938862#comment-15938862
]
Hadoop QA commented on HADOOP-13887:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 13m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 14 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
8s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
3s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
18s{color} | {color:green} branch-2 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
33s{color} | {color:green} branch-2 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
22s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
25s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
25s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
11s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} branch-2 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
18s{color} | {color:green} branch-2 passed with JDK v1.7.0_121 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
20s{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 5m 20s{color}
| {color:red} root-jdk1.8.0_121 with JDK v1.8.0_121 generated 5 new + 884
unchanged - 3 fixed = 889 total (was 887) {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
37s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 6m 37s{color}
| {color:red} root-jdk1.7.0_121 with JDK v1.7.0_121 generated 4 new + 981
unchanged - 2 fixed = 985 total (was 983) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 25s{color} | {color:orange} root: The patch generated 5 new + 21 unchanged -
2 fixed = 26 total (was 23) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 10 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 38s{color}
| {color:red} hadoop-common in the patch failed with JDK v1.7.0_121. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
23s{color} | {color:green} hadoop-aws in the patch passed with JDK v1.7.0_121.
{color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
21s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
|
[jira] [Commented] (HADOOP-11794) distcp can copy blocks in parallel
[ https://issues.apache.org/jira/browse/HADOOP-11794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15938826#comment-15938826 ] Chris Douglas commented on HADOOP-11794: bq. Once the new patch with all above changes is checked in, we need to back port it to older versions of hadoop, which will be addressed by new JIRAs? This is usually handled in the same ticket, and by cherry-picking the patch. Backporting doesn't usually warrant a new JIRA unless the implementation is significantly different. bq. I am not sure why this API is marked deprecated, it seems a good convenient API to exist. Wonder if we should remove the deprecation? If it was deprecated in 0.21, evidently we're not serious about removing it. > distcp can copy blocks in parallel > -- > > Key: HADOOP-11794 > URL: https://issues.apache.org/jira/browse/HADOOP-11794 > Project: Hadoop Common > Issue Type: Improvement > Components: tools/distcp >Affects Versions: 0.21.0 >Reporter: dhruba borthakur >Assignee: Yongjun Zhang > Attachments: HADOOP-11794.001.patch, HADOOP-11794.002.patch, > HADOOP-11794.003.patch, HADOOP-11794.004.patch, HADOOP-11794.005.patch, > HADOOP-11794.006.patch, HADOOP-11794.007.patch, HADOOP-11794.008.patch, > HADOOP-11794.009.patch, HADOOP-11794.010.patch, MAPREDUCE-2257.patch > > > The minimum unit of work for a distcp task is a file. We have files that are > greater than 1 TB with a block size of 1 GB. If we use distcp to copy these > files, the tasks either take a long long long time or finally fails. A better > way for distcp would be to copy all the source blocks in parallel, and then > stich the blocks back to files at the destination via the HDFS Concat API > (HDFS-222) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14222) Create specialized IOException subclass to represent closed filesystem
[ https://issues.apache.org/jira/browse/HADOOP-14222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated HADOOP-14222: -- Labels: newbie (was: ) > Create specialized IOException subclass to represent closed filesystem > -- > > Key: HADOOP-14222 > URL: https://issues.apache.org/jira/browse/HADOOP-14222 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Ted Yu > Labels: newbie > > I was working on HBASE-17287 where hbase master didn't recognize that file > system had closed due to extended unavailability of hdfs. > Chatting with [~steve_l], he suggested creating IOException subclass to > represent closed filesystem so that downstream projects don't have to rely on > the specific exception message. > The string in existing exception message can't be changed. > We should add clear comment around that part to avoid breakage. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14222) Create specialized IOException subclass to represent closed filesystem
Ted Yu created HADOOP-14222: --- Summary: Create specialized IOException subclass to represent closed filesystem Key: HADOOP-14222 URL: https://issues.apache.org/jira/browse/HADOOP-14222 Project: Hadoop Common Issue Type: Improvement Reporter: Ted Yu I was working on HBASE-17287 where hbase master didn't recognize that file system had closed due to extended unavailability of hdfs. Chatting with [~steve_l], he suggested creating IOException subclass to represent closed filesystem so that downstream projects don't have to rely on the specific exception message. The string in existing exception message can't be changed. We should add clear comment around that part to avoid breakage. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11794) distcp can copy blocks in parallel
[ https://issues.apache.org/jira/browse/HADOOP-11794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15938768#comment-15938768 ] Yongjun Zhang commented on HADOOP-11794: Uploaded rev10 to address remaining stuff. About the API mentioned in my last comment, it turned out we don't need to use it so let's leave it as is. Thanks. > distcp can copy blocks in parallel > -- > > Key: HADOOP-11794 > URL: https://issues.apache.org/jira/browse/HADOOP-11794 > Project: Hadoop Common > Issue Type: Improvement > Components: tools/distcp >Affects Versions: 0.21.0 >Reporter: dhruba borthakur >Assignee: Yongjun Zhang > Attachments: HADOOP-11794.001.patch, HADOOP-11794.002.patch, > HADOOP-11794.003.patch, HADOOP-11794.004.patch, HADOOP-11794.005.patch, > HADOOP-11794.006.patch, HADOOP-11794.007.patch, HADOOP-11794.008.patch, > HADOOP-11794.009.patch, HADOOP-11794.010.patch, MAPREDUCE-2257.patch > > > The minimum unit of work for a distcp task is a file. We have files that are > greater than 1 TB with a block size of 1 GB. If we use distcp to copy these > files, the tasks either take a long long long time or finally fails. A better > way for distcp would be to copy all the source blocks in parallel, and then > stich the blocks back to files at the destination via the HDFS Concat API > (HDFS-222) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-11794) distcp can copy blocks in parallel
[ https://issues.apache.org/jira/browse/HADOOP-11794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongjun Zhang updated HADOOP-11794: --- Attachment: HADOOP-11794.010.patch > distcp can copy blocks in parallel > -- > > Key: HADOOP-11794 > URL: https://issues.apache.org/jira/browse/HADOOP-11794 > Project: Hadoop Common > Issue Type: Improvement > Components: tools/distcp >Affects Versions: 0.21.0 >Reporter: dhruba borthakur >Assignee: Yongjun Zhang > Attachments: HADOOP-11794.001.patch, HADOOP-11794.002.patch, > HADOOP-11794.003.patch, HADOOP-11794.004.patch, HADOOP-11794.005.patch, > HADOOP-11794.006.patch, HADOOP-11794.007.patch, HADOOP-11794.008.patch, > HADOOP-11794.009.patch, HADOOP-11794.010.patch, MAPREDUCE-2257.patch > > > The minimum unit of work for a distcp task is a file. We have files that are > greater than 1 TB with a block size of 1 GB. If we use distcp to copy these > files, the tasks either take a long long long time or finally fails. A better > way for distcp would be to copy all the source blocks in parallel, and then > stich the blocks back to files at the destination via the HDFS Concat API > (HDFS-222) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14213) Move Configuration runtime check for hadoop-site.xml to initialization
[ https://issues.apache.org/jira/browse/HADOOP-14213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15938751#comment-15938751 ] Hudson commented on HADOOP-14213: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11446 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/11446/]) HADOOP-14213. Move Configuration runtime check for hadoop-site.xml to (raviprak: rev 595f62e362c08704d6fb692e21c97b512bc7ec49) * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java > Move Configuration runtime check for hadoop-site.xml to initialization > -- > > Key: HADOOP-14213 > URL: https://issues.apache.org/jira/browse/HADOOP-14213 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.8.0 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Fix For: 2.9.0, 3.0.0-beta1 > > Attachments: HADOOP-14213.1.patch, HADOOP-14213.2.patch > > > Each Configuration object that loads defaults checks for hadoop-site.xml. It > has been long deprecated and is not present in most if not nearly all > installations. The getResource check for hadoop-site.xml has to check the > entire classpath since it is not found. This jira proposes to 1) either > remove hadoop-site.xml as a default resource or 2) move the check to static > initialization of the class so the performance hit is only taken once. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14213) Move Configuration runtime check for hadoop-site.xml to initialization
[ https://issues.apache.org/jira/browse/HADOOP-14213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash updated HADOOP-14213: -- Resolution: Fixed Release Note: Move the check for hadoop-site.xml to static initialization of the Configuration class. Status: Resolved (was: Patch Available) > Move Configuration runtime check for hadoop-site.xml to initialization > -- > > Key: HADOOP-14213 > URL: https://issues.apache.org/jira/browse/HADOOP-14213 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.8.0 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Fix For: 2.9.0, 3.0.0-beta1 > > Attachments: HADOOP-14213.1.patch, HADOOP-14213.2.patch > > > Each Configuration object that loads defaults checks for hadoop-site.xml. It > has been long deprecated and is not present in most if not nearly all > installations. The getResource check for hadoop-site.xml has to check the > entire classpath since it is not found. This jira proposes to 1) either > remove hadoop-site.xml as a default resource or 2) move the check to static > initialization of the class so the performance hit is only taken once. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14213) Move Configuration runtime check for hadoop-site.xml to initialization
[ https://issues.apache.org/jira/browse/HADOOP-14213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash updated HADOOP-14213: -- Affects Version/s: 2.8.0 Target Version/s: 2.9.0, 3.0.0-beta1 Fix Version/s: 3.0.0-beta1 2.9.0 > Move Configuration runtime check for hadoop-site.xml to initialization > -- > > Key: HADOOP-14213 > URL: https://issues.apache.org/jira/browse/HADOOP-14213 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.8.0 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Fix For: 2.9.0, 3.0.0-beta1 > > Attachments: HADOOP-14213.1.patch, HADOOP-14213.2.patch > > > Each Configuration object that loads defaults checks for hadoop-site.xml. It > has been long deprecated and is not present in most if not nearly all > installations. The getResource check for hadoop-site.xml has to check the > entire classpath since it is not found. This jira proposes to 1) either > remove hadoop-site.xml as a default resource or 2) move the check to static > initialization of the class so the performance hit is only taken once. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14220) Add S3GuardTool diagnostics command
Steve Loughran created HADOOP-14220: --- Summary: Add S3GuardTool diagnostics command Key: HADOOP-14220 URL: https://issues.apache.org/jira/browse/HADOOP-14220 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: HADOOP-13345 Reporter: Steve Loughran Add a diagnostics command to s3guard which does whatever we need to diagnose problems for a specific (named) s3a url. This is something which can be attached to bug reports as well as used by developers. * Properties to log (with provenance attribute, which can track bucket overrides: s3guard metastore setup, autocreate, capacity, * table present/absent * # of keys in DDB table for that bucket? * any other stats? -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14221) Add s3guardtool dump command
Steve Loughran created HADOOP-14221:
---
Summary: Add s3guardtool dump command
Key: HADOOP-14221
URL: https://issues.apache.org/jira/browse/HADOOP-14221
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: HADOOP-13345
Reporter: Steve Loughran
Priority: Minor
Add a command Dump the database, perhaps matching a pattern, for diagnostics.
e.g
{code}
s3guard dump s3a://steve4/datasets/queries
{code}
issue: large files, throttling, etc. etc.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14213) Move Configuration runtime check for hadoop-site.xml to initialization
[ https://issues.apache.org/jira/browse/HADOOP-14213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15938700#comment-15938700 ] Ravi Prakash commented on HADOOP-14213: --- Thanks for the clarification Jon! LGTM. +1. Since you have not removed hadoop-site from default resources, my understanding is that this is *not* a backward incompatible change. Will commit shortly to trunk and branch-2. > Move Configuration runtime check for hadoop-site.xml to initialization > -- > > Key: HADOOP-14213 > URL: https://issues.apache.org/jira/browse/HADOOP-14213 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: HADOOP-14213.1.patch, HADOOP-14213.2.patch > > > Each Configuration object that loads defaults checks for hadoop-site.xml. It > has been long deprecated and is not present in most if not nearly all > installations. The getResource check for hadoop-site.xml has to check the > entire classpath since it is not found. This jira proposes to 1) either > remove hadoop-site.xml as a default resource or 2) move the check to static > initialization of the class so the performance hit is only taken once. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11794) distcp can copy blocks in parallel
[
https://issues.apache.org/jira/browse/HADOOP-11794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15938672#comment-15938672
]
Yongjun Zhang commented on HADOOP-11794:
Thanks much all!
[~omkarksa],
All the changes except one you need was already in rev9. I will add it in next
rev.
All:
About
{code}
/**
* Same as this(NameNode.getNNAddress(conf), conf);
* @see #DFSClient(InetSocketAddress, Configuration)
* @deprecated Deprecated at 0.21
*/
@Deprecated
public DFSClient(Configuration conf) throws IOException {
this(DFSUtilClient.getNNAddress(conf), conf);
}
{code}
I am not sure why this API is marked deprecated, it seems a good convenient API
to exist. Wonder if we should remove the deprecation?
Thanks.
> distcp can copy blocks in parallel
> --
>
> Key: HADOOP-11794
> URL: https://issues.apache.org/jira/browse/HADOOP-11794
> Project: Hadoop Common
> Issue Type: Improvement
> Components: tools/distcp
>Affects Versions: 0.21.0
>Reporter: dhruba borthakur
>Assignee: Yongjun Zhang
> Attachments: HADOOP-11794.001.patch, HADOOP-11794.002.patch,
> HADOOP-11794.003.patch, HADOOP-11794.004.patch, HADOOP-11794.005.patch,
> HADOOP-11794.006.patch, HADOOP-11794.007.patch, HADOOP-11794.008.patch,
> HADOOP-11794.009.patch, MAPREDUCE-2257.patch
>
>
> The minimum unit of work for a distcp task is a file. We have files that are
> greater than 1 TB with a block size of 1 GB. If we use distcp to copy these
> files, the tasks either take a long long long time or finally fails. A better
> way for distcp would be to copy all the source blocks in parallel, and then
> stich the blocks back to files at the destination via the HDFS Concat API
> (HDFS-222)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-8830) org.apache.hadoop.security.authentication.server.AuthenticationFilter might be called twice, causing kerberos replay errors
[
https://issues.apache.org/jira/browse/HADOOP-8830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15938608#comment-15938608
]
Hadoop QA commented on HADOOP-8830:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s{color}
| {color:red} HADOOP-8830 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-8830 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12605624/HADOOP-8830.20131027.1.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11897/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> org.apache.hadoop.security.authentication.server.AuthenticationFilter might
> be called twice, causing kerberos replay errors
> ---
>
> Key: HADOOP-8830
> URL: https://issues.apache.org/jira/browse/HADOOP-8830
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.0.1-alpha, 2.1.0-beta, 2.1.1-beta, 2.2.0
>Reporter: Moritz Moeller
>Assignee: Omkar Vinit Joshi
>Priority: Critical
> Labels: BB2015-05-TBR
> Attachments: HADOOP-8830.20131026.1.patch,
> HADOOP-8830.20131027.1.patch
>
>
> AuthenticationFilter.doFilter is called twice (not sure if that is
> intentional or not).
> The second time it is called the ServletRequest is already authenticated,
> i.e. httpRequest.getRemoteUser() returns non-null info.
> If the kerberos authentication is triggered a second time it'll return a
> replay attack exception.
> I solved this by adding a if (httpRequest.getRemoteUser() == null) at the
> very beginning of doFilter.
> Alternatively one can set an attribute on the request, or figure out why
> doFilter is called twice.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14213) Move Configuration runtime check for hadoop-site.xml to initialization
[ https://issues.apache.org/jira/browse/HADOOP-14213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15938572#comment-15938572 ] Jonathan Eagles commented on HADOOP-14213: -- Both TestKDiag and TestDNS are intermittent failures not related to this patch. The overall improvement to Configuration loading on my box is around 2.5% which is very tiny by itself, but a few changes like this will make a big difference. What else is needed to get this patch in? > Move Configuration runtime check for hadoop-site.xml to initialization > -- > > Key: HADOOP-14213 > URL: https://issues.apache.org/jira/browse/HADOOP-14213 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: HADOOP-14213.1.patch, HADOOP-14213.2.patch > > > Each Configuration object that loads defaults checks for hadoop-site.xml. It > has been long deprecated and is not present in most if not nearly all > installations. The getResource check for hadoop-site.xml has to check the > entire classpath since it is not found. This jira proposes to 1) either > remove hadoop-site.xml as a default resource or 2) move the check to static > initialization of the class so the performance hit is only taken once. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13887) Support for client-side encryption in S3A file system
[ https://issues.apache.org/jira/browse/HADOOP-13887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Mazur updated HADOOP-13887: Attachment: HADOOP-13897-branch-2-009.patch * Custom method of encryption * Modified and generalized tests * Initial documentation > Support for client-side encryption in S3A file system > - > > Key: HADOOP-13887 > URL: https://issues.apache.org/jira/browse/HADOOP-13887 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Jeeyoung Kim >Assignee: Igor Mazur >Priority: Minor > Attachments: HADOOP-13887-002.patch, HADOOP-13887-007.patch, > HADOOP-13887-branch-2-003.patch, HADOOP-13897-branch-2-004.patch, > HADOOP-13897-branch-2-005.patch, HADOOP-13897-branch-2-006.patch, > HADOOP-13897-branch-2-008.patch, HADOOP-13897-branch-2-009.patch, > HADOOP-14171-001.patch > > > Expose the client-side encryption option documented in Amazon S3 > documentation - > http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingClientSideEncryption.html > Currently this is not exposed in Hadoop but it is exposed as an option in AWS > Java SDK, which Hadoop currently includes. It should be trivial to propagate > this as a parameter passed to the S3client used in S3AFileSystem.java -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-8830) org.apache.hadoop.security.authentication.server.AuthenticationFilter might be called twice, causing kerberos replay errors
[ https://issues.apache.org/jira/browse/HADOOP-8830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15938542#comment-15938542 ] Burak Gursoy commented on HADOOP-8830: -- Hi, Is this ticket still valid or solved elsewhere like YARN-621 ? > org.apache.hadoop.security.authentication.server.AuthenticationFilter might > be called twice, causing kerberos replay errors > --- > > Key: HADOOP-8830 > URL: https://issues.apache.org/jira/browse/HADOOP-8830 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.0.1-alpha, 2.1.0-beta, 2.1.1-beta, 2.2.0 >Reporter: Moritz Moeller >Assignee: Omkar Vinit Joshi >Priority: Critical > Labels: BB2015-05-TBR > Attachments: HADOOP-8830.20131026.1.patch, > HADOOP-8830.20131027.1.patch > > > AuthenticationFilter.doFilter is called twice (not sure if that is > intentional or not). > The second time it is called the ServletRequest is already authenticated, > i.e. httpRequest.getRemoteUser() returns non-null info. > If the kerberos authentication is triggered a second time it'll return a > replay attack exception. > I solved this by adding a if (httpRequest.getRemoteUser() == null) at the > very beginning of doFilter. > Alternatively one can set an attribute on the request, or figure out why > doFilter is called twice. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13887) Support for client-side encryption in S3A file system
[ https://issues.apache.org/jira/browse/HADOOP-13887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15938397#comment-15938397 ] Igor Mazur edited comment on HADOOP-13887 at 3/23/17 2:20 PM: -- Got it. Will update writeDataset. Going to push today one more patch with another kind of client-side encryption - when instead of usage KMS user provides its own key. Also, yesterday applied this patch to trunk branch, but for some unknown reason, it wasn't processed by Jenkins QA job. About documentation: in next patch will be some initial version. was (Author: igor mazur): Got it. Will update writeDataset. Going to push today one more patch with another kind of client-side encryption - when instead of usage KMS user provides its own key. Also, yesterday applied this patch to trunk branch, but for some unknown, reason, it wasn't processed by Jenkins QA job. About documentation: in next patch will be some initial version. > Support for client-side encryption in S3A file system > - > > Key: HADOOP-13887 > URL: https://issues.apache.org/jira/browse/HADOOP-13887 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Jeeyoung Kim >Assignee: Igor Mazur >Priority: Minor > Attachments: HADOOP-13887-002.patch, HADOOP-13887-007.patch, > HADOOP-13887-branch-2-003.patch, HADOOP-13897-branch-2-004.patch, > HADOOP-13897-branch-2-005.patch, HADOOP-13897-branch-2-006.patch, > HADOOP-13897-branch-2-008.patch, HADOOP-14171-001.patch > > > Expose the client-side encryption option documented in Amazon S3 > documentation - > http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingClientSideEncryption.html > Currently this is not exposed in Hadoop but it is exposed as an option in AWS > Java SDK, which Hadoop currently includes. It should be trivial to propagate > this as a parameter passed to the S3client used in S3AFileSystem.java -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13655) document object store use with fs shell and distcp
[
https://issues.apache.org/jira/browse/HADOOP-13655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinayakumar B updated HADOOP-13655:
---
Fix Version/s: 2.8.0
> document object store use with fs shell and distcp
> --
>
> Key: HADOOP-13655
> URL: https://issues.apache.org/jira/browse/HADOOP-13655
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: documentation, fs, fs/s3
>Affects Versions: 2.7.3
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Fix For: 2.8.0, 2.7.4, 3.0.0-alpha2
>
> Attachments: HADOOP-13655.000.patch
>
>
> There's no specific docs for working with object stores from the {{hadoop
> fs}} shell or in distcp; people either suffer from this (performance,
> billing), or learn through trial and error what to do.
> Add a section in both fs shell and distcp docs covering use with object
> stores.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13887) Support for client-side encryption in S3A file system
[ https://issues.apache.org/jira/browse/HADOOP-13887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15938397#comment-15938397 ] Igor Mazur commented on HADOOP-13887: - Got it. Will update writeDataset. Going to push today one more patch with another kind of client-side encryption - when instead of usage KMS user provides its own key. Also, yesterday applied this patch to trunk branch, but for some unknown, reason, it wasn't processed by Jenkins QA job. About documentation: in next patch will be some initial version. > Support for client-side encryption in S3A file system > - > > Key: HADOOP-13887 > URL: https://issues.apache.org/jira/browse/HADOOP-13887 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Jeeyoung Kim >Assignee: Igor Mazur >Priority: Minor > Attachments: HADOOP-13887-002.patch, HADOOP-13887-007.patch, > HADOOP-13887-branch-2-003.patch, HADOOP-13897-branch-2-004.patch, > HADOOP-13897-branch-2-005.patch, HADOOP-13897-branch-2-006.patch, > HADOOP-13897-branch-2-008.patch, HADOOP-14171-001.patch > > > Expose the client-side encryption option documented in Amazon S3 > documentation - > http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingClientSideEncryption.html > Currently this is not exposed in Hadoop but it is exposed as an option in AWS > Java SDK, which Hadoop currently includes. It should be trivial to propagate > this as a parameter passed to the S3client used in S3AFileSystem.java -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14217) Object Storage: support colon in object path
[
https://issues.apache.org/jira/browse/HADOOP-14217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15938328#comment-15938328
]
Steve Loughran commented on HADOOP-14217:
-
Every filesystem has invalid paths and characters. I Dont' see why we should
explicilty be blocking ":"; in fact I think the format of gcs logs uses it.
What is blocking it? {{Path}} itself? Or {{java.net.URI}}?
> Object Storage: support colon in object path
>
>
> Key: HADOOP-14217
> URL: https://issues.apache.org/jira/browse/HADOOP-14217
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs, fs/oss
>Reporter: Genmao Yu
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14219) RumenToSLS: parsing problem with crashed attempts
[
https://issues.apache.org/jira/browse/HADOOP-14219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15938291#comment-15938291
]
Hadoop QA commented on HADOOP-14219:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 11s{color} | {color:orange} hadoop-tools/hadoop-sls: The patch generated 3
new + 11 unchanged - 1 fixed = 14 total (was 12) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
57s{color} | {color:green} hadoop-sls in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 5s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-14219 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12860130/HADOOP-14219.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 641631b2b84b 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / a5a4867 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11895/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-sls.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11895/testReport/ |
| modules | C: hadoop-tools/hadoop-sls U: hadoop-tools/hadoop-sls |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11895/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> RumenToSLS: parsing problem with crashed attempts
> -
>
> Key: HADOOP-14219
> URL: https://issues.apache.org/jira/browse/HADOOP-14219
> Project: Hadoop Common
> Issue Type: Bug
> Components:
[jira] [Commented] (HADOOP-14135) Remove URI parameter in AWSCredentialProvider constructors
[
https://issues.apache.org/jira/browse/HADOOP-14135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15938287#comment-15938287
]
Steve Loughran commented on HADOOP-14135:
-
LGTM
+1
One thing to consider: there may be someone out there who has a custom
credential provider which is going to break. I'm not worried about that as the
API to do this doesn't actually ship until Hadoop 2.8.0; there aren't any
longstanding users. If we got this into branch-2.8 & hence hadoop -2.8.1, then
we'd have nobody using it in production at all.
> Remove URI parameter in AWSCredentialProvider constructors
> --
>
> Key: HADOOP-14135
> URL: https://issues.apache.org/jira/browse/HADOOP-14135
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HADOOP-14135.000.patch, HADOOP-14135.001.patch,
> HADOOP-14135.002.patch, HADOOP-14135.003.patch
>
>
> This was from comment in [HADOOP-13252].
> It looks like the URI parameter is not needed for our AWSCredentialProvider
> constructors. This was useful because we relied on URI parameter for
> retrieving user:pass. Now in binding URIs, we have
> {code}
> 279 S3xLoginHelper.Login creds = getAWSAccessKeys(binding, conf);
> 280 credentials.add(new BasicAWSCredentialsProvider(
> 281 creds.getUser(), creds.getPassword()));
> {code}
> This way, we only need configuration object (if necessary) for all
> AWSCredentialProvider implementations. The benefit is that, if we create
> AWSCredentialProvider list for DynamoDB, we don't have to pass down the
> associated file system URI. This might be useful to S3Guard tools.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13665) Erasure Coding codec should support fallback coder
[
https://issues.apache.org/jira/browse/HADOOP-13665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15938281#comment-15938281
]
Kai Sasaki commented on HADOOP-13665:
-
[~drankye] Thanks for explanation! It's very clear to me. I'll update
accordingly.
> Erasure Coding codec should support fallback coder
> --
>
> Key: HADOOP-13665
> URL: https://issues.apache.org/jira/browse/HADOOP-13665
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: io
>Reporter: Wei-Chiu Chuang
>Assignee: Kai Sasaki
>Priority: Blocker
> Labels: hdfs-ec-3.0-must-do
> Attachments: HADOOP-13665.01.patch, HADOOP-13665.02.patch,
> HADOOP-13665.03.patch, HADOOP-13665.04.patch, HADOOP-13665.05.patch
>
>
> The current EC codec supports a single coder only (by default pure Java
> implementation). If the native coder is specified but is unavailable, it
> should fallback to pure Java implementation.
> One possible solution is to follow the convention of existing Hadoop native
> codec, such as transport encryption (see {{CryptoCodec.java}}). It supports
> fallback by specifying two or multiple coders as the value of property, and
> loads coders in order.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13887) Support for client-side encryption in S3A file system
[
https://issues.apache.org/jira/browse/HADOOP-13887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15938268#comment-15938268
]
Steve Loughran commented on HADOOP-13887:
-
I like how this is coming together; it's always good to keep checkstyle quiet,
even though it complains too much for my personal liking.
On the change to {{writeDataset();}}, how about retaining a method with the
original signature, and calling the new one with the last arg set to true? That
way: fewer changes to the codebase, anything downstream using CTU (I'm the
likeliest culprit) doesn't break.
Other than that, I don't see any more code changes. Any other reviewers want to
add. Anyone tested it yet?
I'm thinking of end user docs. That's something we could just collaborate on in
the comments here, rather than iterate through the code patches, which are
pretty stable to me. As well as some instructions on what to do, and warnings,
it'd be good to have a bit in the troubleshooting section. I can see various
problems arising:
* encryption enabled, no key
* encryption enabled, wrong key
* encryption enabled, no JCE That crops in kerberos BTW; the KDiag entry point
explicitly tests for it. We could say "use kdiag to look for that".
* encryption enabled, no bouncy castle.
* encryption enabled, object store doesn't support it
* encryption disabled, end data encrypted.
It'd be good to have whatever stack traces you've managed to collect as part of
this, otherwise we can make some more; easily done :)
> Support for client-side encryption in S3A file system
> -
>
> Key: HADOOP-13887
> URL: https://issues.apache.org/jira/browse/HADOOP-13887
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Jeeyoung Kim
>Assignee: Igor Mazur
>Priority: Minor
> Attachments: HADOOP-13887-002.patch, HADOOP-13887-007.patch,
> HADOOP-13887-branch-2-003.patch, HADOOP-13897-branch-2-004.patch,
> HADOOP-13897-branch-2-005.patch, HADOOP-13897-branch-2-006.patch,
> HADOOP-13897-branch-2-008.patch, HADOOP-14171-001.patch
>
>
> Expose the client-side encryption option documented in Amazon S3
> documentation -
> http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingClientSideEncryption.html
> Currently this is not exposed in Hadoop but it is exposed as an option in AWS
> Java SDK, which Hadoop currently includes. It should be trivial to propagate
> this as a parameter passed to the S3client used in S3AFileSystem.java
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14219) RumenToSLS: parsing problem with crashed attempts
[
https://issues.apache.org/jira/browse/HADOOP-14219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15938235#comment-15938235
]
Julien Vaudour edited comment on HADOOP-14219 at 3/23/17 1:04 PM:
--
The proposed patched consider all timestamp as the more generic class
java.lang.Number
It also ignores task attempts having null hostName
was (Author: seneque):
The proposed patched consider all as the more generic class java.lang.Number
It also ignores task attempts having null hostName
> RumenToSLS: parsing problem with crashed attempts
> -
>
> Key: HADOOP-14219
> URL: https://issues.apache.org/jira/browse/HADOOP-14219
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools
>Affects Versions: 2.6.0
>Reporter: Julien Vaudour
>Priority: Minor
> Attachments: HADOOP-14219.patch
>
>
> In case of crashed task attempts, we may have in rumen logs task attempts
> with null hostName and finishTime defined to -1
> for example
> {code}
>{
> "resourceUsageMetrics": {
> "heapUsage": 0,
> "physicalMemoryUsage": 0,
> "virtualMemoryUsage": 0,
> "cumulativeCpuUsage": 0
> },
> "vmemKbytes": [],
> "physMemKbytes": [],
> "cpuUsages": [],
> "clockSplits": [],
> "location": null,
> "sortFinished": -1,
> "shuffleFinished": -1,
> "spilledRecords": -1,
> "reduceOutputRecords": -1,
> "reduceShuffleBytes": -1,
> "fileBytesRead": -1,
> "hdfsBytesWritten": -1,
> "hdfsBytesRead": -1,
> "hostName": null,
> "finishTime": -1,
> "startTime": 1489619193378,
> "result": null,
> "attemptID": "attempt_1488896259152_410442_r_15_1",
> "fileBytesWritten": -1,
> "mapInputRecords": -1,
> "mapInputBytes": -1,
> "mapOutputBytes": -1,
> "mapOutputRecords": -1,
> "combineInputRecords": -1,
> "reduceInputGroups": -1,
> "reduceInputRecords": -1
> }
> {code}
> Jackson parser will automatically consider -1 as a java.lang.Integer. However
> RumenToSLSConverter make the assumption than jackson has deserialize all
> timstamp as instance of java.lang.Long, resulting in a ClassCastException.
> RumenToSLSConverter also make the assumption that hostName is not null, so we
> can also have a NullPointerException.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13786) Add S3Guard committer for zero-rename commits to consistent S3 endpoints
[ https://issues.apache.org/jira/browse/HADOOP-13786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15938240#comment-15938240 ] Steve Loughran commented on HADOOP-13786: - Thomas, I'm aware of your needs. I just put that lock in there for now as my integration tests did have s3a as the default FS, and the debug logs had me confused. Will make it an option. How would a webex at 09:00 PST work next week; the EU will have switched TZ so it'll be 17:00 my time,, 18:00 thomas's. Or I can slip to 10:00 PST if that suits > Add S3Guard committer for zero-rename commits to consistent S3 endpoints > > > Key: HADOOP-13786 > URL: https://issues.apache.org/jira/browse/HADOOP-13786 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3 >Affects Versions: HADOOP-13345 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13786-HADOOP-13345-001.patch, > HADOOP-13786-HADOOP-13345-002.patch, HADOOP-13786-HADOOP-13345-003.patch, > HADOOP-13786-HADOOP-13345-004.patch, HADOOP-13786-HADOOP-13345-005.patch, > HADOOP-13786-HADOOP-13345-006.patch, HADOOP-13786-HADOOP-13345-006.patch, > HADOOP-13786-HADOOP-13345-007.patch, HADOOP-13786-HADOOP-13345-009.patch, > HADOOP-13786-HADOOP-13345-010.patch, HADOOP-13786-HADOOP-13345-011.patch, > HADOOP-13786-HADOOP-13345-012.patch, HADOOP-13786-HADOOP-13345-013.patch, > HADOOP-13786-HADOOP-13345-015.patch, HADOOP-13786-HADOOP-13345-016.patch, > HADOOP-13786-HADOOP-13345-017.patch, HADOOP-13786-HADOOP-13345-018.patch, > s3committer-master.zip > > > A goal of this code is "support O(1) commits to S3 repositories in the > presence of failures". Implement it, including whatever is needed to > demonstrate the correctness of the algorithm. (that is, assuming that s3guard > provides a consistent view of the presence/absence of blobs, show that we can > commit directly). > I consider ourselves free to expose the blobstore-ness of the s3 output > streams (ie. not visible until the close()), if we need to use that to allow > us to abort commit operations. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14219) RumenToSLS: parsing problem with crashed attempts
[
https://issues.apache.org/jira/browse/HADOOP-14219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julien Vaudour updated HADOOP-14219:
Status: Patch Available (was: Open)
> RumenToSLS: parsing problem with crashed attempts
> -
>
> Key: HADOOP-14219
> URL: https://issues.apache.org/jira/browse/HADOOP-14219
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools
>Affects Versions: 2.6.0
>Reporter: Julien Vaudour
>Priority: Minor
> Attachments: HADOOP-14219.patch
>
>
> In case of crashed task attempts, we may have in rumen logs task attempts
> with null hostName and finishTime defined to -1
> for example
> {code}
>{
> "resourceUsageMetrics": {
> "heapUsage": 0,
> "physicalMemoryUsage": 0,
> "virtualMemoryUsage": 0,
> "cumulativeCpuUsage": 0
> },
> "vmemKbytes": [],
> "physMemKbytes": [],
> "cpuUsages": [],
> "clockSplits": [],
> "location": null,
> "sortFinished": -1,
> "shuffleFinished": -1,
> "spilledRecords": -1,
> "reduceOutputRecords": -1,
> "reduceShuffleBytes": -1,
> "fileBytesRead": -1,
> "hdfsBytesWritten": -1,
> "hdfsBytesRead": -1,
> "hostName": null,
> "finishTime": -1,
> "startTime": 1489619193378,
> "result": null,
> "attemptID": "attempt_1488896259152_410442_r_15_1",
> "fileBytesWritten": -1,
> "mapInputRecords": -1,
> "mapInputBytes": -1,
> "mapOutputBytes": -1,
> "mapOutputRecords": -1,
> "combineInputRecords": -1,
> "reduceInputGroups": -1,
> "reduceInputRecords": -1
> }
> {code}
> Jackson parser will automatically consider -1 as a java.lang.Integer. However
> RumenToSLSConverter make the assumption than jackson has deserialize all
> timstamp as instance of java.lang.Long, resulting in a ClassCastException.
> RumenToSLSConverter also make the assumption that hostName is not null, so we
> can also have a NullPointerException.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14219) RumenToSLS: parsing problem with crashed attempts
[
https://issues.apache.org/jira/browse/HADOOP-14219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julien Vaudour updated HADOOP-14219:
Attachment: HADOOP-14219.patch
The proposed patched consider all as the more generic class java.lang.Number
It also ignores task attempts having null hostName
> RumenToSLS: parsing problem with crashed attempts
> -
>
> Key: HADOOP-14219
> URL: https://issues.apache.org/jira/browse/HADOOP-14219
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools
>Affects Versions: 2.6.0
>Reporter: Julien Vaudour
>Priority: Minor
> Attachments: HADOOP-14219.patch
>
>
> In case of crashed task attempts, we may have in rumen logs task attempts
> with null hostName and finishTime defined to -1
> for example
> {code}
>{
> "resourceUsageMetrics": {
> "heapUsage": 0,
> "physicalMemoryUsage": 0,
> "virtualMemoryUsage": 0,
> "cumulativeCpuUsage": 0
> },
> "vmemKbytes": [],
> "physMemKbytes": [],
> "cpuUsages": [],
> "clockSplits": [],
> "location": null,
> "sortFinished": -1,
> "shuffleFinished": -1,
> "spilledRecords": -1,
> "reduceOutputRecords": -1,
> "reduceShuffleBytes": -1,
> "fileBytesRead": -1,
> "hdfsBytesWritten": -1,
> "hdfsBytesRead": -1,
> "hostName": null,
> "finishTime": -1,
> "startTime": 1489619193378,
> "result": null,
> "attemptID": "attempt_1488896259152_410442_r_15_1",
> "fileBytesWritten": -1,
> "mapInputRecords": -1,
> "mapInputBytes": -1,
> "mapOutputBytes": -1,
> "mapOutputRecords": -1,
> "combineInputRecords": -1,
> "reduceInputGroups": -1,
> "reduceInputRecords": -1
> }
> {code}
> Jackson parser will automatically consider -1 as a java.lang.Integer. However
> RumenToSLSConverter make the assumption than jackson has deserialize all
> timstamp as instance of java.lang.Long, resulting in a ClassCastException.
> RumenToSLSConverter also make the assumption that hostName is not null, so we
> can also have a NullPointerException.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14120) needless S3AFileSystem.setOptionalPutRequestParameters in S3ABlockOutputStream putObject()
[
https://issues.apache.org/jira/browse/HADOOP-14120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14120:
Resolution: Fixed
Fix Version/s: 2.9.0
Status: Resolved (was: Patch Available)
committed! thanks!
(BTW, on the topic of testing, I did the -Dscale tests too. Why? To make sure
the codepath related to multiple blocks uploaded from the same output stream
got covered. Really we should be running those tests on any patch which touches
BlockOutputStream)
> needless S3AFileSystem.setOptionalPutRequestParameters in
> S3ABlockOutputStream putObject()
> --
>
> Key: HADOOP-14120
> URL: https://issues.apache.org/jira/browse/HADOOP-14120
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Assignee: Yuanbo Liu
>Priority: Minor
> Fix For: 2.9.0
>
> Attachments: HADOOP-14120.001.patch
>
>
> There's a call to {{S3AFileSystem.setOptionalPutRequestParameters()}} in {{
> S3ABlockOutputStream putObject()}}
> The put request has already been created by the FS; this call is only
> superflous and potentially confusing.
> Proposed: cut it, make the {{setOptionalPutRequestParameters()}} method
> private.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14219) RumenToSLS: parsing problem with crashed attempts
Julien Vaudour created HADOOP-14219:
---
Summary: RumenToSLS: parsing problem with crashed attempts
Key: HADOOP-14219
URL: https://issues.apache.org/jira/browse/HADOOP-14219
Project: Hadoop Common
Issue Type: Bug
Components: tools
Affects Versions: 2.6.0
Reporter: Julien Vaudour
Priority: Minor
In case of crashed task attempts, we may have in rumen logs task attempts with
null hostName and finishTime defined to -1
for example
{code}
{
"resourceUsageMetrics": {
"heapUsage": 0,
"physicalMemoryUsage": 0,
"virtualMemoryUsage": 0,
"cumulativeCpuUsage": 0
},
"vmemKbytes": [],
"physMemKbytes": [],
"cpuUsages": [],
"clockSplits": [],
"location": null,
"sortFinished": -1,
"shuffleFinished": -1,
"spilledRecords": -1,
"reduceOutputRecords": -1,
"reduceShuffleBytes": -1,
"fileBytesRead": -1,
"hdfsBytesWritten": -1,
"hdfsBytesRead": -1,
"hostName": null,
"finishTime": -1,
"startTime": 1489619193378,
"result": null,
"attemptID": "attempt_1488896259152_410442_r_15_1",
"fileBytesWritten": -1,
"mapInputRecords": -1,
"mapInputBytes": -1,
"mapOutputBytes": -1,
"mapOutputRecords": -1,
"combineInputRecords": -1,
"reduceInputGroups": -1,
"reduceInputRecords": -1
}
{code}
Jackson parser will automatically consider -1 as a java.lang.Integer. However
RumenToSLSConverter make the assumption than jackson has deserialize all
timstamp as instance of java.lang.Long, resulting in a ClassCastException.
RumenToSLSConverter also make the assumption that hostName is not null, so we
can also have a NullPointerException.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
