[jira] [Commented] (HADOOP-15696) KMS performance regression due to too many open file descriptors after Jetty migration
[
https://issues.apache.org/jira/browse/HADOOP-15696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16594364#comment-16594364
]
Misha Dmitriev commented on HADOOP-15696:
-
Answering my own question: yes, it looks like indeed the KMS client opens a new
HTTP connection for every request, and doesn't seem to close it explicitly. See
methods such in {{KMSClientProvider}} , e.g. {{decryptEncryptedKey()}},
{{getCurrentKey()}}, {{deleteKey()}} etc. They all invoke
{{createConnection()}}, then {{call()}} with this connection to send a request
and get a response, and then I don't see any code that would close that
connection.
It's hard for me to say whether it's by design or by neglect, but anyway, such
a frequent creation of TCP connection and no explicit close/reuse look unusual.
At a minimum, I think it's worth documenting this behavior briefly somewhere,
maybe in the top-level javadoc of the {{KMSClientProvider}} class. I suggest to
do it in the above patch so that this is not left behind.
> KMS performance regression due to too many open file descriptors after Jetty
> migration
> --
>
> Key: HADOOP-15696
> URL: https://issues.apache.org/jira/browse/HADOOP-15696
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 3.0.0-alpha2
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Blocker
> Attachments: HADOOP-15696.001.patch, Screen Shot 2018-08-22 at
> 11.36.16 AM.png, Screen Shot 2018-08-22 at 4.26.51 PM.png, Screen Shot
> 2018-08-22 at 4.26.51 PM.png, Screen Shot 2018-08-22 at 4.27.02 PM.png,
> Screen Shot 2018-08-22 at 4.30.32 PM.png, Screen Shot 2018-08-22 at 4.30.39
> PM.png, Screen Shot 2018-08-24 at 7.08.16 PM.png
>
>
> We recently found KMS performance regressed in Hadoop 3.0, possibly linking
> to the migration from Tomcat to Jetty in HADOOP-13597.
> Symptoms:
> # Hadoop 3.x KMS open file descriptors quickly rises to more than 10 thousand
> under stress, sometimes even exceeds 32K, which is the system limit, causing
> failures for any access to encryption zones. Our internal testing shows the
> openfd number was in the range of a few hundred in Hadoop 2.x, and it
> increases by almost 100x in Hadoop 3.
> # Hadoop 3.x KMS as much as twice the heap size than in Hadoop 2.x. The same
> heap size can go OOM in Hadoop 3.x. Jxray analysis suggests most of them are
> temporary byte arrays associated with open SSL connections.
> # Due to the heap usage, Hadoop 3.x KMS has more frequent GC activities, and
> we observed up to 20% performance reduction due to GC.
> A possible solution is to reduce the idle timeout setting in HttpServer2. It
> is currently hard-coded 10 seconds. By setting it to 1 second, open fds
> dropped from 20 thousand down to 3 thousand in my experiment.
> File this jira to invite open discussion for a solution.
> Credit: [~mi...@cloudera.com] for the proposed Jetty idle timeout remedy;
> [~xiaochen] for digging into this problem.
> Screenshots:
> CDH5 (Hadoop 2) KMS CPU utilization, resident memory and file descriptor
> chart.
> !Screen Shot 2018-08-22 at 4.30.39 PM.png!
> CDH6 (Hadoop 3) KMS CPU utilization, resident memory and file descriptor
> chart.
> !Screen Shot 2018-08-22 at 4.30.32 PM.png!
> CDH5 (Hadoop 2) GC activities on the KMS process
> !Screen Shot 2018-08-22 at 4.26.51 PM.png!
> CDH6 (Hadoop 3) GC activities on the KMS process
> !Screen Shot 2018-08-22 at 4.27.02 PM.png!
> JXray report
> !Screen Shot 2018-08-22 at 11.36.16 AM.png!
> open fd drops from 20 k down to 3k after the proposed change.
> !Screen Shot 2018-08-24 at 7.08.16 PM.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15696) KMS performance regression due to too many open file descriptors after Jetty migration
[ https://issues.apache.org/jira/browse/HADOOP-15696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HADOOP-15696: - Attachment: HADOOP-15696.001.patch > KMS performance regression due to too many open file descriptors after Jetty > migration > -- > > Key: HADOOP-15696 > URL: https://issues.apache.org/jira/browse/HADOOP-15696 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 3.0.0-alpha2 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Blocker > Attachments: HADOOP-15696.001.patch, Screen Shot 2018-08-22 at > 11.36.16 AM.png, Screen Shot 2018-08-22 at 4.26.51 PM.png, Screen Shot > 2018-08-22 at 4.26.51 PM.png, Screen Shot 2018-08-22 at 4.27.02 PM.png, > Screen Shot 2018-08-22 at 4.30.32 PM.png, Screen Shot 2018-08-22 at 4.30.39 > PM.png, Screen Shot 2018-08-24 at 7.08.16 PM.png > > > We recently found KMS performance regressed in Hadoop 3.0, possibly linking > to the migration from Tomcat to Jetty in HADOOP-13597. > Symptoms: > # Hadoop 3.x KMS open file descriptors quickly rises to more than 10 thousand > under stress, sometimes even exceeds 32K, which is the system limit, causing > failures for any access to encryption zones. Our internal testing shows the > openfd number was in the range of a few hundred in Hadoop 2.x, and it > increases by almost 100x in Hadoop 3. > # Hadoop 3.x KMS as much as twice the heap size than in Hadoop 2.x. The same > heap size can go OOM in Hadoop 3.x. Jxray analysis suggests most of them are > temporary byte arrays associated with open SSL connections. > # Due to the heap usage, Hadoop 3.x KMS has more frequent GC activities, and > we observed up to 20% performance reduction due to GC. > A possible solution is to reduce the idle timeout setting in HttpServer2. It > is currently hard-coded 10 seconds. By setting it to 1 second, open fds > dropped from 20 thousand down to 3 thousand in my experiment. > File this jira to invite open discussion for a solution. > Credit: [~mi...@cloudera.com] for the proposed Jetty idle timeout remedy; > [~xiaochen] for digging into this problem. > Screenshots: > CDH5 (Hadoop 2) KMS CPU utilization, resident memory and file descriptor > chart. > !Screen Shot 2018-08-22 at 4.30.39 PM.png! > CDH6 (Hadoop 3) KMS CPU utilization, resident memory and file descriptor > chart. > !Screen Shot 2018-08-22 at 4.30.32 PM.png! > CDH5 (Hadoop 2) GC activities on the KMS process > !Screen Shot 2018-08-22 at 4.26.51 PM.png! > CDH6 (Hadoop 3) GC activities on the KMS process > !Screen Shot 2018-08-22 at 4.27.02 PM.png! > JXray report > !Screen Shot 2018-08-22 at 11.36.16 AM.png! > open fd drops from 20 k down to 3k after the proposed change. > !Screen Shot 2018-08-24 at 7.08.16 PM.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15696) KMS performance regression due to too many open file descriptors after Jetty migration
[
https://issues.apache.org/jira/browse/HADOOP-15696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16594339#comment-16594339
]
Misha Dmitriev commented on HADOOP-15696:
-
To be precise, the suggested measure, that had such a big effect, was to adjust
the timeout in the {{HttpServer2.configureChannelConnector(ServerConnector c)}}
method. Currently it has the {{c.setIdleTimeout(1);}} line. This timeout
should be made configurable in the first place, and looks like we need to
adjust it to a (much) smaller value when {{HttpServer2}} is used by KMS.
Here is a question that I have in this regard. If by closing HTTP connections
on the server side, and thus recycling more quickly, we make KMS work better -
does it mean that the KMS client doesn't reuse any such connections, and/or
doesn't close a connection when it doesn't need it anymore? If so, it doesn't
sound very optimal. I wonder how to prove or disprove this theory.
> KMS performance regression due to too many open file descriptors after Jetty
> migration
> --
>
> Key: HADOOP-15696
> URL: https://issues.apache.org/jira/browse/HADOOP-15696
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 3.0.0-alpha2
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Blocker
> Attachments: Screen Shot 2018-08-22 at 11.36.16 AM.png, Screen Shot
> 2018-08-22 at 4.26.51 PM.png, Screen Shot 2018-08-22 at 4.26.51 PM.png,
> Screen Shot 2018-08-22 at 4.27.02 PM.png, Screen Shot 2018-08-22 at 4.30.32
> PM.png, Screen Shot 2018-08-22 at 4.30.39 PM.png, Screen Shot 2018-08-24 at
> 7.08.16 PM.png
>
>
> We recently found KMS performance regressed in Hadoop 3.0, possibly linking
> to the migration from Tomcat to Jetty in HADOOP-13597.
> Symptoms:
> # Hadoop 3.x KMS open file descriptors quickly rises to more than 10 thousand
> under stress, sometimes even exceeds 32K, which is the system limit, causing
> failures for any access to encryption zones. Our internal testing shows the
> openfd number was in the range of a few hundred in Hadoop 2.x, and it
> increases by almost 100x in Hadoop 3.
> # Hadoop 3.x KMS as much as twice the heap size than in Hadoop 2.x. The same
> heap size can go OOM in Hadoop 3.x. Jxray analysis suggests most of them are
> temporary byte arrays associated with open SSL connections.
> # Due to the heap usage, Hadoop 3.x KMS has more frequent GC activities, and
> we observed up to 20% performance reduction due to GC.
> A possible solution is to reduce the idle timeout setting in HttpServer2. It
> is currently hard-coded 10 seconds. By setting it to 1 second, open fds
> dropped from 20 thousand down to 3 thousand in my experiment.
> File this jira to invite open discussion for a solution.
> Credit: [~mi...@cloudera.com] for the proposed Jetty idle timeout remedy;
> [~xiaochen] for digging into this problem.
> Screenshots:
> CDH5 (Hadoop 2) KMS CPU utilization, resident memory and file descriptor
> chart.
> !Screen Shot 2018-08-22 at 4.30.39 PM.png!
> CDH6 (Hadoop 3) KMS CPU utilization, resident memory and file descriptor
> chart.
> !Screen Shot 2018-08-22 at 4.30.32 PM.png!
> CDH5 (Hadoop 2) GC activities on the KMS process
> !Screen Shot 2018-08-22 at 4.26.51 PM.png!
> CDH6 (Hadoop 3) GC activities on the KMS process
> !Screen Shot 2018-08-22 at 4.27.02 PM.png!
> JXray report
> !Screen Shot 2018-08-22 at 11.36.16 AM.png!
> open fd drops from 20 k down to 3k after the proposed change.
> !Screen Shot 2018-08-24 at 7.08.16 PM.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15633) fs.TrashPolicyDefault: Can't create trash directory
[
https://issues.apache.org/jira/browse/HADOOP-15633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16594308#comment-16594308
]
John Zhuge commented on HADOOP-15633:
-
Committed to branch-3.1 and branch-3.0.
> fs.TrashPolicyDefault: Can't create trash directory
> ---
>
> Key: HADOOP-15633
> URL: https://issues.apache.org/jira/browse/HADOOP-15633
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 2.8.3, 3.1.0, 3.0.3, 2.7.7
>Reporter: Fei Hui
>Assignee: Fei Hui
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HADOOP-15633.001.patch, HADOOP-15633.002.patch,
> HADOOP-15633.003.patch, HADOOP-15633.004.patch, HADOOP-15633.005.patch
>
>
> Reproduce it as follow
> {code:java}
> hadoop fs -mkdir /user/hadoop/aaa
> hadoop fs -touchz /user/hadoop/aaa/bbb
> hadoop fs -rm /user/hadoop/aaa/bbb
> hadoop fs -mkdir /user/hadoop/aaa/bbb
> hadoop fs -touchz /user/hadoop/aaa/bbb/ccc
> hadoop fs -rm /user/hadoop/aaa/bbb/ccc
> {code}
> Then we get errors
> {code:java}
> 18/07/26 17:55:24 WARN fs.TrashPolicyDefault: Can't create trash directory:
> hdfs://xxx/user/hadoop/.Trash/Current/user/hadoop/aaa/bbb
> org.apache.hadoop.fs.FileAlreadyExistsException: Path is not a directory:
> /user/hadoop/.Trash/Current/user/hadoop/aaa/bbb
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:65)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3961)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:984)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:622)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2115)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2111)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1867)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2111)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
> at
> org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:3002)
> at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2970)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:1047)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:1043)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:1061)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:1036)
> at
> org.apache.hadoop.fs.TrashPolicyDefault.moveToTrash(TrashPolicyDefault.java:136)
> at org.apache.hadoop.fs.Trash.moveToTrash(Trash.java:114)
> at org.apache.hadoop.fs.Trash.moveToAppropriateTrash(Trash.java:95)
> at org.apache.hadoop.fs.shell.Delete$Rm.moveToTrash(Delete.java:118)
> at org.apache.hadoop.fs.shell.Delete$Rm.processPath(Delete.java:105)
> at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:317)
> at
> org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:289)
> at
> org.apache.hadoop.fs.shell.Command.processArgument(Command.java:271)
> at
> org.apache.hadoop.fs.shell.Command.processArguments(Command.java:255)
> at
> org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:201)
> at org.apache.hadoop.fs.shell.Command.run(Command.java:165)
> at org.apache.hadoop.fs.FsShell.run(FsShe
[jira] [Commented] (HADOOP-15699) Fix some of testContainerManager failures in Windows
[ https://issues.apache.org/jira/browse/HADOOP-15699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16594182#comment-16594182 ] Hudson commented on HADOOP-15699: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14841 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14841/]) HADOOP-15699. Fix some of testContainerManager failures in Windows. (gifuma: rev 602d13844a8d4c7b08ce185da01fde098ff8b9a6) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java > Fix some of testContainerManager failures in Windows > > > Key: HADOOP-15699 > URL: https://issues.apache.org/jira/browse/HADOOP-15699 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Botong Huang >Assignee: Botong Huang >Priority: Major > Fix For: 3.2.0, 2.9.2 > > Attachments: HADOOP-15699.v1.patch > > > TestContainerManager.testAuxPathHandler: The filename, directory name, or > volume label syntax is incorrect > TestContainerManager.testForcefulShutdownSignal: ProcessStartFile doesn't > exist! > TestContainerManager.testGracefulShutdownSignal: ProcessStartFile doesn't > exist! > TestContainerManager.testOutputThreadDumpSignal: ProcessStartFile doesn't > exist! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15691) Add PathCapabilities to FS and FC to complement StreamCapabilities
[
https://issues.apache.org/jira/browse/HADOOP-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16594178#comment-16594178
]
genericqa commented on HADOOP-15691:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
19m 40s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
34s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 16m
23s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 20s{color} | {color:orange} root: The patch generated 2 new + 454 unchanged
- 0 fixed = 456 total (was 454) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
36s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 25s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
48s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
58s{color} | {color:red} hadoop-common-project_hadoop-common generated 2 new +
0 unchanged - 0 fixed = 2 total (was 0) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 2s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
44s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
54s{color} | {color:green} hadoop-hdfs-httpfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
40s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
17s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
54s{color} | {color:green} hadoop-azure-datalake in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {co
[jira] [Commented] (HADOOP-15699) Fix some of testContainerManager failures in Windows
[ https://issues.apache.org/jira/browse/HADOOP-15699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16594153#comment-16594153 ] Botong Huang commented on HADOOP-15699: --- Thanks [~giovanni.fumarola] for the quick review and commit! > Fix some of testContainerManager failures in Windows > > > Key: HADOOP-15699 > URL: https://issues.apache.org/jira/browse/HADOOP-15699 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Botong Huang >Assignee: Botong Huang >Priority: Major > Fix For: 3.2.0, 2.9.2 > > Attachments: HADOOP-15699.v1.patch > > > TestContainerManager.testAuxPathHandler: The filename, directory name, or > volume label syntax is incorrect > TestContainerManager.testForcefulShutdownSignal: ProcessStartFile doesn't > exist! > TestContainerManager.testGracefulShutdownSignal: ProcessStartFile doesn't > exist! > TestContainerManager.testOutputThreadDumpSignal: ProcessStartFile doesn't > exist! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15699) Fix some of testContainerManager failures in Windows
[ https://issues.apache.org/jira/browse/HADOOP-15699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16594150#comment-16594150 ] Giovanni Matteo Fumarola commented on HADOOP-15699: --- Thanks [~botong] . Committed to trunk, branch-2, branch-2.9. > Fix some of testContainerManager failures in Windows > > > Key: HADOOP-15699 > URL: https://issues.apache.org/jira/browse/HADOOP-15699 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Botong Huang >Assignee: Botong Huang >Priority: Major > Fix For: 3.2.0, 2.9.2 > > Attachments: HADOOP-15699.v1.patch > > > TestContainerManager.testAuxPathHandler: The filename, directory name, or > volume label syntax is incorrect > TestContainerManager.testForcefulShutdownSignal: ProcessStartFile doesn't > exist! > TestContainerManager.testGracefulShutdownSignal: ProcessStartFile doesn't > exist! > TestContainerManager.testOutputThreadDumpSignal: ProcessStartFile doesn't > exist! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15699) Fix some of testContainerManager failures in Windows
[ https://issues.apache.org/jira/browse/HADOOP-15699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni Matteo Fumarola updated HADOOP-15699: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Fix some of testContainerManager failures in Windows > > > Key: HADOOP-15699 > URL: https://issues.apache.org/jira/browse/HADOOP-15699 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Botong Huang >Assignee: Botong Huang >Priority: Major > Fix For: 3.2.0, 2.9.2 > > Attachments: HADOOP-15699.v1.patch > > > TestContainerManager.testAuxPathHandler: The filename, directory name, or > volume label syntax is incorrect > TestContainerManager.testForcefulShutdownSignal: ProcessStartFile doesn't > exist! > TestContainerManager.testGracefulShutdownSignal: ProcessStartFile doesn't > exist! > TestContainerManager.testOutputThreadDumpSignal: ProcessStartFile doesn't > exist! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15699) Fix some of testContainerManager failures in Windows
[ https://issues.apache.org/jira/browse/HADOOP-15699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni Matteo Fumarola updated HADOOP-15699: -- Fix Version/s: 2.9.2 3.2.0 > Fix some of testContainerManager failures in Windows > > > Key: HADOOP-15699 > URL: https://issues.apache.org/jira/browse/HADOOP-15699 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Botong Huang >Assignee: Botong Huang >Priority: Major > Fix For: 3.2.0, 2.9.2 > > Attachments: HADOOP-15699.v1.patch > > > TestContainerManager.testAuxPathHandler: The filename, directory name, or > volume label syntax is incorrect > TestContainerManager.testForcefulShutdownSignal: ProcessStartFile doesn't > exist! > TestContainerManager.testGracefulShutdownSignal: ProcessStartFile doesn't > exist! > TestContainerManager.testOutputThreadDumpSignal: ProcessStartFile doesn't > exist! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15699) Fix some of testContainerManager failures in Windows
[ https://issues.apache.org/jira/browse/HADOOP-15699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16594133#comment-16594133 ] Botong Huang commented on HADOOP-15699: --- The failures before the patch is here [https://builds.apache.org/job/hadoop-trunk-win/533/testReport/] With the patch they are passing in my local Windows box. > Fix some of testContainerManager failures in Windows > > > Key: HADOOP-15699 > URL: https://issues.apache.org/jira/browse/HADOOP-15699 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Botong Huang >Assignee: Botong Huang >Priority: Major > Attachments: HADOOP-15699.v1.patch > > > TestContainerManager.testAuxPathHandler: The filename, directory name, or > volume label syntax is incorrect > TestContainerManager.testForcefulShutdownSignal: ProcessStartFile doesn't > exist! > TestContainerManager.testGracefulShutdownSignal: ProcessStartFile doesn't > exist! > TestContainerManager.testOutputThreadDumpSignal: ProcessStartFile doesn't > exist! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15669) ABFS: Improve HTTPS Performance
[ https://issues.apache.org/jira/browse/HADOOP-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16594128#comment-16594128 ] Steve Loughran commented on HADOOP-15669: - thanks, just caught up with this after being on vacation briefly. FWIW, I Think the user agent should include the values from org.apache.hadoop.util.VersionInfo.getBuildVersion(), as that shows which hadoop build is at work here. Handy if you have a large log of HTTP requests and are trying to work out which client appis causing all your problems > ABFS: Improve HTTPS Performance > --- > > Key: HADOOP-15669 > URL: https://issues.apache.org/jira/browse/HADOOP-15669 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Thomas Marquardt >Assignee: Vishwajeet Dusane >Priority: Major > Attachments: ABFS - Improve HTTPS Performance Over Java Based > Client.pdf, HADOOP-15669-HADOOP-15407-01.patch, > HADOOP-15669-HADOOP-15407-02.patch, HADOOP-15669-HADOOP-15407-03.patch, > HADOOP-15669-HADOOP-15407-04.patch > > > We see approximately 50% worse throughput for ABFS over HTTPs vs HTTP. Lets > perform a detailed measurement and see what can be done to improve throughput. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15692) ABFS: extensible support for custom oauth
[
https://issues.apache.org/jira/browse/HADOOP-15692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16594121#comment-16594121
]
Steve Loughran commented on HADOOP-15692:
-
I'm busy learning about DTs myself (HADOOP-14556) so am now roughly in a
position to review this —at the same time, I'm still learning, so it's good to
see your design and try to pick up ideas. What I'm thinking of implementing as
pluging mechanism is similar to yours, though you are ahead of me in
implementation.
h2. General
Some general questions about overall design.
* Is the expectation that there will be one token provider for every client
configuration? As I've been looking at for S3A having per-fs configs here.
* related: is there ever going to be the need to have >1 link to the same FS
URI for the same client? Daryn's patch for S3A has support for that, which I've
been adopting, even though I'm not sure how I'd imagine using.
I'd like to propose making CustomDelegationTokenManager class a subclass of
AbstractService, using its initialize/start/stop. AbfsDelegationTokenManager
can then subclass CompositeService or simply invoke the lifecycle events of its
tokenManager field as appropriate, as can AzureBlobFileSystem. This is what I
intend to do with my s3a plugin. Why do this?
* Gives everything a unified lifecycle, *including an operation to stop all
threads running*
* Designed for easy subclassing; stable and robust lifecycle implementation.
If someone wanted to implement a plugin which did like the restricted roles of
AWS, where when a DT was requested a token which was returned which offered
only access to the specific container, would this plugin design support it in
future? The use case I'm looking at is: shared Hive cluster with user requests
including DTs granting access only to the given containers. If the Oauth tokens
don't offer such restrictions, I'm not so worried about this, but do consider
this use rather than just "shared Oauth pool service".
h2. Specific
h3. AbfsConfiguration
* can you keep imports in alphabetical order; the renamed
CustomTokenProviderAdapter import needs to move
* You can do a static import of FileSystemConfigurations if that would help the
code readability
h3. AzureBlobFileSystem
L126: add a log entry to log the status of security & DT config; will be useful
for debugging.
L833. Is the javadoc consistent with the current config option/behaviour (i.e.
you need two switches for DTs)?
L840: Feel free to use ? : here rather than the if/else clause
h3. CustomDelegationTokenManager
L40: Add the FS URI to the constructor call. Eventually you'll need it, at
which point all existing implementations will stop linking
L65: maybe have it return false if cancelling isn't supported? Because it's not
in the same category of error as other IOEs.
h3. CustomTokenProviderAdaptee
I know Adaptee is technically correct, but I keep reading it as a typo for the
much more common Adapter.
L74: Recommend `java.time.ZonedDateTime` over `Date`; the latter is too
ambiguous on a global scale. If I in GMT+1 auth with an Azure service in GMT+2,
I still need to know the expiry time relative to me.
h3, DelegationTokenIdentifier order
importing: should be
{code}
java.*
javax.*
--
all non org.apache.
--
org.apache.*
--
statics
{code}
I know, it's rarely changed in existing code, but it's good to get right in the
new stuff. thanks.
L54: use a ref to the constant, to guarantee consistency/spelling
h3. AbfsTokenRenewer
same comment about import ordering.
> ABFS: extensible support for custom oauth
> -
>
> Key: HADOOP-15692
> URL: https://issues.apache.org/jira/browse/HADOOP-15692
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Thomas Marquardt
>Assignee: Da Zhou
>Priority: Major
> Attachments: HADOOP-15692-HADOOP-15407-001.patch
>
>
> ABFS supports oauth in various forms and needs to export interfaces for
> customization of FileSystem.getDelegationToken and getAccessToken.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15699) Fix some of testContainerManager failures in Windows
[
https://issues.apache.org/jira/browse/HADOOP-15699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16594116#comment-16594116
]
genericqa commented on HADOOP-15699:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 5s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 23s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 18m
55s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 71m 5s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | HADOOP-15699 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12937317/HADOOP-15699.v1.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 95eac0fd8260 3.13.0-144-generic #193-Ubuntu SMP Thu Mar 15
17:03:53 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / c9b6395 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15098/testReport/ |
| Max. process+thread count | 302 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15098/console |
| Powered by | Apache Yetus 0.9.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Fix some of test
[jira] [Updated] (HADOOP-15633) fs.TrashPolicyDefault: Can't create trash directory
[
https://issues.apache.org/jira/browse/HADOOP-15633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HADOOP-15633:
Resolution: Fixed
Fix Version/s: 3.2.0
Status: Resolved (was: Patch Available)
Committed to trunk. Tranks [~ferhui] for discovering and fixing the issue!
> fs.TrashPolicyDefault: Can't create trash directory
> ---
>
> Key: HADOOP-15633
> URL: https://issues.apache.org/jira/browse/HADOOP-15633
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 2.8.3, 3.1.0, 3.0.3, 2.7.7
>Reporter: Fei Hui
>Assignee: Fei Hui
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HADOOP-15633.001.patch, HADOOP-15633.002.patch,
> HADOOP-15633.003.patch, HADOOP-15633.004.patch, HADOOP-15633.005.patch
>
>
> Reproduce it as follow
> {code:java}
> hadoop fs -mkdir /user/hadoop/aaa
> hadoop fs -touchz /user/hadoop/aaa/bbb
> hadoop fs -rm /user/hadoop/aaa/bbb
> hadoop fs -mkdir /user/hadoop/aaa/bbb
> hadoop fs -touchz /user/hadoop/aaa/bbb/ccc
> hadoop fs -rm /user/hadoop/aaa/bbb/ccc
> {code}
> Then we get errors
> {code:java}
> 18/07/26 17:55:24 WARN fs.TrashPolicyDefault: Can't create trash directory:
> hdfs://xxx/user/hadoop/.Trash/Current/user/hadoop/aaa/bbb
> org.apache.hadoop.fs.FileAlreadyExistsException: Path is not a directory:
> /user/hadoop/.Trash/Current/user/hadoop/aaa/bbb
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:65)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3961)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:984)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:622)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2115)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2111)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1867)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2111)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
> at
> org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:3002)
> at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2970)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:1047)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:1043)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:1061)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:1036)
> at
> org.apache.hadoop.fs.TrashPolicyDefault.moveToTrash(TrashPolicyDefault.java:136)
> at org.apache.hadoop.fs.Trash.moveToTrash(Trash.java:114)
> at org.apache.hadoop.fs.Trash.moveToAppropriateTrash(Trash.java:95)
> at org.apache.hadoop.fs.shell.Delete$Rm.moveToTrash(Delete.java:118)
> at org.apache.hadoop.fs.shell.Delete$Rm.processPath(Delete.java:105)
> at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:317)
> at
> org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:289)
> at
> org.apache.hadoop.fs.shell.Command.processArgument(Command.java:271)
> at
> org.apache.hadoop.fs.shell.Command.processArguments(Command.java:255)
> at
> org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:201)
> at org.apache.hadoop.fs.shell.C
[jira] [Comment Edited] (HADOOP-15699) Fix some of testContainerManager failures in Windows
[ https://issues.apache.org/jira/browse/HADOOP-15699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16594061#comment-16594061 ] Giovanni Matteo Fumarola edited comment on HADOOP-15699 at 8/27/18 6:25 PM: Thanks [~botong] . Can you share the windows execution with and without the patch? e.g. https://issues.apache.org/jira/browse/HDFS-13509?focusedCommentId=16457054&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16457054 was (Author: giovanni.fumarola): Thanks [~botong] . Can you share the windows execution with and without the patch? e.g. https://issues.apache.org/jira/browse/HDFS-13509 > Fix some of testContainerManager failures in Windows > > > Key: HADOOP-15699 > URL: https://issues.apache.org/jira/browse/HADOOP-15699 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Botong Huang >Assignee: Botong Huang >Priority: Major > Attachments: HADOOP-15699.v1.patch > > > TestContainerManager.testAuxPathHandler: The filename, directory name, or > volume label syntax is incorrect > TestContainerManager.testForcefulShutdownSignal: ProcessStartFile doesn't > exist! > TestContainerManager.testGracefulShutdownSignal: ProcessStartFile doesn't > exist! > TestContainerManager.testOutputThreadDumpSignal: ProcessStartFile doesn't > exist! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15699) Fix some of testContainerManager failures in Windows
[ https://issues.apache.org/jira/browse/HADOOP-15699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16594061#comment-16594061 ] Giovanni Matteo Fumarola edited comment on HADOOP-15699 at 8/27/18 6:24 PM: Thanks [~botong] . Can you share the windows execution with and without the patch? e.g. https://issues.apache.org/jira/browse/HDFS-13509 was (Author: giovanni.fumarola): Thanks [~botong] . Can you share the windows execution with and without the patch? > Fix some of testContainerManager failures in Windows > > > Key: HADOOP-15699 > URL: https://issues.apache.org/jira/browse/HADOOP-15699 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Botong Huang >Assignee: Botong Huang >Priority: Major > Attachments: HADOOP-15699.v1.patch > > > TestContainerManager.testAuxPathHandler: The filename, directory name, or > volume label syntax is incorrect > TestContainerManager.testForcefulShutdownSignal: ProcessStartFile doesn't > exist! > TestContainerManager.testGracefulShutdownSignal: ProcessStartFile doesn't > exist! > TestContainerManager.testOutputThreadDumpSignal: ProcessStartFile doesn't > exist! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15699) Fix some of testContainerManager failures in Windows
[ https://issues.apache.org/jira/browse/HADOOP-15699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16594061#comment-16594061 ] Giovanni Matteo Fumarola commented on HADOOP-15699: --- Thanks [~botong] . Can you share the windows execution with and without the patch? > Fix some of testContainerManager failures in Windows > > > Key: HADOOP-15699 > URL: https://issues.apache.org/jira/browse/HADOOP-15699 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Botong Huang >Assignee: Botong Huang >Priority: Major > Attachments: HADOOP-15699.v1.patch > > > TestContainerManager.testAuxPathHandler: The filename, directory name, or > volume label syntax is incorrect > TestContainerManager.testForcefulShutdownSignal: ProcessStartFile doesn't > exist! > TestContainerManager.testGracefulShutdownSignal: ProcessStartFile doesn't > exist! > TestContainerManager.testOutputThreadDumpSignal: ProcessStartFile doesn't > exist! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15691) Add PathCapabilities to FS and FC to complement StreamCapabilities
[
https://issues.apache.org/jira/browse/HADOOP-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-15691:
Status: Patch Available (was: Open)
> Add PathCapabilities to FS and FC to complement StreamCapabilities
> --
>
> Key: HADOOP-15691
> URL: https://issues.apache.org/jira/browse/HADOOP-15691
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15691-001.patch, HADOOP-15691-002.patch
>
>
> Add a {{PathCapabilities}} interface to both FileSystem and FileContext to
> declare the capabilities under the path of an FS
> This is needed for
> * HADOOP-15407: declare that a dest FS supports permissions
> * object stores to declare that they offer PUT-in-place alongside
> (slow-rename)
> * Anything else where the implementation semantics of an FS is so different
> caller apps would benefit from probing for the underlying semantics
> I know, we want all filesystem to work *exactly* the same. But it doesn't
> hold, especially for object stores —and to efficiently use them, callers need
> to be able to ask for specific features.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15691) Add PathCapabilities to FS and FC to complement StreamCapabilities
[
https://issues.apache.org/jira/browse/HADOOP-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-15691:
Attachment: HADOOP-15691-002.patch
> Add PathCapabilities to FS and FC to complement StreamCapabilities
> --
>
> Key: HADOOP-15691
> URL: https://issues.apache.org/jira/browse/HADOOP-15691
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15691-001.patch, HADOOP-15691-002.patch
>
>
> Add a {{PathCapabilities}} interface to both FileSystem and FileContext to
> declare the capabilities under the path of an FS
> This is needed for
> * HADOOP-15407: declare that a dest FS supports permissions
> * object stores to declare that they offer PUT-in-place alongside
> (slow-rename)
> * Anything else where the implementation semantics of an FS is so different
> caller apps would benefit from probing for the underlying semantics
> I know, we want all filesystem to work *exactly* the same. But it doesn't
> hold, especially for object stores —and to efficiently use them, callers need
> to be able to ask for specific features.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15699) Fix some of testContainerManager failures in Windows
[ https://issues.apache.org/jira/browse/HADOOP-15699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Botong Huang updated HADOOP-15699: -- Status: Patch Available (was: Open) > Fix some of testContainerManager failures in Windows > > > Key: HADOOP-15699 > URL: https://issues.apache.org/jira/browse/HADOOP-15699 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Botong Huang >Assignee: Botong Huang >Priority: Major > Attachments: HADOOP-15699.v1.patch > > > TestContainerManager.testAuxPathHandler: The filename, directory name, or > volume label syntax is incorrect > TestContainerManager.testForcefulShutdownSignal: ProcessStartFile doesn't > exist! > TestContainerManager.testGracefulShutdownSignal: ProcessStartFile doesn't > exist! > TestContainerManager.testOutputThreadDumpSignal: ProcessStartFile doesn't > exist! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15699) Fix some of testContainerManager failures in Windows
[ https://issues.apache.org/jira/browse/HADOOP-15699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Botong Huang updated HADOOP-15699: -- Attachment: HADOOP-15699.v1.patch > Fix some of testContainerManager failures in Windows > > > Key: HADOOP-15699 > URL: https://issues.apache.org/jira/browse/HADOOP-15699 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Botong Huang >Assignee: Botong Huang >Priority: Major > Attachments: HADOOP-15699.v1.patch > > > TestContainerManager.testAuxPathHandler: The filename, directory name, or > volume label syntax is incorrect > TestContainerManager.testForcefulShutdownSignal: ProcessStartFile doesn't > exist! > TestContainerManager.testGracefulShutdownSignal: ProcessStartFile doesn't > exist! > TestContainerManager.testOutputThreadDumpSignal: ProcessStartFile doesn't > exist! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15693) Typo in Secure Mode documentation
[
https://issues.apache.org/jira/browse/HADOOP-15693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16594035#comment-16594035
]
genericqa commented on HADOOP-15693:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 11m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
42m 19s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 11m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 16s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 85m 30s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | HADOOP-15693 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12936982/HADOOP-15693.patch |
| Optional Tests | dupname asflicense mvnsite |
| uname | Linux a95ffcbc199e 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 6eecd25 |
| maven | version: Apache Maven 3.3.9 |
| Max. process+thread count | 302 (vs. ulimit of 1) |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15097/console |
| Powered by | Apache Yetus 0.9.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Typo in Secure Mode documentation
> -
>
> Key: HADOOP-15693
> URL: https://issues.apache.org/jira/browse/HADOOP-15693
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
> Environment: OpenBSD 6.3 GENERIC.MP#4 amd64
>Reporter: Thomas Levine
>Assignee: Thomas Levine
>Priority: Trivial
> Labels: documentation
> Attachments: HADOOP-15693.patch
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15691) Add PathCapabilities to FS and FC to complement StreamCapabilities
[
https://issues.apache.org/jira/browse/HADOOP-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-15691:
Status: Open (was: Patch Available)
> Add PathCapabilities to FS and FC to complement StreamCapabilities
> --
>
> Key: HADOOP-15691
> URL: https://issues.apache.org/jira/browse/HADOOP-15691
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15691-001.patch
>
>
> Add a {{PathCapabilities}} interface to both FileSystem and FileContext to
> declare the capabilities under the path of an FS
> This is needed for
> * HADOOP-15407: declare that a dest FS supports permissions
> * object stores to declare that they offer PUT-in-place alongside
> (slow-rename)
> * Anything else where the implementation semantics of an FS is so different
> caller apps would benefit from probing for the underlying semantics
> I know, we want all filesystem to work *exactly* the same. But it doesn't
> hold, especially for object stores —and to efficiently use them, callers need
> to be able to ask for specific features.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15698) KMS startup logs don't show
[ https://issues.apache.org/jira/browse/HADOOP-15698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16594031#comment-16594031 ] Ajay Kumar commented on HADOOP-15698: - [~knanasi] thanks for submitting the patch. Any plans to add a unit test ? Seems adding unit test is not straight forward but we can test logging in KMSWebServer. > KMS startup logs don't show > --- > > Key: HADOOP-15698 > URL: https://issues.apache.org/jira/browse/HADOOP-15698 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 3.1.0 >Reporter: Kitti Nanasi >Assignee: Kitti Nanasi >Priority: Major > Attachments: HADOOP-15698.001.patch > > > During KMs startup, log4j logs don't show up resulting in important logs > getting omitted. This happens because log4 initialisation only happens in > KMSWebApp#contextInitialized and logs written before that don't show up. > For example the following log never shows up: > [https://github.com/apache/hadoop/blob/a55d6bba71c81c1c4e9d8cd11f55c78f10a548b0/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/ZKSignerSecretProvider.java#L197-L199] > Another example is that the KMS startup message never shows up in the kms > logs. > Note that this works in the unit tests, because MiniKMS sets the log4j system > property. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15691) Add PathCapabilities to FS and FC to complement StreamCapabilities
[
https://issues.apache.org/jira/browse/HADOOP-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-15691:
Status: Patch Available (was: Open)
> Add PathCapabilities to FS and FC to complement StreamCapabilities
> --
>
> Key: HADOOP-15691
> URL: https://issues.apache.org/jira/browse/HADOOP-15691
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15691-001.patch
>
>
> Add a {{PathCapabilities}} interface to both FileSystem and FileContext to
> declare the capabilities under the path of an FS
> This is needed for
> * HADOOP-15407: declare that a dest FS supports permissions
> * object stores to declare that they offer PUT-in-place alongside
> (slow-rename)
> * Anything else where the implementation semantics of an FS is so different
> caller apps would benefit from probing for the underlying semantics
> I know, we want all filesystem to work *exactly* the same. But it doesn't
> hold, especially for object stores —and to efficiently use them, callers need
> to be able to ask for specific features.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15699) Fix some of testContainerManager failures in Windows
[ https://issues.apache.org/jira/browse/HADOOP-15699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Botong Huang updated HADOOP-15699: -- Issue Type: Sub-task (was: Task) Parent: HADOOP-15475 > Fix some of testContainerManager failures in Windows > > > Key: HADOOP-15699 > URL: https://issues.apache.org/jira/browse/HADOOP-15699 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Botong Huang >Assignee: Botong Huang >Priority: Major > > TestContainerManager.testAuxPathHandler: The filename, directory name, or > volume label syntax is incorrect > TestContainerManager.testForcefulShutdownSignal: ProcessStartFile doesn't > exist! > TestContainerManager.testGracefulShutdownSignal: ProcessStartFile doesn't > exist! > TestContainerManager.testOutputThreadDumpSignal: ProcessStartFile doesn't > exist! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15699) Fix some of testContainerManager failures in Windows
Botong Huang created HADOOP-15699: - Summary: Fix some of testContainerManager failures in Windows Key: HADOOP-15699 URL: https://issues.apache.org/jira/browse/HADOOP-15699 Project: Hadoop Common Issue Type: Task Reporter: Botong Huang Assignee: Botong Huang TestContainerManager.testAuxPathHandler: The filename, directory name, or volume label syntax is incorrect TestContainerManager.testForcefulShutdownSignal: ProcessStartFile doesn't exist! TestContainerManager.testGracefulShutdownSignal: ProcessStartFile doesn't exist! TestContainerManager.testOutputThreadDumpSignal: ProcessStartFile doesn't exist! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15633) fs.TrashPolicyDefault: Can't create trash directory
[
https://issues.apache.org/jira/browse/HADOOP-15633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593940#comment-16593940
]
Hudson commented on HADOOP-15633:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14836 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/14836/])
HADOOP-15633. fs.TrashPolicyDefault: Can't create trash directory. (jzhuge: rev
f9c6fd94711458b77ecf3fa425aad7fda5089376)
* (edit)
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestTrash.java
* (edit)
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/TrashPolicyDefault.java
> fs.TrashPolicyDefault: Can't create trash directory
> ---
>
> Key: HADOOP-15633
> URL: https://issues.apache.org/jira/browse/HADOOP-15633
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 2.8.3, 3.1.0, 3.0.3, 2.7.7
>Reporter: Fei Hui
>Assignee: Fei Hui
>Priority: Major
> Attachments: HADOOP-15633.001.patch, HADOOP-15633.002.patch,
> HADOOP-15633.003.patch, HADOOP-15633.004.patch, HADOOP-15633.005.patch
>
>
> Reproduce it as follow
> {code:java}
> hadoop fs -mkdir /user/hadoop/aaa
> hadoop fs -touchz /user/hadoop/aaa/bbb
> hadoop fs -rm /user/hadoop/aaa/bbb
> hadoop fs -mkdir /user/hadoop/aaa/bbb
> hadoop fs -touchz /user/hadoop/aaa/bbb/ccc
> hadoop fs -rm /user/hadoop/aaa/bbb/ccc
> {code}
> Then we get errors
> {code:java}
> 18/07/26 17:55:24 WARN fs.TrashPolicyDefault: Can't create trash directory:
> hdfs://xxx/user/hadoop/.Trash/Current/user/hadoop/aaa/bbb
> org.apache.hadoop.fs.FileAlreadyExistsException: Path is not a directory:
> /user/hadoop/.Trash/Current/user/hadoop/aaa/bbb
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:65)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3961)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:984)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:622)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2115)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2111)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1867)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2111)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
> at
> org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:3002)
> at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2970)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:1047)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:1043)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:1061)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:1036)
> at
> org.apache.hadoop.fs.TrashPolicyDefault.moveToTrash(TrashPolicyDefault.java:136)
> at org.apache.hadoop.fs.Trash.moveToTrash(Trash.java:114)
> at org.apache.hadoop.fs.Trash.moveToAppropriateTrash(Trash.java:95)
> at org.apache.hadoop.fs.shell.Delete$Rm.moveToTrash(Delete.java:118)
> at org.apache.hadoop.fs.shell.Delete$Rm.processPath(Delete.java:105)
> at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:317)
> at
> org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:289)
>
[jira] [Commented] (HADOOP-15691) Add PathCapabilities to FS and FC to complement StreamCapabilities
[
https://issues.apache.org/jira/browse/HADOOP-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593934#comment-16593934
]
Steve Loughran commented on HADOOP-15691:
-
Should be of interest to: [~sanjay.radia], [~tmarquardt], [~fabbri], [~ehiggs]
[~thomasdemoor], [~rdblue], [~arpitagarwal] [~jzhuge] [~sseth]
The goal here initially is just to let callers see if things like concat/append
work before trying, but I really want to line things up for being able to ask
if a store can do a direct write to a dest path without it being visible until
closed (so you don't need to write to a temp path & rename for that
one-operation commit), when to know that rename() doesn't do what posix
expects,
I know this breaks the idea that all filesystems should work just like HDFS,
but that is true today: we just need to be able to determine which APIs *and
semantics* are offered by specific FS instances.
> Add PathCapabilities to FS and FC to complement StreamCapabilities
> --
>
> Key: HADOOP-15691
> URL: https://issues.apache.org/jira/browse/HADOOP-15691
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15691-001.patch
>
>
> Add a {{PathCapabilities}} interface to both FileSystem and FileContext to
> declare the capabilities under the path of an FS
> This is needed for
> * HADOOP-15407: declare that a dest FS supports permissions
> * object stores to declare that they offer PUT-in-place alongside
> (slow-rename)
> * Anything else where the implementation semantics of an FS is so different
> caller apps would benefit from probing for the underlying semantics
> I know, we want all filesystem to work *exactly* the same. But it doesn't
> hold, especially for object stores —and to efficiently use them, callers need
> to be able to ask for specific features.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15691) Add PathCapabilities to FS and FC to complement StreamCapabilities
[
https://issues.apache.org/jira/browse/HADOOP-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593926#comment-16593926
]
Steve Loughran commented on HADOOP-15691:
-
Patch 001: initial implementation. This defines a new interface,
PathCapabiliities, implemented by FileSystem, AbstractFileSystem and
FileContext, to probe for features. It has one method
{code}
boolean hasPathCapability(Path path, String capability) throws IOException;
{code}
* There's a core list of features added in StreamCapabilities (right place?);
for the main optional FS operations & features: permissions, xattrs, storage
policies, snapshots, append, truncate, concat, and wired up for the all built
in filesystems
* No tests; they can go into contract tests (e.g the append test can verify
that having the append capability -> append works, etc)
* Base implementation returns false for everything but the "fs:symlinks" probe,
which is forwarded to {{supportsSymlinks()}}
To do in separate JIRA;
* Object store operations/semantics, where I want to be able to declare when a
store does a PUT at the end of a write (e.g: no rename needed), and to declare
when renames are slow/non-atomic for dirs. We can use that in the mrv2
committers to know when to warn, perhaps.
* Multipart upload availablity
h3. Semantics of return value
w.r/t this patch, the spec says "return true iff you know the feature is
available". This means "false" can mean any of
* The capability is unknown.
* The capability is known, but not available on this instance.
* The capability is known but the connector does not know if it is supported by
this specific instance.
We could think about having an enum of all four states, so that you can
distinguish them, but I fear it gets complicated with forwarded filesystems.
h3. FSLinkResolver
new class {{org.apache.hadoop.fs.FSLinkResolver}} for implementing FileContext
operations as closures, here
{code}
FsLinkResolution.resolve(this,
fixRelativePart(path),
(fs, p) -> fs.hasPathCapability(p, capability));
{code}
It should be straightforward to move all FileContext operations to this & make
the file a lot leaner.
> Add PathCapabilities to FS and FC to complement StreamCapabilities
> --
>
> Key: HADOOP-15691
> URL: https://issues.apache.org/jira/browse/HADOOP-15691
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15691-001.patch
>
>
> Add a {{PathCapabilities}} interface to both FileSystem and FileContext to
> declare the capabilities under the path of an FS
> This is needed for
> * HADOOP-15407: declare that a dest FS supports permissions
> * object stores to declare that they offer PUT-in-place alongside
> (slow-rename)
> * Anything else where the implementation semantics of an FS is so different
> caller apps would benefit from probing for the underlying semantics
> I know, we want all filesystem to work *exactly* the same. But it doesn't
> hold, especially for object stores —and to efficiently use them, callers need
> to be able to ask for specific features.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15691) Add PathCapabilities to FS and FC to complement StreamCapabilities
[
https://issues.apache.org/jira/browse/HADOOP-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-15691:
Attachment: HADOOP-15691-001.patch
> Add PathCapabilities to FS and FC to complement StreamCapabilities
> --
>
> Key: HADOOP-15691
> URL: https://issues.apache.org/jira/browse/HADOOP-15691
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15691-001.patch
>
>
> Add a {{PathCapabilities}} interface to both FileSystem and FileContext to
> declare the capabilities under the path of an FS
> This is needed for
> * HADOOP-15407: declare that a dest FS supports permissions
> * object stores to declare that they offer PUT-in-place alongside
> (slow-rename)
> * Anything else where the implementation semantics of an FS is so different
> caller apps would benefit from probing for the underlying semantics
> I know, we want all filesystem to work *exactly* the same. But it doesn't
> hold, especially for object stores —and to efficiently use them, callers need
> to be able to ask for specific features.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15693) Typo in Secure Mode documentation
[ https://issues.apache.org/jira/browse/HADOOP-15693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593898#comment-16593898 ] Arpit Agarwal commented on HADOOP-15693: Thanks [~tlevine]. +1 pending Jenkins. I've added you as a Hadoop contributor and assigned to you. > Typo in Secure Mode documentation > - > > Key: HADOOP-15693 > URL: https://issues.apache.org/jira/browse/HADOOP-15693 > Project: Hadoop Common > Issue Type: Bug > Components: common > Environment: OpenBSD 6.3 GENERIC.MP#4 amd64 >Reporter: Thomas Levine >Assignee: Thomas Levine >Priority: Trivial > Labels: documentation > Attachments: HADOOP-15693.patch > > Original Estimate: 0h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15693) Typo in Secure Mode documentation
[ https://issues.apache.org/jira/browse/HADOOP-15693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal reassigned HADOOP-15693: -- Assignee: Thomas Levine > Typo in Secure Mode documentation > - > > Key: HADOOP-15693 > URL: https://issues.apache.org/jira/browse/HADOOP-15693 > Project: Hadoop Common > Issue Type: Bug > Components: common > Environment: OpenBSD 6.3 GENERIC.MP#4 amd64 >Reporter: Thomas Levine >Assignee: Thomas Levine >Priority: Trivial > Labels: documentation > Attachments: HADOOP-15693.patch > > Original Estimate: 0h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15693) Typo in Secure Mode documentation
[ https://issues.apache.org/jira/browse/HADOOP-15693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HADOOP-15693: --- Status: Patch Available (was: Open) > Typo in Secure Mode documentation > - > > Key: HADOOP-15693 > URL: https://issues.apache.org/jira/browse/HADOOP-15693 > Project: Hadoop Common > Issue Type: Bug > Components: common > Environment: OpenBSD 6.3 GENERIC.MP#4 amd64 >Reporter: Thomas Levine >Assignee: Thomas Levine >Priority: Trivial > Labels: documentation > Attachments: HADOOP-15693.patch > > Original Estimate: 0h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15697) memory leak in distcp run method
[ https://issues.apache.org/jira/browse/HADOOP-15697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HADOOP-15697: - Attachment: gc-root.png > memory leak in distcp run method > > > Key: HADOOP-15697 > URL: https://issues.apache.org/jira/browse/HADOOP-15697 > Project: Hadoop Common > Issue Type: Bug >Reporter: mahesh kumar behera >Priority: Major > Attachments: gc-root.png > > > in distcp run method, execute is called but the job is not closed. This is > causing OOM error in Hive replication. I think in run method , shouldBlock > should be set to true and execute should be called within try-with-resources. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15697) memory leak in distcp run method
[ https://issues.apache.org/jira/browse/HADOOP-15697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593883#comment-16593883 ] mahesh kumar behera commented on HADOOP-15697: -- !gc-root.png! > memory leak in distcp run method > > > Key: HADOOP-15697 > URL: https://issues.apache.org/jira/browse/HADOOP-15697 > Project: Hadoop Common > Issue Type: Bug >Reporter: mahesh kumar behera >Priority: Major > > in distcp run method, execute is called but the job is not closed. This is > causing OOM error in Hive replication. I think in run method , shouldBlock > should be set to true and execute should be called within try-with-resources. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15697) memory leak in distcp run method
[ https://issues.apache.org/jira/browse/HADOOP-15697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593882#comment-16593882 ] mahesh kumar behera commented on HADOOP-15697: -- https://docs.google.com/document/d/1RNcAcRbATM7RfYboGLSpTmbzHbHPsrU8tRo9O9pPOZM/edit?usp=sharing [~jojochuang] The hprof file is shared. > memory leak in distcp run method > > > Key: HADOOP-15697 > URL: https://issues.apache.org/jira/browse/HADOOP-15697 > Project: Hadoop Common > Issue Type: Bug >Reporter: mahesh kumar behera >Priority: Major > > in distcp run method, execute is called but the job is not closed. This is > causing OOM error in Hive replication. I think in run method , shouldBlock > should be set to true and execute should be called within try-with-resources. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15697) memory leak in distcp run method
[ https://issues.apache.org/jira/browse/HADOOP-15697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593620#comment-16593620 ] Wei-Chiu Chuang commented on HADOOP-15697: -- Hi [~maheshk114] thanks for reporting the issue. Do you have a heap dump after the OOM? Without a heapdump or a stacktrace it's hard to know what went wrong exactly. One source of OOM that we are aware of is the file lists (distcp has to get a list of every files) and it can grow to several gigabytes for a large directory. This known issue is addressed by HADOOP-12502. But there could be other issues as well so we'll need more data to understand better. > memory leak in distcp run method > > > Key: HADOOP-15697 > URL: https://issues.apache.org/jira/browse/HADOOP-15697 > Project: Hadoop Common > Issue Type: Bug >Reporter: mahesh kumar behera >Priority: Major > > in distcp run method, execute is called but the job is not closed. This is > causing OOM error in Hive replication. I think in run method , shouldBlock > should be set to true and execute should be called within try-with-resources. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15621) s3guard: implement time-based (TTL) expiry for DynamoDB Metadata Store
[
https://issues.apache.org/jira/browse/HADOOP-15621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593567#comment-16593567
]
Gabor Bota edited comment on HADOOP-15621 at 8/27/18 12:28 PM:
---
Some comments on the description:
bq. I think we need a new column in the dynamo table "entry last written time".
This is updated each time the entry is written to dynamo.
The current implementation uses {{mod_time}} field when using prune. It would
be wise to use the same because this is an online version of prune. Thus, we
don't need to add a new field to the item.
{quote}After that we can either
1. Have the client simply ignore / elide any entries that are older than the
configured TTL.
2. Have the client delete entries older than the TTL.{quote}
There will be a switch for this - the user can set if she wants the item to be
deleted automatically if the {{current_time - mod_time > ttl}}.
The implementation will behave like the following:
1. {{DynamoDBMetadataStore}} will ignore any entries that are older than the
configured TTL. That means the entry will not be included in the directory
listing, the returned {{DirListingMetadata}} won't be authoritative.
There's an issue with this setting: the listing won't be authoritative if the
entry is older than TTL, but if the client does another full listing, and saves
it to the metadata store, the next listing request will still be automatically
{{isAuthoritative=false}} because of the leftover expired entry in the same
directory. The solution could be that the {{DDBPathMetadata}}, so the dynamo
entry could store if the file already caused a directory listing to be
{{isAuthoritative=false}}, so it won't cause the listing to not be
authoritative again. Of course, if there will be another entry with the same
name, the flag should be removed (overwritten). This could be a flag named
{{expired_entry}}.
2. {{DynamoDBMetadataStore}} deletes entries which are older than the TTL. With
this setting we won't have the same issue as in 1. setting, but will cost more
as we have to calculate with the deletion round trip time to dynamo.
was (Author: gabor.bota):
Some comments on the description:
bq. I think we need a new column in the dynamo table "entry last written time".
This is updated each time the entry is written to dynamo.
The current implementation uses {{mod_time}} field when using prune. It would
be wise to use the same because this is an online version of prune. Thus, we
don't need to add a new field to the item.
{quote}After that we can either
1. Have the client simply ignore / elide any entries that are older than the
configured TTL.
2. Have the client delete entries older than the TTL.{quote}
There will be a switch for this - the user can set if she wants the item to be
deleted automatically if the {{current_time - mod_time > ttl}}.
The implementation will behave like the following:
1. {{DynamoDBMetadataStore}} will ignore any entries that are older than the
configured TTL. That means the entry will not be included in the directory
listing, the returned {{DirListingMetadata}} won't be authoritative.
There's an issue with this setting: the listing won't be authoritative if the
entry is older than TTL, but if the client does another full listing, and saves
it to the metadata store, the next listing request will still be automatically
{{isAuthoritative=false}} because of the leftover expired entry in the same
dictionary. The solution could be that the {{DDBPathMetadata}}, so the dynamo
entry could store if the file already caused a directory listing to be
{{isAuthoritative=false}}, so it won't cause the listing to not be
authoritative again. Of course, if there will be another entry with the same
name, the flag should be removed (overwritten).
2. {{DynamoDBMetadataStore}} deletes entries which are older than the TTL. With
this setting we won't have the same issue as in 1. setting, but will cost more
as we have to calculate with the deletion round trip time to dynamo.
> s3guard: implement time-based (TTL) expiry for DynamoDB Metadata Store
> --
>
> Key: HADOOP-15621
> URL: https://issues.apache.org/jira/browse/HADOOP-15621
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Aaron Fabbri
>Assignee: Gabor Bota
>Priority: Minor
>
> Similar to HADOOP-13649, I think we should add a TTL (time to live) feature
> to the Dynamo metadata store (MS) for S3Guard.
> Think of this as the "online algorithm" version of the CLI prune() function,
> which is the "offline algorithm".
> Why:
> 1. Self healing (soft state): since we do not implement transactions around
> modification of the two systems (s3 and metada
[jira] [Updated] (HADOOP-15696) KMS performance regression due to too many open file descriptors after Jetty migration
[ https://issues.apache.org/jira/browse/HADOOP-15696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HADOOP-15696: - Affects Version/s: (was: 3.0.0) 3.0.0-alpha2 > KMS performance regression due to too many open file descriptors after Jetty > migration > -- > > Key: HADOOP-15696 > URL: https://issues.apache.org/jira/browse/HADOOP-15696 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 3.0.0-alpha2 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Blocker > Attachments: Screen Shot 2018-08-22 at 11.36.16 AM.png, Screen Shot > 2018-08-22 at 4.26.51 PM.png, Screen Shot 2018-08-22 at 4.26.51 PM.png, > Screen Shot 2018-08-22 at 4.27.02 PM.png, Screen Shot 2018-08-22 at 4.30.32 > PM.png, Screen Shot 2018-08-22 at 4.30.39 PM.png, Screen Shot 2018-08-24 at > 7.08.16 PM.png > > > We recently found KMS performance regressed in Hadoop 3.0, possibly linking > to the migration from Tomcat to Jetty in HADOOP-13597. > Symptoms: > # Hadoop 3.x KMS open file descriptors quickly rises to more than 10 thousand > under stress, sometimes even exceeds 32K, which is the system limit, causing > failures for any access to encryption zones. Our internal testing shows the > openfd number was in the range of a few hundred in Hadoop 2.x, and it > increases by almost 100x in Hadoop 3. > # Hadoop 3.x KMS as much as twice the heap size than in Hadoop 2.x. The same > heap size can go OOM in Hadoop 3.x. Jxray analysis suggests most of them are > temporary byte arrays associated with open SSL connections. > # Due to the heap usage, Hadoop 3.x KMS has more frequent GC activities, and > we observed up to 20% performance reduction due to GC. > A possible solution is to reduce the idle timeout setting in HttpServer2. It > is currently hard-coded 10 seconds. By setting it to 1 second, open fds > dropped from 20 thousand down to 3 thousand in my experiment. > File this jira to invite open discussion for a solution. > Credit: [~mi...@cloudera.com] for the proposed Jetty idle timeout remedy; > [~xiaochen] for digging into this problem. > Screenshots: > CDH5 (Hadoop 2) KMS CPU utilization, resident memory and file descriptor > chart. > !Screen Shot 2018-08-22 at 4.30.39 PM.png! > CDH6 (Hadoop 3) KMS CPU utilization, resident memory and file descriptor > chart. > !Screen Shot 2018-08-22 at 4.30.32 PM.png! > CDH5 (Hadoop 2) GC activities on the KMS process > !Screen Shot 2018-08-22 at 4.26.51 PM.png! > CDH6 (Hadoop 3) GC activities on the KMS process > !Screen Shot 2018-08-22 at 4.27.02 PM.png! > JXray report > !Screen Shot 2018-08-22 at 11.36.16 AM.png! > open fd drops from 20 k down to 3k after the proposed change. > !Screen Shot 2018-08-24 at 7.08.16 PM.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15621) s3guard: implement time-based (TTL) expiry for DynamoDB Metadata Store
[
https://issues.apache.org/jira/browse/HADOOP-15621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593567#comment-16593567
]
Gabor Bota commented on HADOOP-15621:
-
Some comments on the description:
bq. I think we need a new column in the dynamo table "entry last written time".
This is updated each time the entry is written to dynamo.
The current implementation uses {{mod_time}} field when using prune. It would
be wise to use the same because this is an online version of prune. Thus, we
don't need to add a new field to the item.
{quote}After that we can either
1. Have the client simply ignore / elide any entries that are older than the
configured TTL.
2. Have the client delete entries older than the TTL.{quote}
There will be a switch for this - the user can set if she wants the item to be
deleted automatically if the {{current_time - mod_time > ttl}}.
The implementation will behave like the following:
1. {{DynamoDBMetadataStore}} will ignore any entries that are older than the
configured TTL. That means the entry will not be included in the directory
listing, the returned {{DirListingMetadata}} won't be authoritative.
There's an issue with this setting: the listing won't be authoritative if the
entry is older than TTL, but if the client does another full listing, and saves
it to the metadata store, the next listing request will still be automatically
{{isAuthoritative=false}} because of the leftover expired entry in the same
dictionary. The solution could be that the {{DDBPathMetadata}}, so the dynamo
entry could store if the file already caused a directory listing to be
{{isAuthoritative=false}}, so it won't cause the listing to not be
authoritative again. Of course, if there will be another entry with the same
name, the flag should be removed (overwritten).
2. {{DynamoDBMetadataStore}} deletes entries which are older than the TTL. With
this setting we won't have the same issue as in 1. setting, but will cost more
as we have to calculate with the deletion round trip time to dynamo.
> s3guard: implement time-based (TTL) expiry for DynamoDB Metadata Store
> --
>
> Key: HADOOP-15621
> URL: https://issues.apache.org/jira/browse/HADOOP-15621
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Aaron Fabbri
>Assignee: Gabor Bota
>Priority: Minor
>
> Similar to HADOOP-13649, I think we should add a TTL (time to live) feature
> to the Dynamo metadata store (MS) for S3Guard.
> Think of this as the "online algorithm" version of the CLI prune() function,
> which is the "offline algorithm".
> Why:
> 1. Self healing (soft state): since we do not implement transactions around
> modification of the two systems (s3 and metadata store), certain failures can
> lead to inconsistency between S3 and the metadata store (MS) state. Having a
> time to live (TTL) on each entry in S3Guard means that any inconsistencies
> will be time bound. Thus "wait and restart your job" becomes a valid, if
> ugly, way to get around any issues with FS client failure leaving things in a
> bad state.
> 2. We could make manual invocation of `hadoop s3guard prune ...` unnecessary,
> depending on the implementation.
> 3. Makes it possible to fix the problem that dynamo MS prune() doesn't prune
> directories due to the lack of true modification time.
> How:
> I think we need a new column in the dynamo table "entry last written time".
> This is updated each time the entry is written to dynamo.
> After that we can either
> 1. Have the client simply ignore / elide any entries that are older than the
> configured TTL.
> 2. Have the client delete entries older than the TTL.
> The issue with #2 is it will increase latency if done inline in the context
> of an FS operation. We could mitigate this some by using an async helper
> thread, or probabilistically doing it "some times" to amortize the expense of
> deleting stale entries (allowing some batching as well).
> Caveats:
> - Clock synchronization as usual is a concern. Many clusters already keep
> clocks close enough via NTP. We should at least document the requirement
> along with the configuration knob that enables the feature.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15676) Cleanup TestSSLHttpServer
[
https://issues.apache.org/jira/browse/HADOOP-15676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593538#comment-16593538
]
Zoltan Siegl commented on HADOOP-15676:
---
Hey [~snemeth]!
Did some review on the patch, and I have a few comments might worth considering.
* {{org/apache/hadoop/http/TestSSLHttpServer.java:70}}
{code:java}
public static final String SERVLET_NAME_LONGHEADER = "longheader";
public static final String SERVLET_PATH_LONGHEADER =
"/" + SERVLET_NAME_LONGHEADER;
public static final String SERVLET_NAME_ECHO = "echo";
public static final String SERVLET_PATH_ECHO = "/" + SERVLET_NAME_ECHO;{code}
If there is not specific reason for these to be public, maybe we should
consider making these private.
* {{org/apache/hadoop/http/TestSSLHttpServer.java:248}}
It looks like there has been an additional empty line added in the patch. If
you create a new patch anyways, should we consider removing this?
> Cleanup TestSSLHttpServer
> -
>
> Key: HADOOP-15676
> URL: https://issues.apache.org/jira/browse/HADOOP-15676
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common
>Affects Versions: 2.6.0
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Minor
> Attachments: HADOOP-15676.001.patch
>
>
> This issue will fix:
> * Several typos in this class
> * Code is not very well readable in some of the places.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15698) KMS startup logs don't show
[ https://issues.apache.org/jira/browse/HADOOP-15698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593456#comment-16593456 ] Kitti Nanasi commented on HADOOP-15698: --- I tried my patch on a cluster and the logs mentioned in the description do show up in the kms.log file. > KMS startup logs don't show > --- > > Key: HADOOP-15698 > URL: https://issues.apache.org/jira/browse/HADOOP-15698 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 3.1.0 >Reporter: Kitti Nanasi >Assignee: Kitti Nanasi >Priority: Major > Attachments: HADOOP-15698.001.patch > > > During KMs startup, log4j logs don't show up resulting in important logs > getting omitted. This happens because log4 initialisation only happens in > KMSWebApp#contextInitialized and logs written before that don't show up. > For example the following log never shows up: > [https://github.com/apache/hadoop/blob/a55d6bba71c81c1c4e9d8cd11f55c78f10a548b0/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/ZKSignerSecretProvider.java#L197-L199] > Another example is that the KMS startup message never shows up in the kms > logs. > Note that this works in the unit tests, because MiniKMS sets the log4j system > property. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15698) KMS startup logs don't show
[ https://issues.apache.org/jira/browse/HADOOP-15698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kitti Nanasi updated HADOOP-15698: -- Attachment: HADOOP-15698.001.patch > KMS startup logs don't show > --- > > Key: HADOOP-15698 > URL: https://issues.apache.org/jira/browse/HADOOP-15698 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 3.1.0 >Reporter: Kitti Nanasi >Assignee: Kitti Nanasi >Priority: Major > Attachments: HADOOP-15698.001.patch > > > During KMs startup, log4j logs don't show up resulting in important logs > getting omitted. This happens because log4 initialisation only happens in > KMSWebApp#contextInitialized and logs written before that don't show up. > For example the following log never shows up: > [https://github.com/apache/hadoop/blob/a55d6bba71c81c1c4e9d8cd11f55c78f10a548b0/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/ZKSignerSecretProvider.java#L197-L199] > Another example is that the KMS startup message never shows up in the kms > logs. > Note that this works in the unit tests, because MiniKMS sets the log4j system > property. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15698) KMS startup logs don't show
Kitti Nanasi created HADOOP-15698: - Summary: KMS startup logs don't show Key: HADOOP-15698 URL: https://issues.apache.org/jira/browse/HADOOP-15698 Project: Hadoop Common Issue Type: Bug Components: kms Affects Versions: 3.1.0 Reporter: Kitti Nanasi Assignee: Kitti Nanasi During KMs startup, log4j logs don't show up resulting in important logs getting omitted. This happens because log4 initialisation only happens in KMSWebApp#contextInitialized and logs written before that don't show up. For example the following log never shows up: [https://github.com/apache/hadoop/blob/a55d6bba71c81c1c4e9d8cd11f55c78f10a548b0/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/ZKSignerSecretProvider.java#L197-L199] Another example is that the KMS startup message never shows up in the kms logs. Note that this works in the unit tests, because MiniKMS sets the log4j system property. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15697) memory leak in distcp run method
mahesh kumar behera created HADOOP-15697: Summary: memory leak in distcp run method Key: HADOOP-15697 URL: https://issues.apache.org/jira/browse/HADOOP-15697 Project: Hadoop Common Issue Type: Bug Reporter: mahesh kumar behera in distcp run method, execute is called but the job is not closed. This is causing OOM error in Hive replication. I think in run method , shouldBlock should be set to true and execute should be called within try-with-resources. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
