[jira] [Updated] (HADOOP-14062) ApplicationMasterProtocolPBClientImpl.allocate fails with EOFException when RPC privacy is enabled
[ https://issues.apache.org/jira/browse/HADOOP-14062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated HADOOP-14062: - Resolution: Fixed Status: Resolved (was: Patch Available) committed the new patch to trunk and branch-2 > ApplicationMasterProtocolPBClientImpl.allocate fails with EOFException when > RPC privacy is enabled > -- > > Key: HADOOP-14062 > URL: https://issues.apache.org/jira/browse/HADOOP-14062 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.0 >Reporter: Steven Rand >Assignee: Steven Rand >Priority: Critical > Fix For: 2.8.0, 3.0.0-alpha3 > > Attachments: HADOOP-14062.001.patch, HADOOP-14062.002.patch, > HADOOP-14062.003.patch, HADOOP-14062.004.patch, > HADOOP-14062-branch-2.8.0.004.patch, HADOOP-14062-branch-2.8.0.005.patch, > HADOOP-14062-branch-2.8.0.005.patch, HADOOP-14062-branch-2.8.0.dummy.patch, > yarn-rm-log.txt > > > When privacy is enabled for RPC (hadoop.rpc.protection = privacy), > {{ApplicationMasterProtocolPBClientImpl.allocate}} sometimes (but not always) > fails with an EOFException. I've reproduced this with Spark 2.0.2 built > against latest branch-2.8 and with a simple distcp job on latest branch-2.8. > Steps to reproduce using distcp: > 1. Set hadoop.rpc.protection equal to privacy > 2. Write data to HDFS. I did this with Spark as follows: > {code} > sc.parallelize(1 to (5*1024*1024)).map(k => Seq(k, > org.apache.commons.lang.RandomStringUtils.random(1024, > "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWxyZ0123456789")).mkString("|")).toDF().repartition(100).write.parquet("hdfs:///tmp/testData") > {code} > 3. Attempt to distcp that data to another location in HDFS. For example: > {code} > hadoop distcp -Dmapreduce.framework.name=yarn hdfs:///tmp/testData > hdfs:///tmp/testDataCopy > {code} > I observed this error in the ApplicationMaster's syslog: > {code} > 2016-12-19 19:13:50,097 INFO [eventHandlingThread] > org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer > setup for JobId: job_1482189777425_0004, File: > hdfs://:8020/tmp/hadoop-yarn/staging//.staging/job_1482189777425_0004/job_1482189777425_0004_1.jhist > 2016-12-19 19:13:51,004 INFO [RMCommunicator Allocator] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before > Scheduling: PendingReds:0 ScheduledMaps:4 ScheduledReds:0 AssignedMaps:0 > AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0 > HostLocal:0 RackLocal:0 > 2016-12-19 19:13:51,031 INFO [RMCommunicator Allocator] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() > for application_1482189777425_0004: ask=1 release= 0 newContainers=0 > finishedContainers=0 resourcelimit= knownNMs=3 > 2016-12-19 19:13:52,043 INFO [RMCommunicator Allocator] > org.apache.hadoop.io.retry.RetryInvocationHandler: Exception while invoking > ApplicationMasterProtocolPBClientImpl.allocate over null. Retrying after > sleeping for 3ms. > java.io.EOFException: End of File Exception between local host is: > "/"; destination host is: "":8030; > : java.io.EOFException; For more details see: > http://wiki.apache.org/hadoop/EOFException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:422) > at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:801) > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:765) > at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1486) > at org.apache.hadoop.ipc.Client.call(Client.java:1428) > at org.apache.hadoop.ipc.Client.call(Client.java:1338) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) > at com.sun.proxy.$Proxy80.allocate(Unknown Source) > at > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:3
[jira] [Updated] (HADOOP-14062) ApplicationMasterProtocolPBClientImpl.allocate fails with EOFException when RPC privacy is enabled
[ https://issues.apache.org/jira/browse/HADOOP-14062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steven Rand updated HADOOP-14062: - Status: Patch Available (was: Reopened) > ApplicationMasterProtocolPBClientImpl.allocate fails with EOFException when > RPC privacy is enabled > -- > > Key: HADOOP-14062 > URL: https://issues.apache.org/jira/browse/HADOOP-14062 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.0 >Reporter: Steven Rand >Assignee: Steven Rand >Priority: Critical > Fix For: 2.8.0, 3.0.0-alpha3 > > Attachments: HADOOP-14062.001.patch, HADOOP-14062.002.patch, > HADOOP-14062.003.patch, HADOOP-14062.004.patch, > HADOOP-14062-branch-2.8.0.004.patch, HADOOP-14062-branch-2.8.0.005.patch, > HADOOP-14062-branch-2.8.0.005.patch, HADOOP-14062-branch-2.8.0.dummy.patch, > yarn-rm-log.txt > > > When privacy is enabled for RPC (hadoop.rpc.protection = privacy), > {{ApplicationMasterProtocolPBClientImpl.allocate}} sometimes (but not always) > fails with an EOFException. I've reproduced this with Spark 2.0.2 built > against latest branch-2.8 and with a simple distcp job on latest branch-2.8. > Steps to reproduce using distcp: > 1. Set hadoop.rpc.protection equal to privacy > 2. Write data to HDFS. I did this with Spark as follows: > {code} > sc.parallelize(1 to (5*1024*1024)).map(k => Seq(k, > org.apache.commons.lang.RandomStringUtils.random(1024, > "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWxyZ0123456789")).mkString("|")).toDF().repartition(100).write.parquet("hdfs:///tmp/testData") > {code} > 3. Attempt to distcp that data to another location in HDFS. For example: > {code} > hadoop distcp -Dmapreduce.framework.name=yarn hdfs:///tmp/testData > hdfs:///tmp/testDataCopy > {code} > I observed this error in the ApplicationMaster's syslog: > {code} > 2016-12-19 19:13:50,097 INFO [eventHandlingThread] > org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer > setup for JobId: job_1482189777425_0004, File: > hdfs://:8020/tmp/hadoop-yarn/staging//.staging/job_1482189777425_0004/job_1482189777425_0004_1.jhist > 2016-12-19 19:13:51,004 INFO [RMCommunicator Allocator] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before > Scheduling: PendingReds:0 ScheduledMaps:4 ScheduledReds:0 AssignedMaps:0 > AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0 > HostLocal:0 RackLocal:0 > 2016-12-19 19:13:51,031 INFO [RMCommunicator Allocator] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() > for application_1482189777425_0004: ask=1 release= 0 newContainers=0 > finishedContainers=0 resourcelimit= knownNMs=3 > 2016-12-19 19:13:52,043 INFO [RMCommunicator Allocator] > org.apache.hadoop.io.retry.RetryInvocationHandler: Exception while invoking > ApplicationMasterProtocolPBClientImpl.allocate over null. Retrying after > sleeping for 3ms. > java.io.EOFException: End of File Exception between local host is: > "/"; destination host is: "":8030; > : java.io.EOFException; For more details see: > http://wiki.apache.org/hadoop/EOFException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:422) > at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:801) > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:765) > at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1486) > at org.apache.hadoop.ipc.Client.call(Client.java:1428) > at org.apache.hadoop.ipc.Client.call(Client.java:1338) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) > at com.sun.proxy.$Proxy80.allocate(Unknown Source) > at > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:398) > at > org.apache.hadoop.io.retry.RetryInvocationHandle
[jira] [Updated] (HADOOP-14062) ApplicationMasterProtocolPBClientImpl.allocate fails with EOFException when RPC privacy is enabled
[ https://issues.apache.org/jira/browse/HADOOP-14062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steven Rand updated HADOOP-14062: - Attachment: HADOOP-14062.004.patch Attaching a new patch for trunk. > ApplicationMasterProtocolPBClientImpl.allocate fails with EOFException when > RPC privacy is enabled > -- > > Key: HADOOP-14062 > URL: https://issues.apache.org/jira/browse/HADOOP-14062 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.0 >Reporter: Steven Rand >Assignee: Steven Rand >Priority: Critical > Fix For: 2.8.0, 3.0.0-alpha3 > > Attachments: HADOOP-14062.001.patch, HADOOP-14062.002.patch, > HADOOP-14062.003.patch, HADOOP-14062.004.patch, > HADOOP-14062-branch-2.8.0.004.patch, HADOOP-14062-branch-2.8.0.005.patch, > HADOOP-14062-branch-2.8.0.005.patch, HADOOP-14062-branch-2.8.0.dummy.patch, > yarn-rm-log.txt > > > When privacy is enabled for RPC (hadoop.rpc.protection = privacy), > {{ApplicationMasterProtocolPBClientImpl.allocate}} sometimes (but not always) > fails with an EOFException. I've reproduced this with Spark 2.0.2 built > against latest branch-2.8 and with a simple distcp job on latest branch-2.8. > Steps to reproduce using distcp: > 1. Set hadoop.rpc.protection equal to privacy > 2. Write data to HDFS. I did this with Spark as follows: > {code} > sc.parallelize(1 to (5*1024*1024)).map(k => Seq(k, > org.apache.commons.lang.RandomStringUtils.random(1024, > "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWxyZ0123456789")).mkString("|")).toDF().repartition(100).write.parquet("hdfs:///tmp/testData") > {code} > 3. Attempt to distcp that data to another location in HDFS. For example: > {code} > hadoop distcp -Dmapreduce.framework.name=yarn hdfs:///tmp/testData > hdfs:///tmp/testDataCopy > {code} > I observed this error in the ApplicationMaster's syslog: > {code} > 2016-12-19 19:13:50,097 INFO [eventHandlingThread] > org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer > setup for JobId: job_1482189777425_0004, File: > hdfs://:8020/tmp/hadoop-yarn/staging//.staging/job_1482189777425_0004/job_1482189777425_0004_1.jhist > 2016-12-19 19:13:51,004 INFO [RMCommunicator Allocator] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before > Scheduling: PendingReds:0 ScheduledMaps:4 ScheduledReds:0 AssignedMaps:0 > AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0 > HostLocal:0 RackLocal:0 > 2016-12-19 19:13:51,031 INFO [RMCommunicator Allocator] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() > for application_1482189777425_0004: ask=1 release= 0 newContainers=0 > finishedContainers=0 resourcelimit= knownNMs=3 > 2016-12-19 19:13:52,043 INFO [RMCommunicator Allocator] > org.apache.hadoop.io.retry.RetryInvocationHandler: Exception while invoking > ApplicationMasterProtocolPBClientImpl.allocate over null. Retrying after > sleeping for 3ms. > java.io.EOFException: End of File Exception between local host is: > "/"; destination host is: "":8030; > : java.io.EOFException; For more details see: > http://wiki.apache.org/hadoop/EOFException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:422) > at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:801) > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:765) > at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1486) > at org.apache.hadoop.ipc.Client.call(Client.java:1428) > at org.apache.hadoop.ipc.Client.call(Client.java:1338) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) > at com.sun.proxy.$Proxy80.allocate(Unknown Source) > at > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:398) > at > org.apache.hadoop.io
[jira] [Updated] (HADOOP-14062) ApplicationMasterProtocolPBClientImpl.allocate fails with EOFException when RPC privacy is enabled
[ https://issues.apache.org/jira/browse/HADOOP-14062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated HADOOP-14062: - Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0-alpha3 2.8.0 Target Version/s: 2.8.0, 3.0.0-alpha3 Status: Resolved (was: Patch Available) > ApplicationMasterProtocolPBClientImpl.allocate fails with EOFException when > RPC privacy is enabled > -- > > Key: HADOOP-14062 > URL: https://issues.apache.org/jira/browse/HADOOP-14062 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.0 >Reporter: Steven Rand >Assignee: Steven Rand >Priority: Critical > Fix For: 2.8.0, 3.0.0-alpha3 > > Attachments: HADOOP-14062.001.patch, HADOOP-14062.002.patch, > HADOOP-14062.003.patch, HADOOP-14062-branch-2.8.0.004.patch, > HADOOP-14062-branch-2.8.0.005.patch, HADOOP-14062-branch-2.8.0.005.patch, > HADOOP-14062-branch-2.8.0.dummy.patch, yarn-rm-log.txt > > > When privacy is enabled for RPC (hadoop.rpc.protection = privacy), > {{ApplicationMasterProtocolPBClientImpl.allocate}} sometimes (but not always) > fails with an EOFException. I've reproduced this with Spark 2.0.2 built > against latest branch-2.8 and with a simple distcp job on latest branch-2.8. > Steps to reproduce using distcp: > 1. Set hadoop.rpc.protection equal to privacy > 2. Write data to HDFS. I did this with Spark as follows: > {code} > sc.parallelize(1 to (5*1024*1024)).map(k => Seq(k, > org.apache.commons.lang.RandomStringUtils.random(1024, > "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWxyZ0123456789")).mkString("|")).toDF().repartition(100).write.parquet("hdfs:///tmp/testData") > {code} > 3. Attempt to distcp that data to another location in HDFS. For example: > {code} > hadoop distcp -Dmapreduce.framework.name=yarn hdfs:///tmp/testData > hdfs:///tmp/testDataCopy > {code} > I observed this error in the ApplicationMaster's syslog: > {code} > 2016-12-19 19:13:50,097 INFO [eventHandlingThread] > org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer > setup for JobId: job_1482189777425_0004, File: > hdfs://:8020/tmp/hadoop-yarn/staging//.staging/job_1482189777425_0004/job_1482189777425_0004_1.jhist > 2016-12-19 19:13:51,004 INFO [RMCommunicator Allocator] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before > Scheduling: PendingReds:0 ScheduledMaps:4 ScheduledReds:0 AssignedMaps:0 > AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0 > HostLocal:0 RackLocal:0 > 2016-12-19 19:13:51,031 INFO [RMCommunicator Allocator] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() > for application_1482189777425_0004: ask=1 release= 0 newContainers=0 > finishedContainers=0 resourcelimit= knownNMs=3 > 2016-12-19 19:13:52,043 INFO [RMCommunicator Allocator] > org.apache.hadoop.io.retry.RetryInvocationHandler: Exception while invoking > ApplicationMasterProtocolPBClientImpl.allocate over null. Retrying after > sleeping for 3ms. > java.io.EOFException: End of File Exception between local host is: > "/"; destination host is: "":8030; > : java.io.EOFException; For more details see: > http://wiki.apache.org/hadoop/EOFException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:422) > at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:801) > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:765) > at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1486) > at org.apache.hadoop.ipc.Client.call(Client.java:1428) > at org.apache.hadoop.ipc.Client.call(Client.java:1338) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) > at com.sun.proxy.$Proxy80.allocate(Unknown Source) > at > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at > org.apache.hadoop.i
[jira] [Updated] (HADOOP-14062) ApplicationMasterProtocolPBClientImpl.allocate fails with EOFException when RPC privacy is enabled
[ https://issues.apache.org/jira/browse/HADOOP-14062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated HADOOP-14062: - Status: Patch Available (was: Open) > ApplicationMasterProtocolPBClientImpl.allocate fails with EOFException when > RPC privacy is enabled > -- > > Key: HADOOP-14062 > URL: https://issues.apache.org/jira/browse/HADOOP-14062 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.0 >Reporter: Steven Rand >Priority: Critical > Attachments: HADOOP-14062.001.patch, HADOOP-14062.002.patch, > HADOOP-14062.003.patch, HADOOP-14062-branch-2.8.0.004.patch, > HADOOP-14062-branch-2.8.0.005.patch, HADOOP-14062-branch-2.8.0.005.patch, > HADOOP-14062-branch-2.8.0.dummy.patch, yarn-rm-log.txt > > > When privacy is enabled for RPC (hadoop.rpc.protection = privacy), > {{ApplicationMasterProtocolPBClientImpl.allocate}} sometimes (but not always) > fails with an EOFException. I've reproduced this with Spark 2.0.2 built > against latest branch-2.8 and with a simple distcp job on latest branch-2.8. > Steps to reproduce using distcp: > 1. Set hadoop.rpc.protection equal to privacy > 2. Write data to HDFS. I did this with Spark as follows: > {code} > sc.parallelize(1 to (5*1024*1024)).map(k => Seq(k, > org.apache.commons.lang.RandomStringUtils.random(1024, > "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWxyZ0123456789")).mkString("|")).toDF().repartition(100).write.parquet("hdfs:///tmp/testData") > {code} > 3. Attempt to distcp that data to another location in HDFS. For example: > {code} > hadoop distcp -Dmapreduce.framework.name=yarn hdfs:///tmp/testData > hdfs:///tmp/testDataCopy > {code} > I observed this error in the ApplicationMaster's syslog: > {code} > 2016-12-19 19:13:50,097 INFO [eventHandlingThread] > org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer > setup for JobId: job_1482189777425_0004, File: > hdfs://:8020/tmp/hadoop-yarn/staging//.staging/job_1482189777425_0004/job_1482189777425_0004_1.jhist > 2016-12-19 19:13:51,004 INFO [RMCommunicator Allocator] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before > Scheduling: PendingReds:0 ScheduledMaps:4 ScheduledReds:0 AssignedMaps:0 > AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0 > HostLocal:0 RackLocal:0 > 2016-12-19 19:13:51,031 INFO [RMCommunicator Allocator] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() > for application_1482189777425_0004: ask=1 release= 0 newContainers=0 > finishedContainers=0 resourcelimit= knownNMs=3 > 2016-12-19 19:13:52,043 INFO [RMCommunicator Allocator] > org.apache.hadoop.io.retry.RetryInvocationHandler: Exception while invoking > ApplicationMasterProtocolPBClientImpl.allocate over null. Retrying after > sleeping for 3ms. > java.io.EOFException: End of File Exception between local host is: > "/"; destination host is: "":8030; > : java.io.EOFException; For more details see: > http://wiki.apache.org/hadoop/EOFException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:422) > at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:801) > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:765) > at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1486) > at org.apache.hadoop.ipc.Client.call(Client.java:1428) > at org.apache.hadoop.ipc.Client.call(Client.java:1338) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) > at com.sun.proxy.$Proxy80.allocate(Unknown Source) > at > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:398) > at > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163) > at > org.apache.hadoop.io.retry.RetryInvocationHandler$Ca
[jira] [Updated] (HADOOP-14062) ApplicationMasterProtocolPBClientImpl.allocate fails with EOFException when RPC privacy is enabled
[ https://issues.apache.org/jira/browse/HADOOP-14062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated HADOOP-14062: - Status: Open (was: Patch Available) > ApplicationMasterProtocolPBClientImpl.allocate fails with EOFException when > RPC privacy is enabled > -- > > Key: HADOOP-14062 > URL: https://issues.apache.org/jira/browse/HADOOP-14062 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.0 >Reporter: Steven Rand >Priority: Critical > Attachments: HADOOP-14062.001.patch, HADOOP-14062.002.patch, > HADOOP-14062.003.patch, HADOOP-14062-branch-2.8.0.004.patch, > HADOOP-14062-branch-2.8.0.005.patch, HADOOP-14062-branch-2.8.0.005.patch, > HADOOP-14062-branch-2.8.0.dummy.patch, yarn-rm-log.txt > > > When privacy is enabled for RPC (hadoop.rpc.protection = privacy), > {{ApplicationMasterProtocolPBClientImpl.allocate}} sometimes (but not always) > fails with an EOFException. I've reproduced this with Spark 2.0.2 built > against latest branch-2.8 and with a simple distcp job on latest branch-2.8. > Steps to reproduce using distcp: > 1. Set hadoop.rpc.protection equal to privacy > 2. Write data to HDFS. I did this with Spark as follows: > {code} > sc.parallelize(1 to (5*1024*1024)).map(k => Seq(k, > org.apache.commons.lang.RandomStringUtils.random(1024, > "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWxyZ0123456789")).mkString("|")).toDF().repartition(100).write.parquet("hdfs:///tmp/testData") > {code} > 3. Attempt to distcp that data to another location in HDFS. For example: > {code} > hadoop distcp -Dmapreduce.framework.name=yarn hdfs:///tmp/testData > hdfs:///tmp/testDataCopy > {code} > I observed this error in the ApplicationMaster's syslog: > {code} > 2016-12-19 19:13:50,097 INFO [eventHandlingThread] > org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer > setup for JobId: job_1482189777425_0004, File: > hdfs://:8020/tmp/hadoop-yarn/staging//.staging/job_1482189777425_0004/job_1482189777425_0004_1.jhist > 2016-12-19 19:13:51,004 INFO [RMCommunicator Allocator] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before > Scheduling: PendingReds:0 ScheduledMaps:4 ScheduledReds:0 AssignedMaps:0 > AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0 > HostLocal:0 RackLocal:0 > 2016-12-19 19:13:51,031 INFO [RMCommunicator Allocator] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() > for application_1482189777425_0004: ask=1 release= 0 newContainers=0 > finishedContainers=0 resourcelimit= knownNMs=3 > 2016-12-19 19:13:52,043 INFO [RMCommunicator Allocator] > org.apache.hadoop.io.retry.RetryInvocationHandler: Exception while invoking > ApplicationMasterProtocolPBClientImpl.allocate over null. Retrying after > sleeping for 3ms. > java.io.EOFException: End of File Exception between local host is: > "/"; destination host is: "":8030; > : java.io.EOFException; For more details see: > http://wiki.apache.org/hadoop/EOFException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:422) > at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:801) > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:765) > at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1486) > at org.apache.hadoop.ipc.Client.call(Client.java:1428) > at org.apache.hadoop.ipc.Client.call(Client.java:1338) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) > at com.sun.proxy.$Proxy80.allocate(Unknown Source) > at > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:398) > at > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163) > at > org.apache.hadoop.io.retry.RetryInvocationHandler$Ca
[jira] [Updated] (HADOOP-14062) ApplicationMasterProtocolPBClientImpl.allocate fails with EOFException when RPC privacy is enabled
[ https://issues.apache.org/jira/browse/HADOOP-14062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated HADOOP-14062: - Attachment: HADOOP-14062-branch-2.8.0.dummy.patch > ApplicationMasterProtocolPBClientImpl.allocate fails with EOFException when > RPC privacy is enabled > -- > > Key: HADOOP-14062 > URL: https://issues.apache.org/jira/browse/HADOOP-14062 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.0 >Reporter: Steven Rand >Priority: Critical > Attachments: HADOOP-14062.001.patch, HADOOP-14062.002.patch, > HADOOP-14062.003.patch, HADOOP-14062-branch-2.8.0.004.patch, > HADOOP-14062-branch-2.8.0.005.patch, HADOOP-14062-branch-2.8.0.005.patch, > HADOOP-14062-branch-2.8.0.dummy.patch, yarn-rm-log.txt > > > When privacy is enabled for RPC (hadoop.rpc.protection = privacy), > {{ApplicationMasterProtocolPBClientImpl.allocate}} sometimes (but not always) > fails with an EOFException. I've reproduced this with Spark 2.0.2 built > against latest branch-2.8 and with a simple distcp job on latest branch-2.8. > Steps to reproduce using distcp: > 1. Set hadoop.rpc.protection equal to privacy > 2. Write data to HDFS. I did this with Spark as follows: > {code} > sc.parallelize(1 to (5*1024*1024)).map(k => Seq(k, > org.apache.commons.lang.RandomStringUtils.random(1024, > "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWxyZ0123456789")).mkString("|")).toDF().repartition(100).write.parquet("hdfs:///tmp/testData") > {code} > 3. Attempt to distcp that data to another location in HDFS. For example: > {code} > hadoop distcp -Dmapreduce.framework.name=yarn hdfs:///tmp/testData > hdfs:///tmp/testDataCopy > {code} > I observed this error in the ApplicationMaster's syslog: > {code} > 2016-12-19 19:13:50,097 INFO [eventHandlingThread] > org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer > setup for JobId: job_1482189777425_0004, File: > hdfs://:8020/tmp/hadoop-yarn/staging//.staging/job_1482189777425_0004/job_1482189777425_0004_1.jhist > 2016-12-19 19:13:51,004 INFO [RMCommunicator Allocator] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before > Scheduling: PendingReds:0 ScheduledMaps:4 ScheduledReds:0 AssignedMaps:0 > AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0 > HostLocal:0 RackLocal:0 > 2016-12-19 19:13:51,031 INFO [RMCommunicator Allocator] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() > for application_1482189777425_0004: ask=1 release= 0 newContainers=0 > finishedContainers=0 resourcelimit= knownNMs=3 > 2016-12-19 19:13:52,043 INFO [RMCommunicator Allocator] > org.apache.hadoop.io.retry.RetryInvocationHandler: Exception while invoking > ApplicationMasterProtocolPBClientImpl.allocate over null. Retrying after > sleeping for 3ms. > java.io.EOFException: End of File Exception between local host is: > "/"; destination host is: "":8030; > : java.io.EOFException; For more details see: > http://wiki.apache.org/hadoop/EOFException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:422) > at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:801) > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:765) > at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1486) > at org.apache.hadoop.ipc.Client.call(Client.java:1428) > at org.apache.hadoop.ipc.Client.call(Client.java:1338) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) > at com.sun.proxy.$Proxy80.allocate(Unknown Source) > at > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:398) > at > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163) > at > org.apache.hadoop.io.retry.RetryInvocat
[jira] [Updated] (HADOOP-14062) ApplicationMasterProtocolPBClientImpl.allocate fails with EOFException when RPC privacy is enabled
[ https://issues.apache.org/jira/browse/HADOOP-14062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated HADOOP-14062: - Attachment: HADOOP-14062-branch-2.8.0.005.patch Attached the same branch-2.8 patch to trigger jenkins report > ApplicationMasterProtocolPBClientImpl.allocate fails with EOFException when > RPC privacy is enabled > -- > > Key: HADOOP-14062 > URL: https://issues.apache.org/jira/browse/HADOOP-14062 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.0 >Reporter: Steven Rand >Priority: Critical > Attachments: HADOOP-14062.001.patch, HADOOP-14062.002.patch, > HADOOP-14062.003.patch, HADOOP-14062-branch-2.8.0.004.patch, > HADOOP-14062-branch-2.8.0.005.patch, HADOOP-14062-branch-2.8.0.005.patch, > yarn-rm-log.txt > > > When privacy is enabled for RPC (hadoop.rpc.protection = privacy), > {{ApplicationMasterProtocolPBClientImpl.allocate}} sometimes (but not always) > fails with an EOFException. I've reproduced this with Spark 2.0.2 built > against latest branch-2.8 and with a simple distcp job on latest branch-2.8. > Steps to reproduce using distcp: > 1. Set hadoop.rpc.protection equal to privacy > 2. Write data to HDFS. I did this with Spark as follows: > {code} > sc.parallelize(1 to (5*1024*1024)).map(k => Seq(k, > org.apache.commons.lang.RandomStringUtils.random(1024, > "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWxyZ0123456789")).mkString("|")).toDF().repartition(100).write.parquet("hdfs:///tmp/testData") > {code} > 3. Attempt to distcp that data to another location in HDFS. For example: > {code} > hadoop distcp -Dmapreduce.framework.name=yarn hdfs:///tmp/testData > hdfs:///tmp/testDataCopy > {code} > I observed this error in the ApplicationMaster's syslog: > {code} > 2016-12-19 19:13:50,097 INFO [eventHandlingThread] > org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer > setup for JobId: job_1482189777425_0004, File: > hdfs://:8020/tmp/hadoop-yarn/staging//.staging/job_1482189777425_0004/job_1482189777425_0004_1.jhist > 2016-12-19 19:13:51,004 INFO [RMCommunicator Allocator] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before > Scheduling: PendingReds:0 ScheduledMaps:4 ScheduledReds:0 AssignedMaps:0 > AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0 > HostLocal:0 RackLocal:0 > 2016-12-19 19:13:51,031 INFO [RMCommunicator Allocator] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() > for application_1482189777425_0004: ask=1 release= 0 newContainers=0 > finishedContainers=0 resourcelimit= knownNMs=3 > 2016-12-19 19:13:52,043 INFO [RMCommunicator Allocator] > org.apache.hadoop.io.retry.RetryInvocationHandler: Exception while invoking > ApplicationMasterProtocolPBClientImpl.allocate over null. Retrying after > sleeping for 3ms. > java.io.EOFException: End of File Exception between local host is: > "/"; destination host is: "":8030; > : java.io.EOFException; For more details see: > http://wiki.apache.org/hadoop/EOFException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:422) > at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:801) > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:765) > at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1486) > at org.apache.hadoop.ipc.Client.call(Client.java:1428) > at org.apache.hadoop.ipc.Client.call(Client.java:1338) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) > at com.sun.proxy.$Proxy80.allocate(Unknown Source) > at > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:398) > at > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163) > at > org.apache.hadoop.
[jira] [Updated] (HADOOP-14062) ApplicationMasterProtocolPBClientImpl.allocate fails with EOFException when RPC privacy is enabled
[ https://issues.apache.org/jira/browse/HADOOP-14062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated HADOOP-14062: - Status: Patch Available (was: Open) > ApplicationMasterProtocolPBClientImpl.allocate fails with EOFException when > RPC privacy is enabled > -- > > Key: HADOOP-14062 > URL: https://issues.apache.org/jira/browse/HADOOP-14062 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.0 >Reporter: Steven Rand >Priority: Critical > Attachments: HADOOP-14062.001.patch, HADOOP-14062.002.patch, > HADOOP-14062.003.patch, HADOOP-14062-branch-2.8.0.004.patch, > HADOOP-14062-branch-2.8.0.005.patch, yarn-rm-log.txt > > > When privacy is enabled for RPC (hadoop.rpc.protection = privacy), > {{ApplicationMasterProtocolPBClientImpl.allocate}} sometimes (but not always) > fails with an EOFException. I've reproduced this with Spark 2.0.2 built > against latest branch-2.8 and with a simple distcp job on latest branch-2.8. > Steps to reproduce using distcp: > 1. Set hadoop.rpc.protection equal to privacy > 2. Write data to HDFS. I did this with Spark as follows: > {code} > sc.parallelize(1 to (5*1024*1024)).map(k => Seq(k, > org.apache.commons.lang.RandomStringUtils.random(1024, > "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWxyZ0123456789")).mkString("|")).toDF().repartition(100).write.parquet("hdfs:///tmp/testData") > {code} > 3. Attempt to distcp that data to another location in HDFS. For example: > {code} > hadoop distcp -Dmapreduce.framework.name=yarn hdfs:///tmp/testData > hdfs:///tmp/testDataCopy > {code} > I observed this error in the ApplicationMaster's syslog: > {code} > 2016-12-19 19:13:50,097 INFO [eventHandlingThread] > org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer > setup for JobId: job_1482189777425_0004, File: > hdfs://:8020/tmp/hadoop-yarn/staging//.staging/job_1482189777425_0004/job_1482189777425_0004_1.jhist > 2016-12-19 19:13:51,004 INFO [RMCommunicator Allocator] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before > Scheduling: PendingReds:0 ScheduledMaps:4 ScheduledReds:0 AssignedMaps:0 > AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0 > HostLocal:0 RackLocal:0 > 2016-12-19 19:13:51,031 INFO [RMCommunicator Allocator] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() > for application_1482189777425_0004: ask=1 release= 0 newContainers=0 > finishedContainers=0 resourcelimit= knownNMs=3 > 2016-12-19 19:13:52,043 INFO [RMCommunicator Allocator] > org.apache.hadoop.io.retry.RetryInvocationHandler: Exception while invoking > ApplicationMasterProtocolPBClientImpl.allocate over null. Retrying after > sleeping for 3ms. > java.io.EOFException: End of File Exception between local host is: > "/"; destination host is: "":8030; > : java.io.EOFException; For more details see: > http://wiki.apache.org/hadoop/EOFException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:422) > at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:801) > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:765) > at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1486) > at org.apache.hadoop.ipc.Client.call(Client.java:1428) > at org.apache.hadoop.ipc.Client.call(Client.java:1338) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) > at com.sun.proxy.$Proxy80.allocate(Unknown Source) > at > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:398) > at > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163) > at > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155) > at > org.apache.hadoop.io.r
[jira] [Updated] (HADOOP-14062) ApplicationMasterProtocolPBClientImpl.allocate fails with EOFException when RPC privacy is enabled
[ https://issues.apache.org/jira/browse/HADOOP-14062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steven Rand updated HADOOP-14062: - Attachment: HADOOP-14062.003.patch [~jianhe] I've attached new patches which create a single new test in {{TestAMRMClient}}. There's some weirdness in that they have to create a new instance of {{MiniYARNCluster}} just for that one test, and I wasn't sure of the best way to handle it, so happy to further edit the patches if there's a better way. > ApplicationMasterProtocolPBClientImpl.allocate fails with EOFException when > RPC privacy is enabled > -- > > Key: HADOOP-14062 > URL: https://issues.apache.org/jira/browse/HADOOP-14062 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.0 >Reporter: Steven Rand >Priority: Critical > Attachments: HADOOP-14062.001.patch, HADOOP-14062.002.patch, > HADOOP-14062.003.patch, HADOOP-14062-branch-2.8.0.004.patch, > HADOOP-14062-branch-2.8.0.005.patch, yarn-rm-log.txt > > > When privacy is enabled for RPC (hadoop.rpc.protection = privacy), > {{ApplicationMasterProtocolPBClientImpl.allocate}} sometimes (but not always) > fails with an EOFException. I've reproduced this with Spark 2.0.2 built > against latest branch-2.8 and with a simple distcp job on latest branch-2.8. > Steps to reproduce using distcp: > 1. Set hadoop.rpc.protection equal to privacy > 2. Write data to HDFS. I did this with Spark as follows: > {code} > sc.parallelize(1 to (5*1024*1024)).map(k => Seq(k, > org.apache.commons.lang.RandomStringUtils.random(1024, > "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWxyZ0123456789")).mkString("|")).toDF().repartition(100).write.parquet("hdfs:///tmp/testData") > {code} > 3. Attempt to distcp that data to another location in HDFS. For example: > {code} > hadoop distcp -Dmapreduce.framework.name=yarn hdfs:///tmp/testData > hdfs:///tmp/testDataCopy > {code} > I observed this error in the ApplicationMaster's syslog: > {code} > 2016-12-19 19:13:50,097 INFO [eventHandlingThread] > org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer > setup for JobId: job_1482189777425_0004, File: > hdfs://:8020/tmp/hadoop-yarn/staging//.staging/job_1482189777425_0004/job_1482189777425_0004_1.jhist > 2016-12-19 19:13:51,004 INFO [RMCommunicator Allocator] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before > Scheduling: PendingReds:0 ScheduledMaps:4 ScheduledReds:0 AssignedMaps:0 > AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0 > HostLocal:0 RackLocal:0 > 2016-12-19 19:13:51,031 INFO [RMCommunicator Allocator] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() > for application_1482189777425_0004: ask=1 release= 0 newContainers=0 > finishedContainers=0 resourcelimit= knownNMs=3 > 2016-12-19 19:13:52,043 INFO [RMCommunicator Allocator] > org.apache.hadoop.io.retry.RetryInvocationHandler: Exception while invoking > ApplicationMasterProtocolPBClientImpl.allocate over null. Retrying after > sleeping for 3ms. > java.io.EOFException: End of File Exception between local host is: > "/"; destination host is: "":8030; > : java.io.EOFException; For more details see: > http://wiki.apache.org/hadoop/EOFException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:422) > at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:801) > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:765) > at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1486) > at org.apache.hadoop.ipc.Client.call(Client.java:1428) > at org.apache.hadoop.ipc.Client.call(Client.java:1338) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) > at com.sun.proxy.$Proxy80.allocate(Unknown Source) > at > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at > org.apache.hadoop.io.retry.Retry
[jira] [Updated] (HADOOP-14062) ApplicationMasterProtocolPBClientImpl.allocate fails with EOFException when RPC privacy is enabled
[ https://issues.apache.org/jira/browse/HADOOP-14062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steven Rand updated HADOOP-14062: - Attachment: HADOOP-14062-branch-2.8.0.005.patch > ApplicationMasterProtocolPBClientImpl.allocate fails with EOFException when > RPC privacy is enabled > -- > > Key: HADOOP-14062 > URL: https://issues.apache.org/jira/browse/HADOOP-14062 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.0 >Reporter: Steven Rand >Priority: Critical > Attachments: HADOOP-14062.001.patch, HADOOP-14062.002.patch, > HADOOP-14062-branch-2.8.0.004.patch, HADOOP-14062-branch-2.8.0.005.patch, > yarn-rm-log.txt > > > When privacy is enabled for RPC (hadoop.rpc.protection = privacy), > {{ApplicationMasterProtocolPBClientImpl.allocate}} sometimes (but not always) > fails with an EOFException. I've reproduced this with Spark 2.0.2 built > against latest branch-2.8 and with a simple distcp job on latest branch-2.8. > Steps to reproduce using distcp: > 1. Set hadoop.rpc.protection equal to privacy > 2. Write data to HDFS. I did this with Spark as follows: > {code} > sc.parallelize(1 to (5*1024*1024)).map(k => Seq(k, > org.apache.commons.lang.RandomStringUtils.random(1024, > "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWxyZ0123456789")).mkString("|")).toDF().repartition(100).write.parquet("hdfs:///tmp/testData") > {code} > 3. Attempt to distcp that data to another location in HDFS. For example: > {code} > hadoop distcp -Dmapreduce.framework.name=yarn hdfs:///tmp/testData > hdfs:///tmp/testDataCopy > {code} > I observed this error in the ApplicationMaster's syslog: > {code} > 2016-12-19 19:13:50,097 INFO [eventHandlingThread] > org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer > setup for JobId: job_1482189777425_0004, File: > hdfs://:8020/tmp/hadoop-yarn/staging//.staging/job_1482189777425_0004/job_1482189777425_0004_1.jhist > 2016-12-19 19:13:51,004 INFO [RMCommunicator Allocator] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before > Scheduling: PendingReds:0 ScheduledMaps:4 ScheduledReds:0 AssignedMaps:0 > AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0 > HostLocal:0 RackLocal:0 > 2016-12-19 19:13:51,031 INFO [RMCommunicator Allocator] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() > for application_1482189777425_0004: ask=1 release= 0 newContainers=0 > finishedContainers=0 resourcelimit= knownNMs=3 > 2016-12-19 19:13:52,043 INFO [RMCommunicator Allocator] > org.apache.hadoop.io.retry.RetryInvocationHandler: Exception while invoking > ApplicationMasterProtocolPBClientImpl.allocate over null. Retrying after > sleeping for 3ms. > java.io.EOFException: End of File Exception between local host is: > "/"; destination host is: "":8030; > : java.io.EOFException; For more details see: > http://wiki.apache.org/hadoop/EOFException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:422) > at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:801) > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:765) > at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1486) > at org.apache.hadoop.ipc.Client.call(Client.java:1428) > at org.apache.hadoop.ipc.Client.call(Client.java:1338) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) > at com.sun.proxy.$Proxy80.allocate(Unknown Source) > at > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:398) > at > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163) > at > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155) > at > org.apache.hadoop.io.retry.
[jira] [Updated] (HADOOP-14062) ApplicationMasterProtocolPBClientImpl.allocate fails with EOFException when RPC privacy is enabled
[ https://issues.apache.org/jira/browse/HADOOP-14062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steven Rand updated HADOOP-14062: - Attachment: (was: YARN-6013-branch-2.8.0.003.patch) > ApplicationMasterProtocolPBClientImpl.allocate fails with EOFException when > RPC privacy is enabled > -- > > Key: HADOOP-14062 > URL: https://issues.apache.org/jira/browse/HADOOP-14062 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.0 >Reporter: Steven Rand >Priority: Critical > Attachments: HADOOP-14062.001.patch, HADOOP-14062.002.patch, > HADOOP-14062-branch-2.8.0.004.patch, yarn-rm-log.txt > > > When privacy is enabled for RPC (hadoop.rpc.protection = privacy), > {{ApplicationMasterProtocolPBClientImpl.allocate}} sometimes (but not always) > fails with an EOFException. I've reproduced this with Spark 2.0.2 built > against latest branch-2.8 and with a simple distcp job on latest branch-2.8. > Steps to reproduce using distcp: > 1. Set hadoop.rpc.protection equal to privacy > 2. Write data to HDFS. I did this with Spark as follows: > {code} > sc.parallelize(1 to (5*1024*1024)).map(k => Seq(k, > org.apache.commons.lang.RandomStringUtils.random(1024, > "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWxyZ0123456789")).mkString("|")).toDF().repartition(100).write.parquet("hdfs:///tmp/testData") > {code} > 3. Attempt to distcp that data to another location in HDFS. For example: > {code} > hadoop distcp -Dmapreduce.framework.name=yarn hdfs:///tmp/testData > hdfs:///tmp/testDataCopy > {code} > I observed this error in the ApplicationMaster's syslog: > {code} > 2016-12-19 19:13:50,097 INFO [eventHandlingThread] > org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer > setup for JobId: job_1482189777425_0004, File: > hdfs://:8020/tmp/hadoop-yarn/staging//.staging/job_1482189777425_0004/job_1482189777425_0004_1.jhist > 2016-12-19 19:13:51,004 INFO [RMCommunicator Allocator] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before > Scheduling: PendingReds:0 ScheduledMaps:4 ScheduledReds:0 AssignedMaps:0 > AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0 > HostLocal:0 RackLocal:0 > 2016-12-19 19:13:51,031 INFO [RMCommunicator Allocator] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() > for application_1482189777425_0004: ask=1 release= 0 newContainers=0 > finishedContainers=0 resourcelimit= knownNMs=3 > 2016-12-19 19:13:52,043 INFO [RMCommunicator Allocator] > org.apache.hadoop.io.retry.RetryInvocationHandler: Exception while invoking > ApplicationMasterProtocolPBClientImpl.allocate over null. Retrying after > sleeping for 3ms. > java.io.EOFException: End of File Exception between local host is: > "/"; destination host is: "":8030; > : java.io.EOFException; For more details see: > http://wiki.apache.org/hadoop/EOFException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:422) > at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:801) > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:765) > at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1486) > at org.apache.hadoop.ipc.Client.call(Client.java:1428) > at org.apache.hadoop.ipc.Client.call(Client.java:1338) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) > at com.sun.proxy.$Proxy80.allocate(Unknown Source) > at > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:398) > at > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163) > at > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155) > at > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invo
[jira] [Updated] (HADOOP-14062) ApplicationMasterProtocolPBClientImpl.allocate fails with EOFException when RPC privacy is enabled
[ https://issues.apache.org/jira/browse/HADOOP-14062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steven Rand updated HADOOP-14062: - Attachment: (was: YARN-6013-branch-2.8.0.002.patch) > ApplicationMasterProtocolPBClientImpl.allocate fails with EOFException when > RPC privacy is enabled > -- > > Key: HADOOP-14062 > URL: https://issues.apache.org/jira/browse/HADOOP-14062 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.0 >Reporter: Steven Rand >Priority: Critical > Attachments: HADOOP-14062.001.patch, HADOOP-14062.002.patch, > HADOOP-14062-branch-2.8.0.004.patch, yarn-rm-log.txt > > > When privacy is enabled for RPC (hadoop.rpc.protection = privacy), > {{ApplicationMasterProtocolPBClientImpl.allocate}} sometimes (but not always) > fails with an EOFException. I've reproduced this with Spark 2.0.2 built > against latest branch-2.8 and with a simple distcp job on latest branch-2.8. > Steps to reproduce using distcp: > 1. Set hadoop.rpc.protection equal to privacy > 2. Write data to HDFS. I did this with Spark as follows: > {code} > sc.parallelize(1 to (5*1024*1024)).map(k => Seq(k, > org.apache.commons.lang.RandomStringUtils.random(1024, > "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWxyZ0123456789")).mkString("|")).toDF().repartition(100).write.parquet("hdfs:///tmp/testData") > {code} > 3. Attempt to distcp that data to another location in HDFS. For example: > {code} > hadoop distcp -Dmapreduce.framework.name=yarn hdfs:///tmp/testData > hdfs:///tmp/testDataCopy > {code} > I observed this error in the ApplicationMaster's syslog: > {code} > 2016-12-19 19:13:50,097 INFO [eventHandlingThread] > org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer > setup for JobId: job_1482189777425_0004, File: > hdfs://:8020/tmp/hadoop-yarn/staging//.staging/job_1482189777425_0004/job_1482189777425_0004_1.jhist > 2016-12-19 19:13:51,004 INFO [RMCommunicator Allocator] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before > Scheduling: PendingReds:0 ScheduledMaps:4 ScheduledReds:0 AssignedMaps:0 > AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0 > HostLocal:0 RackLocal:0 > 2016-12-19 19:13:51,031 INFO [RMCommunicator Allocator] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() > for application_1482189777425_0004: ask=1 release= 0 newContainers=0 > finishedContainers=0 resourcelimit= knownNMs=3 > 2016-12-19 19:13:52,043 INFO [RMCommunicator Allocator] > org.apache.hadoop.io.retry.RetryInvocationHandler: Exception while invoking > ApplicationMasterProtocolPBClientImpl.allocate over null. Retrying after > sleeping for 3ms. > java.io.EOFException: End of File Exception between local host is: > "/"; destination host is: "":8030; > : java.io.EOFException; For more details see: > http://wiki.apache.org/hadoop/EOFException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:422) > at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:801) > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:765) > at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1486) > at org.apache.hadoop.ipc.Client.call(Client.java:1428) > at org.apache.hadoop.ipc.Client.call(Client.java:1338) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) > at com.sun.proxy.$Proxy80.allocate(Unknown Source) > at > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:398) > at > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163) > at > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155) > at > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invo
[jira] [Updated] (HADOOP-14062) ApplicationMasterProtocolPBClientImpl.allocate fails with EOFException when RPC privacy is enabled
[ https://issues.apache.org/jira/browse/HADOOP-14062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steven Rand updated HADOOP-14062: - Attachment: HADOOP-14062.002.patch HADOOP-14062-branch-2.8.0.004.patch Attaching updated patches for branch-2.8.0 and trunk. > ApplicationMasterProtocolPBClientImpl.allocate fails with EOFException when > RPC privacy is enabled > -- > > Key: HADOOP-14062 > URL: https://issues.apache.org/jira/browse/HADOOP-14062 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.0 >Reporter: Steven Rand >Priority: Critical > Attachments: HADOOP-14062.001.patch, HADOOP-14062.002.patch, > HADOOP-14062-branch-2.8.0.004.patch, YARN-6013-branch-2.8.0.002.patch, > YARN-6013-branch-2.8.0.003.patch, yarn-rm-log.txt > > > When privacy is enabled for RPC (hadoop.rpc.protection = privacy), > {{ApplicationMasterProtocolPBClientImpl.allocate}} sometimes (but not always) > fails with an EOFException. I've reproduced this with Spark 2.0.2 built > against latest branch-2.8 and with a simple distcp job on latest branch-2.8. > Steps to reproduce using distcp: > 1. Set hadoop.rpc.protection equal to privacy > 2. Write data to HDFS. I did this with Spark as follows: > {code} > sc.parallelize(1 to (5*1024*1024)).map(k => Seq(k, > org.apache.commons.lang.RandomStringUtils.random(1024, > "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWxyZ0123456789")).mkString("|")).toDF().repartition(100).write.parquet("hdfs:///tmp/testData") > {code} > 3. Attempt to distcp that data to another location in HDFS. For example: > {code} > hadoop distcp -Dmapreduce.framework.name=yarn hdfs:///tmp/testData > hdfs:///tmp/testDataCopy > {code} > I observed this error in the ApplicationMaster's syslog: > {code} > 2016-12-19 19:13:50,097 INFO [eventHandlingThread] > org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer > setup for JobId: job_1482189777425_0004, File: > hdfs://:8020/tmp/hadoop-yarn/staging//.staging/job_1482189777425_0004/job_1482189777425_0004_1.jhist > 2016-12-19 19:13:51,004 INFO [RMCommunicator Allocator] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before > Scheduling: PendingReds:0 ScheduledMaps:4 ScheduledReds:0 AssignedMaps:0 > AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0 > HostLocal:0 RackLocal:0 > 2016-12-19 19:13:51,031 INFO [RMCommunicator Allocator] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() > for application_1482189777425_0004: ask=1 release= 0 newContainers=0 > finishedContainers=0 resourcelimit= knownNMs=3 > 2016-12-19 19:13:52,043 INFO [RMCommunicator Allocator] > org.apache.hadoop.io.retry.RetryInvocationHandler: Exception while invoking > ApplicationMasterProtocolPBClientImpl.allocate over null. Retrying after > sleeping for 3ms. > java.io.EOFException: End of File Exception between local host is: > "/"; destination host is: "":8030; > : java.io.EOFException; For more details see: > http://wiki.apache.org/hadoop/EOFException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:422) > at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:801) > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:765) > at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1486) > at org.apache.hadoop.ipc.Client.call(Client.java:1428) > at org.apache.hadoop.ipc.Client.call(Client.java:1338) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) > at com.sun.proxy.$Proxy80.allocate(Unknown Source) > at > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:398) > at > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163) > at > org.apache.
[jira] [Updated] (HADOOP-14062) ApplicationMasterProtocolPBClientImpl.allocate fails with EOFException when RPC privacy is enabled
[ https://issues.apache.org/jira/browse/HADOOP-14062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steven Rand updated HADOOP-14062: - Attachment: HADOOP-14062.001.patch Reproduced the issue on trunk, and also that the same change to {{Client}} makes the tests pass. Attaching the same patch for trunk. Today I'll try to find a nicer way of writing a unit test. > ApplicationMasterProtocolPBClientImpl.allocate fails with EOFException when > RPC privacy is enabled > -- > > Key: HADOOP-14062 > URL: https://issues.apache.org/jira/browse/HADOOP-14062 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.0 >Reporter: Steven Rand >Priority: Critical > Attachments: HADOOP-14062.001.patch, > YARN-6013-branch-2.8.0.002.patch, YARN-6013-branch-2.8.0.003.patch, > yarn-rm-log.txt > > > When privacy is enabled for RPC (hadoop.rpc.protection = privacy), > {{ApplicationMasterProtocolPBClientImpl.allocate}} sometimes (but not always) > fails with an EOFException. I've reproduced this with Spark 2.0.2 built > against latest branch-2.8 and with a simple distcp job on latest branch-2.8. > Steps to reproduce using distcp: > 1. Set hadoop.rpc.protection equal to privacy > 2. Write data to HDFS. I did this with Spark as follows: > {code} > sc.parallelize(1 to (5*1024*1024)).map(k => Seq(k, > org.apache.commons.lang.RandomStringUtils.random(1024, > "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWxyZ0123456789")).mkString("|")).toDF().repartition(100).write.parquet("hdfs:///tmp/testData") > {code} > 3. Attempt to distcp that data to another location in HDFS. For example: > {code} > hadoop distcp -Dmapreduce.framework.name=yarn hdfs:///tmp/testData > hdfs:///tmp/testDataCopy > {code} > I observed this error in the ApplicationMaster's syslog: > {code} > 2016-12-19 19:13:50,097 INFO [eventHandlingThread] > org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer > setup for JobId: job_1482189777425_0004, File: > hdfs://:8020/tmp/hadoop-yarn/staging//.staging/job_1482189777425_0004/job_1482189777425_0004_1.jhist > 2016-12-19 19:13:51,004 INFO [RMCommunicator Allocator] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before > Scheduling: PendingReds:0 ScheduledMaps:4 ScheduledReds:0 AssignedMaps:0 > AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0 > HostLocal:0 RackLocal:0 > 2016-12-19 19:13:51,031 INFO [RMCommunicator Allocator] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() > for application_1482189777425_0004: ask=1 release= 0 newContainers=0 > finishedContainers=0 resourcelimit= knownNMs=3 > 2016-12-19 19:13:52,043 INFO [RMCommunicator Allocator] > org.apache.hadoop.io.retry.RetryInvocationHandler: Exception while invoking > ApplicationMasterProtocolPBClientImpl.allocate over null. Retrying after > sleeping for 3ms. > java.io.EOFException: End of File Exception between local host is: > "/"; destination host is: "":8030; > : java.io.EOFException; For more details see: > http://wiki.apache.org/hadoop/EOFException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:422) > at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:801) > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:765) > at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1486) > at org.apache.hadoop.ipc.Client.call(Client.java:1428) > at org.apache.hadoop.ipc.Client.call(Client.java:1338) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) > at com.sun.proxy.$Proxy80.allocate(Unknown Source) > at > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:398) > at > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163
[jira] [Updated] (HADOOP-14062) ApplicationMasterProtocolPBClientImpl.allocate fails with EOFException when RPC privacy is enabled
[ https://issues.apache.org/jira/browse/HADOOP-14062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steven Rand updated HADOOP-14062: - Attachment: YARN-6013-branch-2.8.0.003.patch I'm attaching a patch with a unit test, but would definitely want to refactor the way the test is done. Ideally I think I'd add a test to either {{TestSaslRPC}} or {{TestAMRMClient}}, but I haven't found a way of doing so yet, so the patch in its current form duplicates lots of code in {{TestAMRMClient}}. I'll work on improving it, but wanted to have at least something by the end of today, and this approach was the most obvious / easiest to implement. The test in {{TestAMRMClientWithEncryptedRpc}} fails without the change to {{Client}}, and succeeds with the change. > ApplicationMasterProtocolPBClientImpl.allocate fails with EOFException when > RPC privacy is enabled > -- > > Key: HADOOP-14062 > URL: https://issues.apache.org/jira/browse/HADOOP-14062 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.0 >Reporter: Steven Rand >Priority: Critical > Attachments: YARN-6013-branch-2.8.0.002.patch, > YARN-6013-branch-2.8.0.003.patch, yarn-rm-log.txt > > > When privacy is enabled for RPC (hadoop.rpc.protection = privacy), > {{ApplicationMasterProtocolPBClientImpl.allocate}} sometimes (but not always) > fails with an EOFException. I've reproduced this with Spark 2.0.2 built > against latest branch-2.8 and with a simple distcp job on latest branch-2.8. > Steps to reproduce using distcp: > 1. Set hadoop.rpc.protection equal to privacy > 2. Write data to HDFS. I did this with Spark as follows: > {code} > sc.parallelize(1 to (5*1024*1024)).map(k => Seq(k, > org.apache.commons.lang.RandomStringUtils.random(1024, > "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWxyZ0123456789")).mkString("|")).toDF().repartition(100).write.parquet("hdfs:///tmp/testData") > {code} > 3. Attempt to distcp that data to another location in HDFS. For example: > {code} > hadoop distcp -Dmapreduce.framework.name=yarn hdfs:///tmp/testData > hdfs:///tmp/testDataCopy > {code} > I observed this error in the ApplicationMaster's syslog: > {code} > 2016-12-19 19:13:50,097 INFO [eventHandlingThread] > org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer > setup for JobId: job_1482189777425_0004, File: > hdfs://:8020/tmp/hadoop-yarn/staging//.staging/job_1482189777425_0004/job_1482189777425_0004_1.jhist > 2016-12-19 19:13:51,004 INFO [RMCommunicator Allocator] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before > Scheduling: PendingReds:0 ScheduledMaps:4 ScheduledReds:0 AssignedMaps:0 > AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0 > HostLocal:0 RackLocal:0 > 2016-12-19 19:13:51,031 INFO [RMCommunicator Allocator] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() > for application_1482189777425_0004: ask=1 release= 0 newContainers=0 > finishedContainers=0 resourcelimit= knownNMs=3 > 2016-12-19 19:13:52,043 INFO [RMCommunicator Allocator] > org.apache.hadoop.io.retry.RetryInvocationHandler: Exception while invoking > ApplicationMasterProtocolPBClientImpl.allocate over null. Retrying after > sleeping for 3ms. > java.io.EOFException: End of File Exception between local host is: > "/"; destination host is: "":8030; > : java.io.EOFException; For more details see: > http://wiki.apache.org/hadoop/EOFException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:422) > at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:801) > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:765) > at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1486) > at org.apache.hadoop.ipc.Client.call(Client.java:1428) > at org.apache.hadoop.ipc.Client.call(Client.java:1338) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) > at com.sun.proxy.$Proxy80.allocate(Unknown Source) > at > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at