[jira] [Commented] (MAPREDUCE-6743) nativetask unit tests need to provide usable output; fix link errors during mvn test
[ https://issues.apache.org/jira/browse/MAPREDUCE-6743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15703579#comment-15703579 ] Hudson commented on MAPREDUCE-6743: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10900 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10900/]) MAPREDUCE-6743. nativetask unit tests need to provide usable output; fix (aw: rev be88d574a9a49f384d4e80a661b1a7446ba6a92e) * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/gtest/include/gtest/gtest.h * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/gtest/gtest-all.cc * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/CMakeLists.txt * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/pom.xml > nativetask unit tests need to provide usable output; fix link errors during > mvn test > > > Key: MAPREDUCE-6743 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6743 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: nativetask >Affects Versions: 3.0.0-alpha1 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Fix For: 3.0.0-alpha2 > > Attachments: MAPREDUCE-5743.00.patch, MAPREDUCE-5743.01.patch, > MAPREDUCE-5743.02.patch, MAPREDUCE-5743.03.patch > > > Currently, hadoop-mapreduce-client-nativetask creates a nttest binary which > provides an binary exit code to determine failure. This means there is no > output generated by the Jenkins run to actually debug or provide hints as to > what failed. Given that nttest is written with gtest, it should be > configured to either spit out junit or TAP which can then be used to provide > further analysis. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6787) Allow job_conf.xml to be downloadable on the job overview page in JHS
[
https://issues.apache.org/jira/browse/MAPREDUCE-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15703486#comment-15703486
]
Robert Kanter commented on MAPREDUCE-6787:
--
Actually, how come the file uses {{}} instead of {{}}?
{noformat:title=file from JHS page}
hdfs://0.0.0.0:23010/tmp/hadoop-yarn/staging/history/done/2016/11/28/00/job_1480375039927_0001_conf.xml
...
{noformat}
{noformat:title=file from HDFS}
dfs.journalnode.rpc-address0.0.0.0:8485falsehdfs-default.xmljob.xml
...
{noformat}
> Allow job_conf.xml to be downloadable on the job overview page in JHS
> -
>
> Key: MAPREDUCE-6787
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6787
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobhistoryserver
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Attachments: job_1478210774848_0001.xml, mapreduce6787.001.patch,
> mapreduce6787.002.patch, mapreduce6787.002.patch, mapreduce6787.003.patch
>
>
> The job overview page in JHS provides the path to the job.xml file, but it is
> not a link that users can click on to download the job xml file directly from
> JHS. We could provide a download link in JHS for better usability.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6743) nativetask unit tests need to provide usable output; fix link errors during mvn test
[ https://issues.apache.org/jira/browse/MAPREDUCE-6743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-6743: Resolution: Fixed Fix Version/s: 3.0.0-alpha2 Release Note: As part of this patch, the Google test framework code was updated to v1.8.0 Status: Resolved (was: Patch Available) Thanks. Committed to trunk. > nativetask unit tests need to provide usable output; fix link errors during > mvn test > > > Key: MAPREDUCE-6743 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6743 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: nativetask >Affects Versions: 3.0.0-alpha1 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Fix For: 3.0.0-alpha2 > > Attachments: MAPREDUCE-5743.00.patch, MAPREDUCE-5743.01.patch, > MAPREDUCE-5743.02.patch, MAPREDUCE-5743.03.patch > > > Currently, hadoop-mapreduce-client-nativetask creates a nttest binary which > provides an binary exit code to determine failure. This means there is no > output generated by the Jenkins run to actually debug or provide hints as to > what failed. Given that nttest is written with gtest, it should be > configured to either spit out junit or TAP which can then be used to provide > further analysis. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6787) Allow job_conf.xml to be downloadable on the job overview page in JHS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15703470#comment-15703470 ] Robert Kanter commented on MAPREDUCE-6787: -- +1 on the 003 patch > Allow job_conf.xml to be downloadable on the job overview page in JHS > - > > Key: MAPREDUCE-6787 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6787 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: job_1478210774848_0001.xml, mapreduce6787.001.patch, > mapreduce6787.002.patch, mapreduce6787.002.patch, mapreduce6787.003.patch > > > The job overview page in JHS provides the path to the job.xml file, but it is > not a link that users can click on to download the job xml file directly from > JHS. We could provide a download link in JHS for better usability. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6711) JobImpl fails to handle preemption events on state COMMITTING
[
https://issues.apache.org/jira/browse/MAPREDUCE-6711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15703307#comment-15703307
]
Junping Du commented on MAPREDUCE-6711:
---
Thanks [~Prabhu Joseph] for explanation. That sounds reasonable. Given that, +1
on the latest patch to ignore both events in committing stage. Will commit it
tomorrow if no additional comments.
> JobImpl fails to handle preemption events on state COMMITTING
> -
>
> Key: MAPREDUCE-6711
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6711
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Li Lu
>Assignee: Prabhu Joseph
> Attachments: MAPREDUCE-6711.1.patch, MAPREDUCE-6711.patch
>
>
> When a MR app being preempted on COMMITTING state, we saw the following
> exceptions in its log:
> {code}
> ERROR [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Can't handle this event
> at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
> JOB_TASK_ATTEMPT_COMPLETED at COMMITTING
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:996)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:138)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1289)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1285)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:182)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:108)
> at java.lang.Thread.run(Thread.java:744)
> {code}
> and
> {code}
> ERROR [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Can't handle this event
> at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
> JOB_MAP_TASK_RESCHEDULED at COMMITTING
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:996)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:138)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1289)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1285)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:182)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:108)
> at java.lang.Thread.run(Thread.java:744)
> {code}
> Seems like we need to handle those preemption related events when the job is
> being committed?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Assigned] (MAPREDUCE-6571) JobEndNotification info logs are missing in AM container syslog
[ https://issues.apache.org/jira/browse/MAPREDUCE-6571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen reassigned MAPREDUCE-6571: - Assignee: Haibo Chen > JobEndNotification info logs are missing in AM container syslog > --- > > Key: MAPREDUCE-6571 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6571 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster >Affects Versions: 2.7.0 >Reporter: Prabhu Joseph >Assignee: Haibo Chen >Priority: Minor > > JobEndNotification logs are not written by MRAppMaster and JobEndNotifier > classes even though Log.info is present. The reason was > MRAppMaster.this.stop() has been called before the JobEndNotification and > hence somewhere during the stop log appenders also made null. > AM container syslog is not having below logs from JobEndNotifier >Job end notification trying + urlToNotify >Job end notification to + urlToNotify + succeeded / failed -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6565) Configuration to use host name in delegation token service is not read from job.xml during MapReduce job execution.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702867#comment-15702867
]
Junping Du commented on MAPREDUCE-6565:
---
Patch LGTM too.
> Configuration to use host name in delegation token service is not read from
> job.xml during MapReduce job execution.
> ---
>
> Key: MAPREDUCE-6565
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6565
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Chris Nauroth
>Assignee: Li Lu
> Attachments: MAPREDUCE-6565-trunk.001.patch
>
>
> By default, the service field of a delegation token is populated based on
> server IP address. Setting {{hadoop.security.token.service.use_ip}} to
> {{false}} changes this behavior to use host name instead of IP address.
> However, this configuration property is not read from job.xml. Instead, it's
> read from a separate {{Configuration}} instance created during static
> initialization of {{SecurityUtil}}. This does not work correctly with
> MapReduce jobs if the framework is distributed by setting
> {{mapreduce.application.framework.path}} and the
> {{mapreduce.application.classpath}} is isolated to avoid reading
> core-site.xml from the cluster nodes. MapReduce tasks will fail to
> authenticate to HDFS, because they'll try to find a delegation token based on
> the NameNode IP address, even though at job submission time the tokens were
> generated using the host name.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6565) Configuration to use host name in delegation token service is not read from job.xml during MapReduce job execution.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702812#comment-15702812
]
Jason Lowe commented on MAPREDUCE-6565:
---
Patch looks good to me, +1. It will address the specific issue for MapReduce.
Other frameworks, such as Spark, will need to make similar modifications.
> Configuration to use host name in delegation token service is not read from
> job.xml during MapReduce job execution.
> ---
>
> Key: MAPREDUCE-6565
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6565
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Chris Nauroth
>Assignee: Li Lu
> Attachments: MAPREDUCE-6565-trunk.001.patch
>
>
> By default, the service field of a delegation token is populated based on
> server IP address. Setting {{hadoop.security.token.service.use_ip}} to
> {{false}} changes this behavior to use host name instead of IP address.
> However, this configuration property is not read from job.xml. Instead, it's
> read from a separate {{Configuration}} instance created during static
> initialization of {{SecurityUtil}}. This does not work correctly with
> MapReduce jobs if the framework is distributed by setting
> {{mapreduce.application.framework.path}} and the
> {{mapreduce.application.classpath}} is isolated to avoid reading
> core-site.xml from the cluster nodes. MapReduce tasks will fail to
> authenticate to HDFS, because they'll try to find a delegation token based on
> the NameNode IP address, even though at job submission time the tokens were
> generated using the host name.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6288) mapred job -status fails with AccessControlException
[
https://issues.apache.org/jira/browse/MAPREDUCE-6288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702251#comment-15702251
]
Jason Lowe commented on MAPREDUCE-6288:
---
We need to get this resolved since it's blocking the 2.8 release.
I agree with [~revans2] that the permission change is undesirable. I haven't
seen how his original concerns have been addressed, specifically:
* It exposes implementation details that were never exposed before. If the
backend changes to the timeline server then the client will try to find them in
the filesystem and break.
* Users that have view ACLs for the job but who are _not_ the job's user will
not be able to access the file even with the directory permission change.
* There's a race condition because the client could be trying to access a file
that the history server can be moving at the same time.
I also agree with his analysis on a proper way to fix it:
{quote}
If you want the job conf create a new RPC call for it in the history server and
in the MR AM, then have the client download it through those RPC APIs. This
will not expose any internal detail and will support the ACLs correctly across
the board.
{quote}
It preserves the separation of concerns for the history backend implementation,
properly implements ACLs, and avoids the race condition.
> mapred job -status fails with AccessControlException
> -
>
> Key: MAPREDUCE-6288
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6288
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Blocker
> Attachments: MAPREDUCE-6288-gera-001.patch, MAPREDUCE-6288.002.patch,
> MAPREDUCE-6288.patch
>
>
> After MAPREDUCE-5875, we're seeing this Exception when trying to do {{mapred
> job -status job_1427080398288_0001}}
> {noformat}
> Exception in thread "main" org.apache.hadoop.security.AccessControlException:
> Permission denied: user=jenkins, access=EXECUTE,
> inode="/user/history/done":mapred:hadoop:drwxrwx---
> at
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:257)
> at
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:238)
> at
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkTraverse(DefaultAuthorizationProvider.java:180)
> at
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:137)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:138)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6553)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6535)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPathAccess(FSNamesystem.java:6460)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1919)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1870)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1850)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1822)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:545)
> at
> org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
>
[jira] [Commented] (MAPREDUCE-6743) nativetask unit tests need to provide usable output; fix link errors during mvn test
[ https://issues.apache.org/jira/browse/MAPREDUCE-6743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702107#comment-15702107 ] Akira Ajisaka commented on MAPREDUCE-6743: -- +1 from me, too. > nativetask unit tests need to provide usable output; fix link errors during > mvn test > > > Key: MAPREDUCE-6743 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6743 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: nativetask >Affects Versions: 3.0.0-alpha1 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Attachments: MAPREDUCE-5743.00.patch, MAPREDUCE-5743.01.patch, > MAPREDUCE-5743.02.patch, MAPREDUCE-5743.03.patch > > > Currently, hadoop-mapreduce-client-nativetask creates a nttest binary which > provides an binary exit code to determine failure. This means there is no > output generated by the Jenkins run to actually debug or provide hints as to > what failed. Given that nttest is written with gtest, it should be > configured to either spit out junit or TAP which can then be used to provide > further analysis. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
