[jira] [Assigned] (MAPREDUCE-6878) Method length of MRAppMaster #serviceInit() is too long
[ https://issues.apache.org/jira/browse/MAPREDUCE-6878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton reassigned MAPREDUCE-6878: --- Assignee: Yousef Abu-Salah > Method length of MRAppMaster #serviceInit() is too long > --- > > Key: MAPREDUCE-6878 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6878 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Reporter: Yufei Gu >Assignee: Yousef Abu-Salah >Priority: Minor > Labels: newbie++ > > Reported by style checking: Method length is 216 lines (max allowed is 150). > [MethodLength] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7188) [Clean-up] Remove NULL check before instanceof and fix checkstyle issue in TaskResult
[ https://issues.apache.org/jira/browse/MAPREDUCE-7188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated MAPREDUCE-7188: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.3.0 Status: Resolved (was: Patch Available) +1 for the latest patch. Thanks, [~shwetayakkali], and thanks to [~adam.antal] for the review. Committed to trunk. Rather than go back and forth over trivialities, I fixed one last spacing quibble in the commit. > [Clean-up] Remove NULL check before instanceof and fix checkstyle issue in > TaskResult > - > > Key: MAPREDUCE-7188 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7188 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Shweta >Assignee: Shweta >Priority: Minor > Fix For: 3.3.0 > > Attachments: MAPREDUCE-7188.001.patch, MAPREDUCE-7188.002.patch, > MAPREDUCE-7188.003.patch, MAPREDUCE-7188.004.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7188) [Clean-up] Remove NULL check before instanceof and fix checkstyle issue in TaskResult
[ https://issues.apache.org/jira/browse/MAPREDUCE-7188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16786283#comment-16786283 ] Daniel Templeton commented on MAPREDUCE-7188: - Making the default constructor private will have the same impact as removing it. You can't find the dynamic uses of it from reading the code. I'd say to just leave it. > [Clean-up] Remove NULL check before instanceof and fix checkstyle issue in > TaskResult > - > > Key: MAPREDUCE-7188 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7188 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Shweta >Assignee: Shweta >Priority: Minor > Attachments: MAPREDUCE-7188.001.patch, MAPREDUCE-7188.002.patch, > MAPREDUCE-7188.003.patch, MAPREDUCE-7188.004.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7188) [Clean-up] Remove NULL check before instanceof and fix checkstyle issue in TaskResult
[
https://issues.apache.org/jira/browse/MAPREDUCE-7188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16785120#comment-16785120
]
Daniel Templeton commented on MAPREDUCE-7188:
-
Thanks, [~shwetayakkali]. It's a minor thing, but splitting lines on the dot
is a Scala thing, and we're not writing Scala. It's been slowly creeping into
our source base, but I'd rather not encourage it. Also, are you sure the
{{TaskResult()}} constructor is unused? Sometimes a default constructor is
there to enable instance creation through reflection or for Serialization
purposes. Unit tests might not catch those. Unless you have a good reason to
remove it, I'd leave it.
> [Clean-up] Remove NULL check before instanceof and fix checkstyle issue in
> TaskResult
> -
>
> Key: MAPREDUCE-7188
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7188
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: Shweta
>Assignee: Shweta
>Priority: Minor
> Attachments: MAPREDUCE-7188.001.patch, MAPREDUCE-7188.002.patch,
> MAPREDUCE-7188.003.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Assigned] (MAPREDUCE-7188) [Clean-up] Remove NULL check before instanceof and fix checkstyle issue in TaskResult
[ https://issues.apache.org/jira/browse/MAPREDUCE-7188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton reassigned MAPREDUCE-7188: --- Assignee: Shweta > [Clean-up] Remove NULL check before instanceof and fix checkstyle issue in > TaskResult > - > > Key: MAPREDUCE-7188 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7188 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Shweta >Assignee: Shweta >Priority: Minor > Attachments: MAPREDUCE-7188.001.patch, MAPREDUCE-7188.002.patch, > MAPREDUCE-7188.003.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-7180) Relaunching Failed Containers
[
https://issues.apache.org/jira/browse/MAPREDUCE-7180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782164#comment-16782164
]
Daniel Templeton edited comment on MAPREDUCE-7180 at 3/1/19 10:23 PM:
--
This sounds like a fun discussion. Let me throw in my 2 cents!
The 80/20 should cover most cases, except for blatant misconfiguration. That
makes this feature very niche. In the cases where there is a misconfiguration,
we want the user to fix it, not limp along on the good will of a self-healing
system.
The 80/20 probably isn't going to cover 100% of the cases, and there are
definitely cases where the configuration is out of the user's hands, e.g. Hive.
In those cases, which can be enumerated by the cluster owner, it would make
sense to let the admin turn on the "idiot switch" to hide failures because of
memory overruns behind a smart retry policy.
On the flip side, if we see an MR container fail due to memory overruns, we
could save some resources by not retrying the task. If it fails once on a
memory overrun, it will fail on all the retries as well. Just kill the job and
be done. Since YARN can't know anything about MR, it would have to be the MR
AM that notices and kills itself.
I like the idea of using the power of YARN for good (for once), to hide details
and reduce end-user pain. Yeah, we should be careful to make sure it's not
going to unexpectedly hide problems in a bad way, but I think that should be
pretty easy to do.
was (Author: templedf):
This sounds like a fun discussion. Let me throw in my 2 cents!
The 80/20 should cover most cases, except for blatant misconfiguration. That
makes this feature very niche. In the cases where there is a misconfiguration,
we want the user to fix it, not limp along on the good will of a self-healing
system.
The 80/20 probably isn't going to cover 100% of the cases, and there are
definitely cases where the configuration is out of the user's hands, e.g. Hive.
In those cases, which can be enumerated by the cluster owner, it would make
sense to let the admin turn on the "idiot switch" to hide failures because of
memory overruns behind a smart retry policy.
On the flip side, if we see an MR container fail due to memory overruns, we
could save some resources by not retrying the task. If it fails once on a
memory overrun, it will fail on all the retries as well. Just kill the job and
be done. Since YARN can't know anything about MR, it would have to be the MR
AM that notices and kills itself.
I like the idea of using the power of YARN for good (for once), to hide details
and reduce end-user pain. Yeah, we should be careful to make sure it's not
going to unexpectedly hide problems in a bad way, but I like that's pretty easy
to do.
> Relaunching Failed Containers
> -
>
> Key: MAPREDUCE-7180
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7180
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv1, mrv2
>Reporter: BELUGA BEHR
>Priority: Major
>
> In my experience, it is very common that a MR job completely fails because a
> single Mapper/Reducer container is using more memory than has been reserved
> in YARN. The following message is logging the the MapReduce
> ApplicationMaster:
> {code}
> Container [pid=46028,containerID=container_e54_1435155934213_16721_01_003666]
> is running beyond physical memory limits.
> Current usage: 1.0 GB of 1 GB physical memory used; 2.7 GB of 2.1 GB virtual
> memory used. Killing container.
> {code}
> In this case, the container is re-launched on another node, and of course, it
> is killed again for the same reason. This process happens three (maybe
> four?) times before the entire MapReduce job fails. It's often said that the
> definition of insanity is doing the same thing over and over and expecting
> different results.
> For all intents and purposes, the amount of resources requested by Mappers
> and Reducers is a fixed amount; based on the default configuration values.
> Users can set the memory on a per-job basis, but it's a pain, not exact, and
> requires intimate knowledge of the MapReduce framework and its memory usage
> patterns.
> I propose that if the MR ApplicationMaster detects that a container is killed
> because of this specific memory resource constraint, that it requests a
> larger container for the subsequent task attempt.
> For example, increase the requested memory size by 50% each time the
> container fails and the task is retried. This will prevent many Job failures
> and allow for additional memory tuning, per-Job, after the fact, to get
> better performance (v.s. fail/succeed).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MAPREDUCE-7180) Relaunching Failed Containers
[
https://issues.apache.org/jira/browse/MAPREDUCE-7180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782164#comment-16782164
]
Daniel Templeton commented on MAPREDUCE-7180:
-
This sounds like a fun discussion. Let me throw in my 2 cents!
The 80/20 should cover most cases, except for blatant misconfiguration. That
makes this feature very niche. In the cases where there is a misconfiguration,
we want the user to fix it, not limp along on the good will of a self-healing
system.
The 80/20 probably isn't going to cover 100% of the cases, and there are
definitely cases where the configuration is out of the user's hands, e.g. Hive.
In those cases, which can be enumerated by the cluster owner, it would make
sense to let the admin turn on the "idiot switch" to hide failures because of
memory overruns behind a smart retry policy.
On the flip side, if we see an MR container fail due to memory overruns, we
could save some resources by not retrying the task. If it fails once on a
memory overrun, it will fail on all the retries as well. Just kill the job and
be done. Since YARN can't know anything about MR, it would have to be the MR
AM that notices and kills itself.
I like the idea of using the power of YARN for good (for once), to hide details
and reduce end-user pain. Yeah, we should be careful to make sure it's not
going to unexpectedly hide problems in a bad way, but I like that's pretty easy
to do.
> Relaunching Failed Containers
> -
>
> Key: MAPREDUCE-7180
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7180
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv1, mrv2
>Reporter: BELUGA BEHR
>Priority: Major
>
> In my experience, it is very common that a MR job completely fails because a
> single Mapper/Reducer container is using more memory than has been reserved
> in YARN. The following message is logging the the MapReduce
> ApplicationMaster:
> {code}
> Container [pid=46028,containerID=container_e54_1435155934213_16721_01_003666]
> is running beyond physical memory limits.
> Current usage: 1.0 GB of 1 GB physical memory used; 2.7 GB of 2.1 GB virtual
> memory used. Killing container.
> {code}
> In this case, the container is re-launched on another node, and of course, it
> is killed again for the same reason. This process happens three (maybe
> four?) times before the entire MapReduce job fails. It's often said that the
> definition of insanity is doing the same thing over and over and expecting
> different results.
> For all intents and purposes, the amount of resources requested by Mappers
> and Reducers is a fixed amount; based on the default configuration values.
> Users can set the memory on a per-job basis, but it's a pain, not exact, and
> requires intimate knowledge of the MapReduce framework and its memory usage
> patterns.
> I propose that if the MR ApplicationMaster detects that a container is killed
> because of this specific memory resource constraint, that it requests a
> larger container for the subsequent task attempt.
> For example, increase the requested memory size by 50% each time the
> container fails and the task is retried. This will prevent many Job failures
> and allow for additional memory tuning, per-Job, after the fact, to get
> better performance (v.s. fail/succeed).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7172) Wildcard functionality of -libjar is broken when jars are located in same remote FS
[
https://issues.apache.org/jira/browse/MAPREDUCE-7172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16719005#comment-16719005
]
Daniel Templeton commented on MAPREDUCE-7172:
-
Playing around with a cluster a bit, I don't see an issue. I setup a
single-node cluster using trunk. I submitted a job with two libjars as
suggested by [~leftnoteasy] above. I can see that wild card paths are enabled
in the job.xml. I can see that both libjars are uploaded to the libjars
directory in the job's staging directory and that both libjars are downloaded
into the job's working directory on the node.
Looking at the source code, I also fail to see what the issue is.
[~leftnoteasy], what am I missing?
> Wildcard functionality of -libjar is broken when jars are located in same
> remote FS
> ---
>
> Key: MAPREDUCE-7172
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7172
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Wangda Tan
>Priority: Critical
>
> We recently found that when -libjar specified jars on the same remote FS,
> jars will not be properly added to classpath.
> The reason is MAPREDUCE-6719 added the wildcard functionality, but the follow
> logic assumes files are all placed under job's submission directory. (Inside
> JobResourceUploader)
> {code:java}
> if (useWildcard && !foundFragment) {
> // Add the whole directory to the cache using a wild card
> Path libJarsDirWildcard =
> jtFs.makeQualified(new Path(libjarsDir, DistributedCache.WILDCARD));
> DistributedCache.addCacheFile(libJarsDirWildcard.toUri(), conf);
> }{code}
> However, in the same method, specified resources will be only uploaded when
> two FSes are different, see copyRemoteFiles:
> {code:java}
> if (FileUtil.compareFs(remoteFs, jtFs)) {
> return originalPath;
> } {code}
> Workaround of this issue is pass:
> mapreduce.client.libjars.wildcard = false.
> When the MR job got launched.
> Example commandline to reproduce this issue is:
> {code:java}
> hadoop jar abc.jar org.ABC -libjars
> "wasb://host/path1/jar1,wasb://host/path2/jar2..."{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7007) Add LOG.isDebugEnabled() guard for each LOG.debug()
[
https://issues.apache.org/jira/browse/MAPREDUCE-7007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16250184#comment-16250184
]
Daniel Templeton commented on MAPREDUCE-7007:
-
It's probably worth it where there are concatenations. When doing reviews, I
don't tend to flag minor concatenations in unguarded log statements, but I
probably should.
Just be aware that if you try to fix them all in one big patch, you might have
trouble getting the source base to hold still long enough to get the patch
reviewed and committed.
> Add LOG.isDebugEnabled() guard for each LOG.debug()
> ---
>
> Key: MAPREDUCE-7007
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7007
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: Mehran Hassani
>Priority: Minor
>
> I am conducting research on log related bugs. I tried to make a tool to fix
> repetitive yet simple patterns of bugs that are related to logs. In these
> files, there are debug level logging statements containing multiple string
> concatenation without the if statement before them:
> hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestKeyValueTextInputFormat.java,
> LOG.debug("splits["+j+"]="+splits[j]+" count=" + count);, 123
> hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestTextInputFormat.java,
> LOG.debug("splits["+j+"]="+splits[j]+" count=" + count);, 137
> hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestTextInputFormat.java,
> LOG.debug("splits[" + j + "]=" + splits[j] + " count=" + counter);, 254
> hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestTextInputFormat.java,
> LOG.debug("splits["+j+"]="+splits[j]+" count=" + counter);, 315
> hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestMRKeyValueTextInputFormat.java,
> LOG.debug("splits[" + j + "]=" + splits.get(j) +" count=" + count);, 146
> hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestMRKeyValueTextInputFormat.java,
> LOG.debug("splits["+j+"]="+splits.get(j)+" count=" + count);, 244
> Would you be interested in adding the if before these logging statements?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Assigned] (MAPREDUCE-6837) Add an equivalent to Crunch's Pair class
[
https://issues.apache.org/jira/browse/MAPREDUCE-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton reassigned MAPREDUCE-6837:
---
Assignee: Peter Cseh
> Add an equivalent to Crunch's Pair class
>
>
> Key: MAPREDUCE-6837
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6837
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
>Reporter: Daniel Templeton
>Assignee: Peter Cseh
> Labels: newbie++
>
> Crunch has this great {{Pair}} class
> (https://crunch.apache.org/apidocs/0.14.0/org/apache/crunch/Pair.html) that
> saves one from constantly implementing composite writables. It seems silly
> that we still don't have an equivalent in MR.
> I would like to see a new class with the following API:
> {code}
> package org.apache.hadoop.io;
> public class CompositeWritable WritableComparable> implements WritableComparable {
> public CompositeWritable(P primary, S secondary);
> public P getPrimary();
> public void setPrimary(P primary);
> public S getSecondary();
> public void setSecondary(S secondary);
> // Return true if both primaries and both secondaries are equal
> public boolean equals(CompositeWritable o);
> // Return the primary's hash code
> public long hashCode();
> // Sort first by primary and then by secondary
> public int compareTo(CompositeWritable o);
> public void readFields(DataInput in);
> public void write(DataOutput out);
> }
> {code}
> With such a class, implementing a secondary sort would mean just implementing
> a custom grouping comparator. That comparator could also be implemented as
> part of this JIRA:
> {code}
> package org.apache.hadoop.io;
> public class CompositeGroupingComparator extends WritableComparator {
> ...
> }
> {code}
> Or some such.
> Crunch also provides {{Tuple3}}, {{Tuple4}}, and {{TupleN}} classes, but I
> don't think we need to add equivalents. If someone really wants that
> capability, they can nest composite keys.
> Don't forget to add unit tests!
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6870) Add configuration for MR job to finish when all reducers are complete (even with unfinished mappers)
[ https://issues.apache.org/jira/browse/MAPREDUCE-6870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124318#comment-16124318 ] Daniel Templeton commented on MAPREDUCE-6870: - Good question. The change breaks the current behavior, but only in unusual cases, and there is nothing that explicitly states the contract is that a job will remain running until all the mappers complete. In fact, with speculative execution, it's just the opposite. What you're breaking here is tradition as opposed to an execution contract, so I would say that it's not meaningfully incompatible. However, the contract around applications is that we don't break anything ever if at all possible. So, strictly speaking, this change is incompatible, and we should mark it as such. To go into branch-2, it will have to be off by default. > Add configuration for MR job to finish when all reducers are complete (even > with unfinished mappers) > > > Key: MAPREDUCE-6870 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6870 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 2.6.1 >Reporter: Zhe Zhang >Assignee: Peter Bacsko > Fix For: 3.0.0-beta1 > > Attachments: MAPREDUCE-6870-001.patch, MAPREDUCE-6870-002.patch, > MAPREDUCE-6870-003.patch, MAPREDUCE-6870-004.patch, MAPREDUCE-6870-005.patch, > MAPREDUCE-6870-006.patch, MAPREDUCE-6870-007.patch > > > Even with MAPREDUCE-5817, there could still be cases where mappers get > scheduled before all reducers are complete, but those mappers run for long > time, even after all reducers are complete. This could hurt the performance > of large MR jobs. > In some cases, mappers don't have any materialize-able outcome other than > providing intermediate data to reducers. In that case, the job owner should > have the config option to finish the job once all reducers are complete. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6930) mapreduce.map.cpu.vcores and mapreduce.reduce.cpu.vcores are both present twice in mapred-default.xml
Daniel Templeton created MAPREDUCE-6930: --- Summary: mapreduce.map.cpu.vcores and mapreduce.reduce.cpu.vcores are both present twice in mapred-default.xml Key: MAPREDUCE-6930 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6930 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 3.0.0-alpha4 Reporter: Daniel Templeton The second set should be deleted. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6767) TestSlive fails after a common change
[ https://issues.apache.org/jira/browse/MAPREDUCE-6767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated MAPREDUCE-6767: Fix Version/s: 3.0.0-alpha1 > TestSlive fails after a common change > - > > Key: MAPREDUCE-6767 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6767 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Kihwal Lee >Assignee: Daniel Templeton > Fix For: 3.0.0-alpha1 > > Attachments: MAPREDUCE-6767.001.patch > > > It looks like this was broken after HADOOP-12726. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6914) Tests use assertTrue(....equals(...)) instead of assertEquals()
[ https://issues.apache.org/jira/browse/MAPREDUCE-6914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated MAPREDUCE-6914: Attachment: MAPREDUCE-6914.001.patch > Tests use assertTrue(equals(...)) instead of assertEquals() > --- > > Key: MAPREDUCE-6914 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6914 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: test >Affects Versions: 2.8.1, 3.0.0-alpha4 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Minor > Attachments: MAPREDUCE-6914.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6914) Tests use assertTrue(....equals(...)) instead of assertEquals()
[ https://issues.apache.org/jira/browse/MAPREDUCE-6914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated MAPREDUCE-6914: Status: Patch Available (was: Open) > Tests use assertTrue(equals(...)) instead of assertEquals() > --- > > Key: MAPREDUCE-6914 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6914 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: test >Affects Versions: 3.0.0-alpha4, 2.8.1 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Minor > Attachments: MAPREDUCE-6914.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6914) Tests use assertTrue(....equals(...)) instead of assertEquals()
Daniel Templeton created MAPREDUCE-6914: --- Summary: Tests use assertTrue(equals(...)) instead of assertEquals() Key: MAPREDUCE-6914 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6914 Project: Hadoop Map/Reduce Issue Type: Improvement Components: test Affects Versions: 3.0.0-alpha4, 2.8.1 Reporter: Daniel Templeton Assignee: Daniel Templeton Priority: Minor -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6896) Document wrong spelling in usage of MapredTestDriver tools.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated MAPREDUCE-6896:
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha4
Status: Resolved (was: Patch Available)
Thanks for the patch, [~GeLiXin]. Committed to trunk.
> Document wrong spelling in usage of MapredTestDriver tools.
> ---
>
> Key: MAPREDUCE-6896
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6896
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: documentation
>Affects Versions: 3.0.0-alpha3
>Reporter: LiXin Ge
>Assignee: LiXin Ge
> Labels: easyfix
> Fix For: 3.0.0-alpha4
>
> Attachments: MAPREDUCE-6896.001.patch
>
>
> When the user run performance test of MapredTestDriver, a spelling mistake
> seems exposed to user:
> {quote}
> ./hadoop org.apache.hadoop.test.MapredTestDriver
> An example program must be given as the first argument.
> Valid program names are:
> ..
> timelineperformance: A job that launches mappers to test *timline* service
> performance.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6896) Document wrong spelling in usage of MapredTestDriver tools.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16041236#comment-16041236
]
Daniel Templeton commented on MAPREDUCE-6896:
-
LGTM +1
> Document wrong spelling in usage of MapredTestDriver tools.
> ---
>
> Key: MAPREDUCE-6896
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6896
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: documentation
>Affects Versions: 3.0.0-alpha3
>Reporter: LiXin Ge
>Assignee: LiXin Ge
> Labels: easyfix
> Attachments: MAPREDUCE-6896.001.patch
>
>
> When the user run performance test of MapredTestDriver, a spelling mistake
> seems exposed to user:
> {quote}
> ./hadoop org.apache.hadoop.test.MapredTestDriver
> An example program must be given as the first argument.
> Valid program names are:
> ..
> timelineperformance: A job that launches mappers to test *timline* service
> performance.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Assigned] (MAPREDUCE-6896) Document wrong spelling in usage of MapredTestDriver tools.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton reassigned MAPREDUCE-6896:
---
Assignee: LiXin Ge
> Document wrong spelling in usage of MapredTestDriver tools.
> ---
>
> Key: MAPREDUCE-6896
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6896
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: documentation
>Affects Versions: 3.0.0-alpha3
>Reporter: LiXin Ge
>Assignee: LiXin Ge
> Labels: easyfix
> Attachments: MAPREDUCE-6896.001.patch
>
>
> When the user run performance test of MapredTestDriver, a spelling mistake
> seems exposed to user:
> {quote}
> ./hadoop org.apache.hadoop.test.MapredTestDriver
> An example program must be given as the first argument.
> Valid program names are:
> ..
> timelineperformance: A job that launches mappers to test *timline* service
> performance.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6883) AuditLogger and TestAuditLogger are dead code
Daniel Templeton created MAPREDUCE-6883:
---
Summary: AuditLogger and TestAuditLogger are dead code
Key: MAPREDUCE-6883
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6883
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: client
Affects Versions: 2.8.0
Reporter: Daniel Templeton
Priority: Minor
The {{AuditLogger}} and {{TestAuditLogger}} classes appear to be dead code. I
can't find anything that uses or references {{AuditLogger}}. No one has
touched the code 2011. I think it's safe to remove.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6846) Fragments specified for libjar paths are not handled correctly
[
https://issues.apache.org/jira/browse/MAPREDUCE-6846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated MAPREDUCE-6846:
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha3
2.9.0
Status: Resolved (was: Patch Available)
Thanks for the patch, [~ctrezzo], and thanks for the reviews, [~sjlee0] and
[~jlowe]! Committed to trunk and branch-2.
> Fragments specified for libjar paths are not handled correctly
> --
>
> Key: MAPREDUCE-6846
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6846
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.6.0, 2.7.3, 3.0.0-alpha2
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>Priority: Minor
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: MAPREDUCE-6846-trunk.001.patch,
> MAPREDUCE-6846-trunk.002.patch, MAPREDUCE-6846-trunk.003.patch,
> MAPREDUCE-6846-trunk.004.patch, MAPREDUCE-6846-trunk.005.patch,
> MAPREDUCE-6846-trunk.006.patch
>
>
> If a user specifies a fragment for a libjars path via generic options parser,
> the client crashes with a FileNotFoundException:
> {noformat}
> java.io.FileNotFoundException: File file:/home/mapred/test.txt#testFrag.txt
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:638)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:864)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:628)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:442)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:363)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:314)
> at
> org.apache.hadoop.mapreduce.JobResourceUploader.copyRemoteFiles(JobResourceUploader.java:387)
> at
> org.apache.hadoop.mapreduce.JobResourceUploader.uploadLibJars(JobResourceUploader.java:154)
> at
> org.apache.hadoop.mapreduce.JobResourceUploader.uploadResources(JobResourceUploader.java:105)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:102)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:197)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1344)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1892)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1341)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1362)
> at
> org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:306)
> at
> org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:359)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at
> org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:367)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
> at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
> at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
> {noformat}
> This is actually inconsistent with the behavior for files and archives. Here
> is a table showing the current behavior for each type of path and resource:
> | || Qualified path (i.e. file://home/mapred/test.txt#frag.txt) || Absolute
> path (i.e. /home/mapred/test.txt#frag.txt) || Relative path (i.e.
> test.txt#frag.txt) ||
> || -libjars | FileNotFound | FileNotFound|FileNotFound|
> || -files | (/) | (/) | (/) |
> || -archives | (/) | (/) | (/) |
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
[jira] [Commented] (MAPREDUCE-6846) Fragments specified for libjar paths are not handled correctly
[
https://issues.apache.org/jira/browse/MAPREDUCE-6846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15951374#comment-15951374
]
Daniel Templeton commented on MAPREDUCE-6846:
-
Sorry, I have a couple more nits:
# In {{uploadLibJars()}} you should use the diamond operator here:{code}
Collection libjarURIs = new LinkedList();{code}
# In {{uploadLibJars()}}, you have: {code}try {
tmpURI = new URI(tmpjars);
} catch (URISyntaxException e) {
throw new IllegalArgumentException(e);
}{code} Can we make the exception message as clear as the others?
# Call me old fashioned, but I'd rather you didn't change this line in
{{uploadLibJars()}} to break on the dot instead of the space: {code}
Path libJarsDirWildcard = jtFs
.makeQualified(new Path(libjarsDir,
DistributedCache.WILDCARD));{code}
Thanks!
> Fragments specified for libjar paths are not handled correctly
> --
>
> Key: MAPREDUCE-6846
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6846
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.6.0, 2.7.3, 3.0.0-alpha2
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>Priority: Minor
> Attachments: MAPREDUCE-6846-trunk.001.patch,
> MAPREDUCE-6846-trunk.002.patch, MAPREDUCE-6846-trunk.003.patch,
> MAPREDUCE-6846-trunk.004.patch, MAPREDUCE-6846-trunk.005.patch
>
>
> If a user specifies a fragment for a libjars path via generic options parser,
> the client crashes with a FileNotFoundException:
> {noformat}
> java.io.FileNotFoundException: File file:/home/mapred/test.txt#testFrag.txt
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:638)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:864)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:628)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:442)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:363)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:314)
> at
> org.apache.hadoop.mapreduce.JobResourceUploader.copyRemoteFiles(JobResourceUploader.java:387)
> at
> org.apache.hadoop.mapreduce.JobResourceUploader.uploadLibJars(JobResourceUploader.java:154)
> at
> org.apache.hadoop.mapreduce.JobResourceUploader.uploadResources(JobResourceUploader.java:105)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:102)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:197)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1344)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1892)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1341)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1362)
> at
> org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:306)
> at
> org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:359)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at
> org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:367)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
> at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
> at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
> {noformat}
> This is actually inconsistent with the behavior for files and archives. Here
> is a table showing the current behavior for each type of path and resource:
> | || Qualified path (i.e.
[jira] [Assigned] (MAPREDUCE-6848) MRApps.setMRFrameworkClasspath() unnecessarily declares that it throws IOException
[ https://issues.apache.org/jira/browse/MAPREDUCE-6848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton reassigned MAPREDUCE-6848: --- Assignee: Wendy Turner (was: Daniel Templeton) > MRApps.setMRFrameworkClasspath() unnecessarily declares that it throws > IOException > -- > > Key: MAPREDUCE-6848 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6848 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 2.8.0 >Reporter: Daniel Templeton >Assignee: Wendy Turner >Priority: Trivial > Labels: newbie > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-6846) Fragments specified for libjar paths are not handled correctly
[
https://issues.apache.org/jira/browse/MAPREDUCE-6846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941283#comment-15941283
]
Daniel Templeton edited comment on MAPREDUCE-6846 at 3/24/17 10:41 PM:
---
What I meant was: {code}
Path newPath =
copyRemoteFiles(libjarsDir, tmp, conf, submitReplication);
try {
DistributedCache.addFileToClassPath(newPath, conf, jtFs, false);
if (!foundFragment) {
foundFragment = tmpURI.getFragment() != null;
}
libjarURIs.add(getPathURI(newPath, tmpURI.getFragment()));
}
{code} but after taking a second look, I'm fine with it as is.
was (Author: templedf):
What I meant was: {code}
Path newPath =
copyRemoteFiles(libjarsDir, tmp, conf, submitReplication);
try {
DistributedCache.addFileToClassPath(newPath), conf, jtFs, false);
if (!foundFragment) {
foundFragment = tmpURI.getFragment() != null;
}
libjarURIs.add(getPathURI(newPath, tmpURI.getFragment()));
}
{code} but after taking a second look, I'm fine with it as is.
> Fragments specified for libjar paths are not handled correctly
> --
>
> Key: MAPREDUCE-6846
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6846
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.6.0, 2.7.3, 3.0.0-alpha2
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>Priority: Minor
> Attachments: MAPREDUCE-6846-trunk.001.patch,
> MAPREDUCE-6846-trunk.002.patch
>
>
> If a user specifies a fragment for a libjars path via generic options parser,
> the client crashes with a FileNotFoundException:
> {noformat}
> java.io.FileNotFoundException: File file:/home/mapred/test.txt#testFrag.txt
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:638)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:864)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:628)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:442)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:363)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:314)
> at
> org.apache.hadoop.mapreduce.JobResourceUploader.copyRemoteFiles(JobResourceUploader.java:387)
> at
> org.apache.hadoop.mapreduce.JobResourceUploader.uploadLibJars(JobResourceUploader.java:154)
> at
> org.apache.hadoop.mapreduce.JobResourceUploader.uploadResources(JobResourceUploader.java:105)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:102)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:197)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1344)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1892)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1341)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1362)
> at
> org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:306)
> at
> org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:359)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at
> org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:367)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
> at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
> at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
> {noformat}
> This is actually
[jira] [Commented] (MAPREDUCE-6846) Fragments specified for libjar paths are not handled correctly
[
https://issues.apache.org/jira/browse/MAPREDUCE-6846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941283#comment-15941283
]
Daniel Templeton commented on MAPREDUCE-6846:
-
What I meant was: {code}
Path newPath =
copyRemoteFiles(libjarsDir, tmp, conf, submitReplication);
try {
DistributedCache.addFileToClassPath(newPath), conf, jtFs, false);
if (!foundFragment) {
foundFragment = tmpURI.getFragment() != null;
}
libjarURIs.add(getPathURI(newPath, tmpURI.getFragment()));
}
{code} but after taking a second look, I'm fine with it as is.
> Fragments specified for libjar paths are not handled correctly
> --
>
> Key: MAPREDUCE-6846
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6846
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.6.0, 2.7.3, 3.0.0-alpha2
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>Priority: Minor
> Attachments: MAPREDUCE-6846-trunk.001.patch,
> MAPREDUCE-6846-trunk.002.patch
>
>
> If a user specifies a fragment for a libjars path via generic options parser,
> the client crashes with a FileNotFoundException:
> {noformat}
> java.io.FileNotFoundException: File file:/home/mapred/test.txt#testFrag.txt
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:638)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:864)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:628)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:442)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:363)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:314)
> at
> org.apache.hadoop.mapreduce.JobResourceUploader.copyRemoteFiles(JobResourceUploader.java:387)
> at
> org.apache.hadoop.mapreduce.JobResourceUploader.uploadLibJars(JobResourceUploader.java:154)
> at
> org.apache.hadoop.mapreduce.JobResourceUploader.uploadResources(JobResourceUploader.java:105)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:102)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:197)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1344)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1892)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1341)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1362)
> at
> org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:306)
> at
> org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:359)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at
> org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:367)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
> at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
> at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
> {noformat}
> This is actually inconsistent with the behavior for files and archives. Here
> is a table showing the current behavior for each type of path and resource:
> | || Qualified path (i.e. file://home/mapred/test.txt#frag.txt) || Absolute
> path (i.e. /home/mapred/test.txt#frag.txt) || Relative path (i.e.
> test.txt#frag.txt) ||
> || -libjars | FileNotFound | FileNotFound|FileNotFound|
> || -files | (/) | (/) | (/) |
> || -archives | (/) | (/) | (/) |
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (MAPREDUCE-6848) MRApps.setMRFrameworkClasspath() unnecessarily declares that it throws IOException
[ https://issues.apache.org/jira/browse/MAPREDUCE-6848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton reassigned MAPREDUCE-6848: --- Assignee: Daniel Templeton > MRApps.setMRFrameworkClasspath() unnecessarily declares that it throws > IOException > -- > > Key: MAPREDUCE-6848 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6848 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 2.8.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Trivial > Labels: newbie > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6846) Fragments specified for libjar paths are not handled correctly
[
https://issues.apache.org/jira/browse/MAPREDUCE-6846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15933034#comment-15933034
]
Daniel Templeton commented on MAPREDUCE-6846:
-
Thanks, [~ctrezzo]. Here's my comments:
# I found this code to be a little confusing: {code} URI pathURI =
getPathURI(newPath, tmpURI.getFragment());
if (!foundFragment) {
foundFragment = pathURI.getFragment() != null;
}
DistributedCache.addFileToClassPath(new Path(pathURI.getPath()), conf,
jtFs, false);
libjarURIs.add(pathURI);{code} It seems odd to create {{pathURI}}
and then do nothing with it that you couldn't do with {{tmpURI}} until the end.
For me it would be clearer as:{code}
DistributedCache.addFileToClassPath(new Path(tmpURI.getPath()), conf,
jtFs, false);
if (!foundFragment) {
foundFragment = tmpURI.getFragment() != null;
}
libjarURIs.add(getPathURI(newPath, tmpURI.getFragment()));{code}
# I really hate the practice of catching meaningful exceptions and rethrowing
them wrapped in something generic, but in the case of the
{{URISyntaxException}} that shoudn't happen, I don't see where the API gives
you much choice. Since it is an unexpected exception that we're going to
deliver ultimately to the application client, it would be nice to be really
explicit about what went wrong. You're bundling the original exception, but
that doesn't always make it all the way through to the end user.
# Looks like you're not qualifying the URIs before adding them, as was done
before: {code}for (URI uri : libjarURIs) {
DistributedCache.addCacheFile(uri, conf);
}{code}
# If you rearrange {{uploadLibJars()}} like I suggested above, I don't think
you need the change to {{getPathURI()}}. Or am I missing something?
# From the perspective of the person trying to understand the unit test failure
report, I don't think replacing the {{fail()}} calls with {{assertTrue()}} and
{{assertFalse()}} makes anything clearer. The assert message is (correctly) a
plain statement of misconfiguration, which doesn't make sense with an
{{assertTrue()}}/{{assertFalse()}}.
# Why are you splitting the {{ResourceLimitsConf}} into two classes?
Inheritance among inner classes always strikes me as a strange thing to do.
# Given that the {{ResourceLimitsConf.Builder}} is an inner class of
{{ResourceLimitsConf}}, I don't think that renaming it to
{{ResourceLimitsConfBuilder}} is particularly needful, and it makes the code a
bit more cluttered. I assume it's because you now have two classes and two
builders, and you want to separate them, but see the previous point.
# Tiny nit, but I'd rather you spell out distributed cache than use DC.
# Might be nice to add a comment to the stub's {{mkdirs()}} to say why it's OK
that it doesn't actually do anything. You also might want to add a comment to
the {{JobResourceUploader.mkdirs()}} method to explain that it's there so that
it can be overridden.
> Fragments specified for libjar paths are not handled correctly
> --
>
> Key: MAPREDUCE-6846
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6846
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.6.0, 2.7.3, 3.0.0-alpha2
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>Priority: Minor
> Attachments: MAPREDUCE-6846-trunk.001.patch,
> MAPREDUCE-6846-trunk.002.patch
>
>
> If a user specifies a fragment for a libjars path via generic options parser,
> the client crashes with a FileNotFoundException:
> {noformat}
> java.io.FileNotFoundException: File file:/home/mapred/test.txt#testFrag.txt
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:638)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:864)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:628)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:442)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:363)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:314)
> at
> org.apache.hadoop.mapreduce.JobResourceUploader.copyRemoteFiles(JobResourceUploader.java:387)
> at
> org.apache.hadoop.mapreduce.JobResourceUploader.uploadLibJars(JobResourceUploader.java:154)
> at
> org.apache.hadoop.mapreduce.JobResourceUploader.uploadResources(JobResourceUploader.java:105)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:102)
> at
>
[jira] [Created] (MAPREDUCE-6864) Hadoop streaming creates 2 mappers when the input has only one block
Daniel Templeton created MAPREDUCE-6864: --- Summary: Hadoop streaming creates 2 mappers when the input has only one block Key: MAPREDUCE-6864 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6864 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.7.3 Reporter: Daniel Templeton If a streaming job is run against input that is less than 2 blocks, 2 mappers will be created, both operating on the same split, both producing (duplicate) output. In some cases the second mapper will consistently fail. I've not seen the failure with input less than 10 bytes or more than a couple MB. I have seen it with a 4kB input. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6825) YARNRunner#createApplicationSubmissionContext method is longer than 150 lines
[
https://issues.apache.org/jira/browse/MAPREDUCE-6825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated MAPREDUCE-6825:
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha3
2.9.0
Status: Resolved (was: Patch Available)
Thanks for the patch, [~GergelyNovak]. Committed to trunk and branch-2.
> YARNRunner#createApplicationSubmissionContext method is longer than 150 lines
> -
>
> Key: MAPREDUCE-6825
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6825
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Chris Trezzo
>Assignee: Gergely Novák
>Priority: Trivial
> Labels: newbie
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: MAPREDUCE-6825.001.patch, MAPREDUCE-6825.002.patch,
> MAPREDUCE-6825.003.patch, MAPREDUCE-6825.004.patch, MAPREDUCE-6825.005.patch,
> MAPREDUCE-6825.006.patch
>
>
> bq.
> ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java:341:
> public ApplicationSubmissionContext createApplicationSubmissionContext(:3:
> Method length is 249 lines (max allowed is 150).
> {{YARNRunner#createApplicationSubmissionContext}} is longer than 150 lines
> and needs to be refactored.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6825) YARNRunner#createApplicationSubmissionContext method is longer than 150 lines
[
https://issues.apache.org/jira/browse/MAPREDUCE-6825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876729#comment-15876729
]
Daniel Templeton commented on MAPREDUCE-6825:
-
Looks good. I want to take one more pass through the whole patch before I give
my +1. I'll try to do that this week and get it in.
> YARNRunner#createApplicationSubmissionContext method is longer than 150 lines
> -
>
> Key: MAPREDUCE-6825
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6825
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Chris Trezzo
>Assignee: Gergely Novák
>Priority: Trivial
> Labels: newbie
> Attachments: MAPREDUCE-6825.001.patch, MAPREDUCE-6825.002.patch,
> MAPREDUCE-6825.003.patch, MAPREDUCE-6825.004.patch, MAPREDUCE-6825.005.patch,
> MAPREDUCE-6825.006.patch
>
>
> bq.
> ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java:341:
> public ApplicationSubmissionContext createApplicationSubmissionContext(:3:
> Method length is 249 lines (max allowed is 150).
> {{YARNRunner#createApplicationSubmissionContext}} is longer than 150 lines
> and needs to be refactored.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6825) YARNRunner#createApplicationSubmissionContext method is longer than 150 lines
[
https://issues.apache.org/jira/browse/MAPREDUCE-6825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated MAPREDUCE-6825:
Sounds good to me.
> YARNRunner#createApplicationSubmissionContext method is longer than 150 lines
> -
>
> Key: MAPREDUCE-6825
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6825
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Chris Trezzo
>Assignee: Gergely Novák
>Priority: Trivial
> Labels: newbie
> Attachments: MAPREDUCE-6825.001.patch, MAPREDUCE-6825.002.patch,
> MAPREDUCE-6825.003.patch, MAPREDUCE-6825.004.patch, MAPREDUCE-6825.005.patch
>
>
> bq.
> ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java:341:
> public ApplicationSubmissionContext createApplicationSubmissionContext(:3:
> Method length is 249 lines (max allowed is 150).
> {{YARNRunner#createApplicationSubmissionContext}} is longer than 150 lines
> and needs to be refactored.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6825) YARNRunner#createApplicationSubmissionContext method is longer than 150 lines
[
https://issues.apache.org/jira/browse/MAPREDUCE-6825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15868864#comment-15868864
]
Daniel Templeton commented on MAPREDUCE-6825:
-
Thanks for the updated patch, [~GergelyNovak]. Looks good. My only quibble is
that the summary for the {{@throws IOException}} isn't quite accurate. The
exception happens when 1) any of the many paths cannot be resolved, 2) the
tokens cannot be written, or 3) there's a path conflict in the distributed
cache artifacts. Sadly, no succinct way to summarize that.
> YARNRunner#createApplicationSubmissionContext method is longer than 150 lines
> -
>
> Key: MAPREDUCE-6825
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6825
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Chris Trezzo
>Assignee: Gergely Novák
>Priority: Trivial
> Labels: newbie
> Attachments: MAPREDUCE-6825.001.patch, MAPREDUCE-6825.002.patch,
> MAPREDUCE-6825.003.patch, MAPREDUCE-6825.004.patch, MAPREDUCE-6825.005.patch
>
>
> bq.
> ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java:341:
> public ApplicationSubmissionContext createApplicationSubmissionContext(:3:
> Method length is 249 lines (max allowed is 150).
> {{YARNRunner#createApplicationSubmissionContext}} is longer than 150 lines
> and needs to be refactored.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6848) MRApps.setMRFrameworkClasspath() unnecessarily declares that it throws IOException
Daniel Templeton created MAPREDUCE-6848: --- Summary: MRApps.setMRFrameworkClasspath() unnecessarily declares that it throws IOException Key: MAPREDUCE-6848 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6848 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 2.8.0 Reporter: Daniel Templeton Priority: Trivial -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6441) Improve temporary directory name generation in LocalDistributedCacheManager for concurrent processes
[
https://issues.apache.org/jira/browse/MAPREDUCE-6441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15857142#comment-15857142
]
Daniel Templeton commented on MAPREDUCE-6441:
-
Looks generally good to me. My only concern is that it would be cleaner to use
an executor in the test instead of handling all the threads directly.
> Improve temporary directory name generation in LocalDistributedCacheManager
> for concurrent processes
>
>
> Key: MAPREDUCE-6441
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6441
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: William Watson
>Assignee: Ray Chiang
> Attachments: HADOOP-10924.02.patch,
> HADOOP-10924.03.jobid-plus-uuid.patch, MAPREDUCE-6441.004.patch,
> MAPREDUCE-6441.005.patch
>
>
> Kicking off many sqoop processes in different threads results in:
> {code}
> 2014-08-01 13:47:24 -0400: INFO - 14/08/01 13:47:22 ERROR tool.ImportTool:
> Encountered IOException running import job: java.io.IOException:
> java.util.concurrent.ExecutionException: java.io.IOException: Rename cannot
> overwrite non empty destination directory
> /tmp/hadoop-hadoop/mapred/local/1406915233073
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.hadoop.mapred.LocalDistributedCacheManager.setup(LocalDistributedCacheManager.java:149)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.hadoop.mapred.LocalJobRunner$Job.(LocalJobRunner.java:163)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:731)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:432)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
> 2014-08-01 13:47:24 -0400: INFO -at
> java.security.AccessController.doPrivileged(Native Method)
> 2014-08-01 13:47:24 -0400: INFO -at
> javax.security.auth.Subject.doAs(Subject.java:415)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.sqoop.mapreduce.ImportJobBase.doSubmitJob(ImportJobBase.java:186)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:159)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:239)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.sqoop.manager.SqlManager.importQuery(SqlManager.java:645)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:415)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.sqoop.tool.ImportTool.run(ImportTool.java:502)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.sqoop.Sqoop.run(Sqoop.java:145)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.sqoop.Sqoop.main(Sqoop.java:238)
> {code}
> If two are kicked off in the same second. The issue is the following lines of
> code in the org.apache.hadoop.mapred.LocalDistributedCacheManager class:
> {code}
> // Generating unique numbers for FSDownload.
> AtomicLong uniqueNumberGenerator =
>new AtomicLong(System.currentTimeMillis());
> {code}
> and
> {code}
> Long.toString(uniqueNumberGenerator.incrementAndGet())),
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6842) Update the links in PiEstimator document
[ https://issues.apache.org/jira/browse/MAPREDUCE-6842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated MAPREDUCE-6842: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0-alpha3 2.9.0 Status: Resolved (was: Patch Available) Committed to branch-2 and trunk. Thanks, [~Jungyoo]! > Update the links in PiEstimator document > > > Key: MAPREDUCE-6842 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6842 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: documentation >Reporter: Akira Ajisaka >Assignee: Jung Yoo >Priority: Minor > Labels: newbie > Fix For: 2.9.0, 3.0.0-alpha3 > > Attachments: MAPREDUCE-6842.001.patch > > > In the package-info.html, There are links to > * > http://developer.yahoo.net/blogs/hadoop/2009/05/hadoop_sorts_a_petabyte_in_162.html > * > http://developer.yahoo.net/blogs/hadoop/2009/05/hadoop_computes_the_10151st_bi.html > The pages were moved to > * > http://yahoohadoop.tumblr.com/post/98338791001/hadoop-sorts-a-petabyte-in-1625-hours-and-a > * > http://yahoohadoop.tumblr.com/post/98338598026/hadoop-computes-the-10-15-1st-bit-of-%CF%80 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6825) YARNRunner#createApplicationSubmissionContext method is longer than 150 lines
[
https://issues.apache.org/jira/browse/MAPREDUCE-6825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854425#comment-15854425
]
Daniel Templeton commented on MAPREDUCE-6825:
-
Thanks, [~GergelyNovak]. Some comments:
{code}warnForJavaLibPath(conf.get(MRJobConfig.MAPRED_MAP_ADMIN_JAVA_OPTS,
""),
"map",
MRJobConfig.MAPRED_MAP_ADMIN_JAVA_OPTS,
MRJobConfig.MAPRED_ADMIN_USER_ENV);
warnForJavaLibPath(conf.get(MRJobConfig.REDUCE_JAVA_OPTS, ""), "reduce",
MRJobConfig.REDUCE_JAVA_OPTS, MRJobConfig.REDUCE_ENV);
warnForJavaLibPath(conf.get(MRJobConfig.MAPRED_REDUCE_ADMIN_JAVA_OPTS, ""),
"reduce",
MRJobConfig.MAPRED_REDUCE_ADMIN_JAVA_OPTS,
MRJobConfig.MAPRED_ADMIN_USER_ENV);
{code}
The first and third call have the same style regarding line breaks, but the
second is different. May as well make them all consistent.
{code}
Map localResources =
setupLocalResources(jobConf, jobSubmitDir);
// Setup security tokens
DataOutputBuffer dob = new DataOutputBuffer();
ts.writeTokenStorageToStream(dob);
ByteBuffer securityTokens = ByteBuffer.wrap(
dob.getData(), 0, dob.getLength());
{code}
Similarly, {{localResources}} and {{securityTokens}} are split in different
ways. I prefer to break the line on the equals, like {{localResources}}.
While you're messing with stuff, you may as well switch all the collections to
use the diamond operator.
And, while it's not your fault, the javadoc is missing for
{{createApplicationSubmissionContext()}}. Could you please add it?
> YARNRunner#createApplicationSubmissionContext method is longer than 150 lines
> -
>
> Key: MAPREDUCE-6825
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6825
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Chris Trezzo
>Assignee: Gergely Novák
>Priority: Trivial
> Labels: newbie
> Attachments: MAPREDUCE-6825.001.patch, MAPREDUCE-6825.002.patch,
> MAPREDUCE-6825.003.patch
>
>
> bq.
> ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java:341:
> public ApplicationSubmissionContext createApplicationSubmissionContext(:3:
> Method length is 249 lines (max allowed is 150).
> {{YARNRunner#createApplicationSubmissionContext}} is longer than 150 lines
> and needs to be refactored.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6842) Update the links in PiEstimator document
[ https://issues.apache.org/jira/browse/MAPREDUCE-6842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852427#comment-15852427 ] Daniel Templeton commented on MAPREDUCE-6842: - Jenkins in happy (enough). I'll commit it soon. > Update the links in PiEstimator document > > > Key: MAPREDUCE-6842 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6842 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: documentation >Reporter: Akira Ajisaka >Assignee: Jung Yoo >Priority: Minor > Labels: newbie > Attachments: MAPREDUCE-6842.001.patch > > > In the package-info.html, There are links to > * > http://developer.yahoo.net/blogs/hadoop/2009/05/hadoop_sorts_a_petabyte_in_162.html > * > http://developer.yahoo.net/blogs/hadoop/2009/05/hadoop_computes_the_10151st_bi.html > The pages were moved to > * > http://yahoohadoop.tumblr.com/post/98338791001/hadoop-sorts-a-petabyte-in-1625-hours-and-a > * > http://yahoohadoop.tumblr.com/post/98338598026/hadoop-computes-the-10-15-1st-bit-of-%CF%80 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6842) Update the links in PiEstimator document
[ https://issues.apache.org/jira/browse/MAPREDUCE-6842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852160#comment-15852160 ] Daniel Templeton commented on MAPREDUCE-6842: - LGTM +1 pending Jenkins not surprising us. > Update the links in PiEstimator document > > > Key: MAPREDUCE-6842 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6842 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: documentation >Reporter: Akira Ajisaka >Assignee: Jung Yoo >Priority: Minor > Labels: newbie > Attachments: MAPREDUCE-6842.001.patch > > > In the package-info.html, There are links to > * > http://developer.yahoo.net/blogs/hadoop/2009/05/hadoop_sorts_a_petabyte_in_162.html > * > http://developer.yahoo.net/blogs/hadoop/2009/05/hadoop_computes_the_10151st_bi.html > The pages were moved to > * > http://yahoohadoop.tumblr.com/post/98338791001/hadoop-sorts-a-petabyte-in-1625-hours-and-a > * > http://yahoohadoop.tumblr.com/post/98338598026/hadoop-computes-the-10-15-1st-bit-of-%CF%80 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Assigned] (MAPREDUCE-6841) Fix dead link in MapReduce tutorial document
[
https://issues.apache.org/jira/browse/MAPREDUCE-6841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton reassigned MAPREDUCE-6841:
---
Assignee: Victor Nee
> Fix dead link in MapReduce tutorial document
>
>
> Key: MAPREDUCE-6841
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6841
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: documentation
>Reporter: Akira Ajisaka
>Assignee: Victor Nee
>Priority: Minor
> Labels: newbie
>
> {noformat:title=MapReduceTutorial.md}
> ([FileOutputFormat.setOutputPath(Path)](../../api/org/apache/hadoop/mapreduce/lib/input/FileOutputFormat.html)).
> {noformat}
> input should be output.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Assigned] (MAPREDUCE-6842) Update the links in PiEstimator document
[ https://issues.apache.org/jira/browse/MAPREDUCE-6842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton reassigned MAPREDUCE-6842: --- Assignee: Jung Yoo > Update the links in PiEstimator document > > > Key: MAPREDUCE-6842 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6842 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: documentation >Reporter: Akira Ajisaka >Assignee: Jung Yoo >Priority: Minor > Labels: newbie > > In the package-info.html, There are links to > * > http://developer.yahoo.net/blogs/hadoop/2009/05/hadoop_sorts_a_petabyte_in_162.html > * > http://developer.yahoo.net/blogs/hadoop/2009/05/hadoop_computes_the_10151st_bi.html > The pages were moved to > * > http://yahoohadoop.tumblr.com/post/98338791001/hadoop-sorts-a-petabyte-in-1625-hours-and-a > * > http://yahoohadoop.tumblr.com/post/98338598026/hadoop-computes-the-10-15-1st-bit-of-%CF%80 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Assigned] (MAPREDUCE-6824) TaskAttemptImpl#createCommonContainerLaunchContext is longer than 150 lines
[
https://issues.apache.org/jira/browse/MAPREDUCE-6824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton reassigned MAPREDUCE-6824:
---
Assignee: Udai Kiran Potluri
> TaskAttemptImpl#createCommonContainerLaunchContext is longer than 150 lines
> ---
>
> Key: MAPREDUCE-6824
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6824
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Chris Trezzo
>Assignee: Udai Kiran Potluri
>Priority: Trivial
> Labels: newbie
>
> bq.
> ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java:752:
> private static ContainerLaunchContext createCommonContainerLaunchContext(:3:
> Method length is 172 lines (max allowed is 150).
> {{TaskAttemptImpl#createCommonContainerLaunchContext}} is longer than 150
> lines and needs to be refactored.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6837) Add an equivalent to Crunch's Pair class
[
https://issues.apache.org/jira/browse/MAPREDUCE-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated MAPREDUCE-6837:
Description:
Crunch has this great {{Pair}} class
(https://crunch.apache.org/apidocs/0.14.0/org/apache/crunch/Pair.html) that
saves one from constantly implementing composite writables. It seems silly
that we still don't have an equivalent in MR.
I would like to see a new class with the following API:
{code}
package org.apache.hadoop.io;
public class CompositeWritable implements WritableComparable {
public CompositeWritable(P primary, S secondary);
public P getPrimary();
public void setPrimary(P primary);
public S getSecondary();
public void setSecondary(S secondary);
// Return true if both primaries and both secondaries are equal
public boolean equals(CompositeWritable o);
// Return the primary's hash code
public long hashCode();
// Sort first by primary and then by secondary
public int compareTo(CompositeWritable o);
public void readFields(DataInput in);
public void write(DataOutput out);
}
{code}
With such a class, implementing a secondary sort would mean just implementing a
custom grouping comparator. That comparator could also be implemented as part
of this JIRA:
{code}
package org.apache.hadoop.io;
public class CompositeGroupingComparator extends WritableComparator {
...
}
{code}
Or some such.
Crunch also provides {{Tuple3}}, {{Tuple4}}, and {{TupleN}} classes, but I
don't think we need to add equivalents. If someone really wants that
capability, they can nest composite keys.
Don't forget to add unit tests!
was:
Crunch has this great {{Pair}} class
(https://crunch.apache.org/apidocs/0.14.0/org/apache/crunch/Pair.html) that
save you from constantly implementing composite writables. It seems silly that
we still don't have an equivalent in MR.
I would like to see a new class with the following API:
{code}
package org.apache.hadoop.io;
public class CompositeWritable implements WritableComparable {
public CompositeWritable(P primary, S secondary);
public P getPrimary();
public void setPrimary(P primary);
public S getSecondary();
public void setSecondary(S secondary);
// Return true if both primaries and both secondaries are equal
public boolean equals(CompositeWritable o);
// Return the primary's hash code
public long hashCode();
// Sort first by primary and then by secondary
public int compareTo(CompositeWritable o);
public void readFields(DataInput in);
public void write(DataOutput out);
}
{code}
With such a class, implementing a secondary sort would mean just implementing a
custom grouping comparator. That comparator could be implemented as part of
this JIRA:
{code}
package org.apache.hadoop.io;
public class CompositeGroupingComparator extends WritableComparator {
...
}
{code}
Or some such.
Crunch also provides {{Tuple3}}, {{Tuple4}}, and {{TupleN}} classes, but I
don't think we need to add equivalents. If someone really wants that
capability, they can nest composite keys.
Don't forget to add unit tests!
> Add an equivalent to Crunch's Pair class
>
>
> Key: MAPREDUCE-6837
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6837
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
>Reporter: Daniel Templeton
> Labels: newbie++
>
> Crunch has this great {{Pair}} class
> (https://crunch.apache.org/apidocs/0.14.0/org/apache/crunch/Pair.html) that
> saves one from constantly implementing composite writables. It seems silly
> that we still don't have an equivalent in MR.
> I would like to see a new class with the following API:
> {code}
> package org.apache.hadoop.io;
> public class CompositeWritable WritableComparable> implements WritableComparable {
> public CompositeWritable(P primary, S secondary);
> public P getPrimary();
> public void setPrimary(P primary);
> public S getSecondary();
> public void setSecondary(S secondary);
> // Return true if both primaries and both secondaries are equal
> public boolean equals(CompositeWritable o);
> // Return the primary's hash code
> public long hashCode();
> // Sort first by primary and then by secondary
> public int compareTo(CompositeWritable o);
> public void readFields(DataInput in);
> public void write(DataOutput out);
> }
> {code}
> With such a class, implementing a secondary sort would mean just implementing
> a custom grouping comparator. That comparator could also be implemented as
> part of this JIRA:
> {code}
> package org.apache.hadoop.io;
> public class CompositeGroupingComparator extends WritableComparator {
> ...
> }
> {code}
> Or some such.
> Crunch also provides {{Tuple3}}, {{Tuple4}}, and {{TupleN}} classes, but I
> don't think we need to
[jira] [Created] (MAPREDUCE-6837) Add an equivalent to Crunch's Pair class
Daniel Templeton created MAPREDUCE-6837:
---
Summary: Add an equivalent to Crunch's Pair class
Key: MAPREDUCE-6837
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6837
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2
Reporter: Daniel Templeton
Crunch has this great {{Pair}} class
(https://crunch.apache.org/apidocs/0.14.0/org/apache/crunch/Pair.html) that
save you from constantly implementing composite writables. It seems silly that
we still don't have an equivalent in MR.
I would like to see a new class with the following API:
{code}
package org.apache.hadoop.io;
public class CompositeWritable implements WritableComparable {
public CompositeWritable(P primary, S secondary);
public P getPrimary();
public void setPrimary(P primary);
public S getSecondary();
public void setSecondary(S secondary);
// Return true if both primaries and both secondaries are equal
public boolean equals(CompositeWritable o);
// Return the primary's hash code
public long hashCode();
// Sort first by primary and then by secondary
public int compareTo(CompositeWritable o);
public void readFields(DataInput in);
public void write(DataOutput out);
}
{code}
With such a class, implementing a secondary sort would mean just implementing a
custom grouping comparator. That comparator could be implemented as part of
this JIRA:
{code}
package org.apache.hadoop.io;
public class CompositeGroupingComparator extends WritableComparator {
...
}
{code}
Or some such.
Crunch also provides {{Tuple3}}, {{Tuple4}}, and {{TupleN}} classes, but I
don't think we need to add equivalents. If someone really wants that
capability, they can nest composite keys.
Don't forget to add unit tests!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6808) Log map attempts as part of shuffle handler audit log
[ https://issues.apache.org/jira/browse/MAPREDUCE-6808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated MAPREDUCE-6808: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0-alpha3 Status: Resolved (was: Patch Available) Committed to trunk. Thanks for the patch, [~pairg], and the reviews, [~kshukla]! > Log map attempts as part of shuffle handler audit log > - > > Key: MAPREDUCE-6808 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6808 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 3.0.0-alpha1 >Reporter: Jonathan Eagles >Assignee: Gergő Pásztor > Fix For: 3.0.0-alpha3 > > Attachments: MAPREDUCE-6808_v1.patch, MAPREDUCE-6808_v2.patch > > > With the introduction tez, multiple unrelated reducers in the same job will > have the same id. The audit log won't be able to distinguish which map > attempt the reduce was fetching. This jira is to discuss the possibility to > add map attempts logging to distinguish between the two. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6808) Log map attempts as part of shuffle handler audit log
[ https://issues.apache.org/jira/browse/MAPREDUCE-6808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15838716#comment-15838716 ] Daniel Templeton commented on MAPREDUCE-6808: - Sweet. +1 > Log map attempts as part of shuffle handler audit log > - > > Key: MAPREDUCE-6808 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6808 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 3.0.0-alpha1 >Reporter: Jonathan Eagles >Assignee: Gergő Pásztor > Attachments: MAPREDUCE-6808_v1.patch, MAPREDUCE-6808_v2.patch > > > With the introduction tez, multiple unrelated reducers in the same job will > have the same id. The audit log won't be able to distinguish which map > attempt the reduce was fetching. This jira is to discuss the possibility to > add map attempts logging to distinguish between the two. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6808) Log map attempts as part of shuffle handler audit log
[ https://issues.apache.org/jira/browse/MAPREDUCE-6808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15836456#comment-15836456 ] Daniel Templeton commented on MAPREDUCE-6808: - Tiny nit, but can we make "mapper" be "mappers"? > Log map attempts as part of shuffle handler audit log > - > > Key: MAPREDUCE-6808 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6808 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 3.0.0-alpha1 >Reporter: Jonathan Eagles >Assignee: Gergő Pásztor > Attachments: MAPREDUCE-6808_v1.patch > > > With the introduction tez, multiple unrelated reducers in the same job will > have the same id. The audit log won't be able to distinguish which map > attempt the reduce was fetching. This jira is to discuss the possibility to > add map attempts logging to distinguish between the two. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6715) Fix Several Unsafe Practices
[
https://issues.apache.org/jira/browse/MAPREDUCE-6715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated MAPREDUCE-6715:
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha2
2.9.0
Status: Resolved (was: Patch Available)
Thanks for the patch, [~yufeigu]. Committed to trunk and branch-2
> Fix Several Unsafe Practices
>
>
> Key: MAPREDUCE-6715
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6715
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.9.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: MAPREDUCE-6715.001.patch, MAPREDUCE-6715.002.patch,
> MAPREDUCE-6715.003.patch, MAPREDUCE-6715.004.patch, MAPREDUCE-6715.005.patch,
> MAPREDUCE-6715.006.patch
>
>
> {code}
> Null DereferenceCleanupQueue.java:139
> Weak SecurityManager Check: Overridable Method
> LocalDistributedCacheManager.java:229
> Null DereferenceTextOutputFormat.java:137
> Null DereferenceShuffleSchedulerImpl.java:422
> Null DereferenceMapTask.java:415
> Null DereferencePentomino.java:160
> Unreleased Resource: StreamsTeraScheduler.java:77
> Unreleased Resource: StreamsCLI.java:570
> Null DereferenceCLI.java:370
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6715) Fix Several Unsafe Practices
[
https://issues.apache.org/jira/browse/MAPREDUCE-6715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated MAPREDUCE-6715:
Summary: Fix Several Unsafe Practices (was: Fix Bad Practices)
> Fix Several Unsafe Practices
>
>
> Key: MAPREDUCE-6715
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6715
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.9.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: MAPREDUCE-6715.001.patch, MAPREDUCE-6715.002.patch,
> MAPREDUCE-6715.003.patch, MAPREDUCE-6715.004.patch, MAPREDUCE-6715.005.patch,
> MAPREDUCE-6715.006.patch
>
>
> {code}
> Null DereferenceCleanupQueue.java:139
> Weak SecurityManager Check: Overridable Method
> LocalDistributedCacheManager.java:229
> Null DereferenceTextOutputFormat.java:137
> Null DereferenceShuffleSchedulerImpl.java:422
> Null DereferenceMapTask.java:415
> Null DereferencePentomino.java:160
> Unreleased Resource: StreamsTeraScheduler.java:77
> Unreleased Resource: StreamsCLI.java:570
> Null DereferenceCLI.java:370
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6715) Fix Bad Practices
[
https://issues.apache.org/jira/browse/MAPREDUCE-6715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15803071#comment-15803071
]
Daniel Templeton commented on MAPREDUCE-6715:
-
Yay! LGTM +1 I'll commit shortly.
> Fix Bad Practices
> -
>
> Key: MAPREDUCE-6715
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6715
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.9.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: MAPREDUCE-6715.001.patch, MAPREDUCE-6715.002.patch,
> MAPREDUCE-6715.003.patch, MAPREDUCE-6715.004.patch, MAPREDUCE-6715.005.patch,
> MAPREDUCE-6715.006.patch
>
>
> {code}
> Null DereferenceCleanupQueue.java:139
> Weak SecurityManager Check: Overridable Method
> LocalDistributedCacheManager.java:229
> Null DereferenceTextOutputFormat.java:137
> Null DereferenceShuffleSchedulerImpl.java:422
> Null DereferenceMapTask.java:415
> Null DereferencePentomino.java:160
> Unreleased Resource: StreamsTeraScheduler.java:77
> Unreleased Resource: StreamsCLI.java:570
> Null DereferenceCLI.java:370
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6715) Fix Bad Practices
[
https://issues.apache.org/jira/browse/MAPREDUCE-6715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15802758#comment-15802758
]
Daniel Templeton commented on MAPREDUCE-6715:
-
That part looks fine to me. I'm looking more closely at the
{{ShuffleSchedulerImpl}} changes now. First, throwing {{InterruptedException}}
in the wrong thing to do. Second, the analyzer is wrong--{{host}} can't be
null. Maybe if you rearrange the code like this: {code} Iterator
iter = pendingHosts.iterator();
// Safe to take one because we know pendingHosts isn't empty
host = iter.next();
int numToPick = random.nextInt(pendingHosts.size());
for (int i=0; i < numToPick; ++i) {
host = iter.next();
}{code} the analyzer would be happier?
> Fix Bad Practices
> -
>
> Key: MAPREDUCE-6715
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6715
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.9.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: MAPREDUCE-6715.001.patch, MAPREDUCE-6715.002.patch,
> MAPREDUCE-6715.003.patch, MAPREDUCE-6715.004.patch, MAPREDUCE-6715.005.patch
>
>
> {code}
> Null DereferenceCleanupQueue.java:139
> Weak SecurityManager Check: Overridable Method
> LocalDistributedCacheManager.java:229
> Null DereferenceTextOutputFormat.java:137
> Null DereferenceShuffleSchedulerImpl.java:422
> Null DereferenceMapTask.java:415
> Null DereferencePentomino.java:160
> Unreleased Resource: StreamsTeraScheduler.java:77
> Unreleased Resource: StreamsCLI.java:570
> Null DereferenceCLI.java:370
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6715) Fix Bad Practices
[
https://issues.apache.org/jira/browse/MAPREDUCE-6715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15801651#comment-15801651
]
Daniel Templeton commented on MAPREDUCE-6715:
-
LGTM. About that pointless null check... If it's there to make a code
analyzer happy, I'm OK with that. It might be good to add a comment to explain
that, though, so that someone else doesn't remove it later.
> Fix Bad Practices
> -
>
> Key: MAPREDUCE-6715
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6715
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.9.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: MAPREDUCE-6715.001.patch, MAPREDUCE-6715.002.patch,
> MAPREDUCE-6715.003.patch, MAPREDUCE-6715.004.patch
>
>
> {code}
> Null DereferenceCleanupQueue.java:139
> Weak SecurityManager Check: Overridable Method
> LocalDistributedCacheManager.java:229
> Null DereferenceTextOutputFormat.java:137
> Null DereferenceShuffleSchedulerImpl.java:422
> Null DereferenceMapTask.java:415
> Null DereferencePentomino.java:160
> Unreleased Resource: StreamsTeraScheduler.java:77
> Unreleased Resource: StreamsCLI.java:570
> Null DereferenceCLI.java:370
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6715) Fix Bad Practices
[
https://issues.apache.org/jira/browse/MAPREDUCE-6715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15795977#comment-15795977
]
Daniel Templeton commented on MAPREDUCE-6715:
-
Thanks for the updates. Just noticed a couple other things:
* {{lastException.getMessage()}} is often empty, so
{{lastException.toString()}} is better
* {{ReflectionUtils.newInstance()}} cannot return null, so the added null check
is pointless
* {{"Error in last collector was :"}} should be {{"Error in last collector was:
"}}
* {code} String line = in.readLine();
while (line != null) {
result.add(line);
line = in.readLine();
}{code} can be simplified to: {code} String line;
while ((line = in.readLine()) != null) {
result.add(line);
}{code} Your call if that's actually simpler
> Fix Bad Practices
> -
>
> Key: MAPREDUCE-6715
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6715
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.9.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: MAPREDUCE-6715.001.patch, MAPREDUCE-6715.002.patch,
> MAPREDUCE-6715.003.patch
>
>
> {code}
> Null DereferenceCleanupQueue.java:139
> Weak SecurityManager Check: Overridable Method
> LocalDistributedCacheManager.java:229
> Null DereferenceTextOutputFormat.java:137
> Null DereferenceShuffleSchedulerImpl.java:422
> Null DereferenceMapTask.java:415
> Null DereferencePentomino.java:160
> Unreleased Resource: StreamsTeraScheduler.java:77
> Unreleased Resource: StreamsCLI.java:570
> Null DereferenceCLI.java:370
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Resolved] (MAPREDUCE-6827) Failed to traverse Iterable values the second time in reduce() method
[
https://issues.apache.org/jira/browse/MAPREDUCE-6827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton resolved MAPREDUCE-6827.
-
Resolution: Not A Problem
That is known, documented, and intended behavior. The {{ValueIterator}}'s
{{hasNext()}} and {{next()}} methods defer defer to the {{ReduceContextImpl}}'s
{{BackupStore}} instance, so creating a new iterator won't help. The reason we
only go through the values once is to allow the data to be efficiently streamed.
> Failed to traverse Iterable values the second time in reduce() method
> -
>
> Key: MAPREDUCE-6827
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6827
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: task
>Affects Versions: 3.0.0-alpha1
> Environment: hadoop2.7.3
>Reporter: javaloveme
>
> Failed to traverse Iterable values the second time in reduce() method
> The following code is a reduce() method (of WordCount):
> {code:title=WordCount.java|borderStyle=solid}
> public static class WcReducer extends Reducer IntWritable> {
> @Override
> protected void reduce(Text key, Iterable values,
> Context context)
> throws IOException, InterruptedException {
> // print some logs
> List vals = new LinkedList<>();
> for(IntWritable i : values) {
> vals.add(i.toString());
> }
> System.out.println(String.format(" reduce(%s,
> [%s])",
> key, String.join(", ", vals)));
> // sum of values
> int sum = 0;
> for(IntWritable i : values) {
> sum += i.get();
> }
> System.out.println(String.format(" reduced(%s, %s)",
> key, sum));
>
> context.write(key, new IntWritable(sum));
> }
> }
> {code}
> After running it, we got the result that all sums were zero!
> After debugging, it was found that the second foreach-loop was not executed,
> and the root cause was the returned value of Iterable.iterator(), it returned
> the same instance in the two calls called by foreach-loop. In general,
> Iterable.iterator() should return a new instance in each call, such as
> ArrayList.iterator().
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-5155) Race condition in test case TestFetchFailure cause it to fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-5155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated MAPREDUCE-5155: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0-alpha2 Status: Resolved (was: Patch Available) Thanks for the patch, [~haibochen]. Committed to trunk. If you want it to go into branch-2, please provide a branch-2 patch. (Java 7 wants the var used in the anon inner class to be final.) > Race condition in test case TestFetchFailure cause it to fail > - > > Key: MAPREDUCE-5155 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5155 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 2.0.3-alpha > Environment: Suse x86_64 GNU/Linux > Java(TM) SE Runtime Environment (build 1.6.0_32-b05 >Reporter: Nemon Lou >Assignee: Haibo Chen >Priority: Minor > Fix For: 3.0.0-alpha2 > > Attachments: MAPREDUCE-5155.01.patch, > org.apache.hadoop.mapreduce.v2.app.TestFetchFailure-output.txt, > org.apache.hadoop.mapreduce.v2.app.TestFetchFailure.txt > > > I run into this once: > testFetchFailureWithRecovery(org.apache.hadoop.mapreduce.v2.app.TestFetchFailure): > Num completion events not correct expected:<1> but was:<0> > There is a race condition between job.getTaskAttemptCompletionEvents and > dealing with JOB_TASK_ATTEMPT_COMPLETED event. > If job.getTaskAttemptCompletionEvents invoked because of task in SUCCEEDED > state ,but before JOB_TASK_ATTEMPT_COMPLETED event scheduled,the test case > will fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-5155) Race condition in test case TestFetchFailure cause it to fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-5155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15783963#comment-15783963 ] Daniel Templeton commented on MAPREDUCE-5155: - Looks good to me. +1. I'll commit tomorrow unless there are other comments. > Race condition in test case TestFetchFailure cause it to fail > - > > Key: MAPREDUCE-5155 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5155 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 2.0.3-alpha > Environment: Suse x86_64 GNU/Linux > Java(TM) SE Runtime Environment (build 1.6.0_32-b05 >Reporter: Nemon Lou >Assignee: Haibo Chen >Priority: Minor > Attachments: MAPREDUCE-5155.01.patch, > org.apache.hadoop.mapreduce.v2.app.TestFetchFailure-output.txt, > org.apache.hadoop.mapreduce.v2.app.TestFetchFailure.txt > > > I run into this once: > testFetchFailureWithRecovery(org.apache.hadoop.mapreduce.v2.app.TestFetchFailure): > Num completion events not correct expected:<1> but was:<0> > There is a race condition between job.getTaskAttemptCompletionEvents and > dealing with JOB_TASK_ATTEMPT_COMPLETED event. > If job.getTaskAttemptCompletionEvents invoked because of task in SUCCEEDED > state ,but before JOB_TASK_ATTEMPT_COMPLETED event scheduled,the test case > will fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6715) Fix Bad Practices
[
https://issues.apache.org/jira/browse/MAPREDUCE-6715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15783953#comment-15783953
]
Daniel Templeton commented on MAPREDUCE-6715:
-
Thanks, [~yufeigu]. Some comments:
* In {{CleanupQueue}}, you're missing a space in the text at the line break
* In {{ShuffleSchedulerImpl}}, the {{LOG.debug()}} should be in a conditional
* In {{MapTask}} if you return {{null}}, you're going to cause NPEs
> Fix Bad Practices
> -
>
> Key: MAPREDUCE-6715
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6715
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.9.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: MAPREDUCE-6715.001.patch, MAPREDUCE-6715.002.patch
>
>
> {code}
> Null DereferenceCleanupQueue.java:139
> Weak SecurityManager Check: Overridable Method
> LocalDistributedCacheManager.java:229
> Null DereferenceTextOutputFormat.java:137
> Null DereferenceShuffleSchedulerImpl.java:422
> Null DereferenceMapTask.java:415
> Null DereferencePentomino.java:160
> Unreleased Resource: StreamsTeraScheduler.java:77
> Unreleased Resource: StreamsCLI.java:570
> Null DereferenceCLI.java:370
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-6734) Add option to distcp to preserve file path structure of source files at the destination
[ https://issues.apache.org/jira/browse/MAPREDUCE-6734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15730367#comment-15730367 ] Daniel Templeton edited comment on MAPREDUCE-6734 at 12/8/16 12:24 AM: --- I meant the priority. Oh, whoops. I meant bump the priority down to Major. was (Author: templedf): I meant the priority. > Add option to distcp to preserve file path structure of source files at the > destination > --- > > Key: MAPREDUCE-6734 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6734 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: distcp >Affects Versions: 3.0.0-alpha2 > Environment: Software platform >Reporter: Frederick Tucker >Priority: Critical > Labels: distcp, newbie, patch > Fix For: 3.0.0-alpha2 > > Attachments: MAPREDUCE-6734.3.0.0-alpha2.patch, > MAPREDUCE-6734.3.0.0-alpha2.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > When copying files using distcp with globbed source files, all the matched > files in the glob are copied in a single flat directory. This causes > problems when the file structure at the source is important. It also is an > issue when there are two files matched in the glob with the same name because > it causes a duplicate file error at the target. I'd like to have an option > to preserve the file structure of the source files when globbing inputs. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6734) Add option to distcp to preserve file path structure of source files at the destination
[
https://issues.apache.org/jira/browse/MAPREDUCE-6734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15730406#comment-15730406
]
Daniel Templeton commented on MAPREDUCE-6734:
-
Comments:
* In {{CopyListing}}, I wouldn't make the call to {{adjustPath()}} an
assignment. You're modifying the {{Text}} in the method, so the assignment
just looks suspicious. Same in {{CopyMapper.map()}}.
* In {{DistCpUtils.adjustPath()}}, the javadoc probably shouldn't talk about
keys and values since it's used in a broader context, e.g. {{CopyListing}}
* {{adjustPath()}} doesn't tolerate leaving out the leading slash. Seems a
logical thing for a user to want to do.
* In your tests, please add assert messages, especially for
{{assertTrue()}}/{{assertFalse()}}
* In the fail message, if it includes an exception, please use {{toString()}}
instead of {{getMessage()}}
* There's a lot of overlap between the tests in {{TestCopyMapper}}. Think you
can extract the common logic into a shared method? Also, it would be nice to
add some cleanup for the created files. I don't see any other methods doing
that, but maybe you'll start a trend!
* In the {{TestCopyListing}} test, you don't need the first
{{TestDistCpUtils.delete(fs, "/tmp");}}. I'd also make the
{{InvalidInputException}} expected instead of catching and ignoring
* In the {{TestDistCpUtils}} test, I think you can move that repeated logic
into a separate method and call that four times
> Add option to distcp to preserve file path structure of source files at the
> destination
> ---
>
> Key: MAPREDUCE-6734
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6734
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: distcp
>Affects Versions: 3.0.0-alpha2
> Environment: Software platform
>Reporter: Frederick Tucker
>Priority: Critical
> Labels: distcp, newbie, patch
> Fix For: 3.0.0-alpha2
>
> Attachments: MAPREDUCE-6734.3.0.0-alpha2.patch,
> MAPREDUCE-6734.3.0.0-alpha2.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> When copying files using distcp with globbed source files, all the matched
> files in the glob are copied in a single flat directory. This causes
> problems when the file structure at the source is important. It also is an
> issue when there are two files matched in the glob with the same name because
> it causes a duplicate file error at the target. I'd like to have an option
> to preserve the file structure of the source files when globbing inputs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6734) Add option to distcp to preserve file path structure of source files at the destination
[ https://issues.apache.org/jira/browse/MAPREDUCE-6734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15730367#comment-15730367 ] Daniel Templeton commented on MAPREDUCE-6734: - I meant the priority. > Add option to distcp to preserve file path structure of source files at the > destination > --- > > Key: MAPREDUCE-6734 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6734 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: distcp >Affects Versions: 3.0.0-alpha2 > Environment: Software platform >Reporter: Frederick Tucker >Priority: Critical > Labels: distcp, newbie, patch > Fix For: 3.0.0-alpha2 > > Attachments: MAPREDUCE-6734.3.0.0-alpha2.patch, > MAPREDUCE-6734.3.0.0-alpha2.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > When copying files using distcp with globbed source files, all the matched > files in the glob are copied in a single flat directory. This causes > problems when the file structure at the source is important. It also is an > issue when there are two files matched in the glob with the same name because > it causes a duplicate file error at the target. I'd like to have an option > to preserve the file structure of the source files when globbing inputs. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6734) Add option to distcp to preserve file path structure of source files at the destination
[ https://issues.apache.org/jira/browse/MAPREDUCE-6734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15730316#comment-15730316 ] Daniel Templeton commented on MAPREDUCE-6734: - Thanks, [~fctucker], for the patch. I'm taking a look at it now. In the meantime, can we drop this to Important? > Add option to distcp to preserve file path structure of source files at the > destination > --- > > Key: MAPREDUCE-6734 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6734 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: distcp >Affects Versions: 3.0.0-alpha2 > Environment: Software platform >Reporter: Frederick Tucker >Priority: Critical > Labels: distcp, newbie, patch > Fix For: 3.0.0-alpha2 > > Attachments: MAPREDUCE-6734.3.0.0-alpha2.patch, > MAPREDUCE-6734.3.0.0-alpha2.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > When copying files using distcp with globbed source files, all the matched > files in the glob are copied in a single flat directory. This causes > problems when the file structure at the source is important. It also is an > issue when there are two files matched in the glob with the same name because > it causes a duplicate file error at the target. I'd like to have an option > to preserve the file structure of the source files when globbing inputs. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-4683) We need to fix our build to create/distribute hadoop-mapreduce-client-core-tests.jar
[ https://issues.apache.org/jira/browse/MAPREDUCE-4683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15730298#comment-15730298 ] Daniel Templeton commented on MAPREDUCE-4683: - Seems fine to me. +1 Shall we get this in? > We need to fix our build to create/distribute > hadoop-mapreduce-client-core-tests.jar > > > Key: MAPREDUCE-4683 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4683 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Reporter: Arun C Murthy >Assignee: Akira Ajisaka >Priority: Critical > Attachments: MAPREDUCE-4683.patch > > > We need to fix our build to create/distribute > hadoop-mapreduce-client-core-tests.jar, need this before MAPREDUCE-4253 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6704) Container fail to launch for mapred application
[
https://issues.apache.org/jira/browse/MAPREDUCE-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15730181#comment-15730181
]
Daniel Templeton commented on MAPREDUCE-6704:
-
Works for me.
> Container fail to launch for mapred application
> ---
>
> Key: MAPREDUCE-6704
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6704
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Blocker
> Attachments: 0001-MAPREDUCE-6704.patch, 0001-YARN-5026.patch,
> ClusterSetup.html, MAPREDUCE-6704.0002.patch, MR-6704-branch2.8.tar.gz,
> MR-6704-trunk-tempPatch.tar.gz, MR-6704-trunk.tar.gz, SingleCluster.html,
> container-whitelist-env-wip.patch, temp.patch
>
>
> Container fail to launch for mapred application.
> As part for launch script {{HADOOP_MAPRED_HOME}} default value is not set
> .After
> https://github.com/apache/hadoop/commit/9d4d30243b0fc9630da51a2c17b543ef671d035c
>{{HADOOP_MAPRED_HOME}} is not able to get from {{builder.environment()}}
> since {{DefaultContainerExecutor#buildCommandExecutor}} sets inherit to false.
> {noformat}
> 16/05/02 09:16:05 INFO mapreduce.Job: Job job_1462155939310_0004 failed with
> state FAILED due to: Application application_1462155939310_0004 failed 2
> times due to AM Container for appattempt_1462155939310_0004_02 exited
> with exitCode: 1
> Failing this attempt.Diagnostics: Exception from container-launch.
> Container id: container_1462155939310_0004_02_01
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:946)
> at org.apache.hadoop.util.Shell.run(Shell.java:850)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:227)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:385)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:281)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:89)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier;
> support was removed in 8.0
> Error: Could not find or load main class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier;
> support was removed in 8.0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6704) Container fail to launch for mapred application
[
https://issues.apache.org/jira/browse/MAPREDUCE-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15687800#comment-15687800
]
Daniel Templeton commented on MAPREDUCE-6704:
-
Thanks, [~tangzhankun]. Just to be clear, you're saying that after resolving
your versioning issues, the procedure documented in this patch worked for you?
> Container fail to launch for mapred application
> ---
>
> Key: MAPREDUCE-6704
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6704
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Blocker
> Attachments: 0001-MAPREDUCE-6704.patch, 0001-YARN-5026.patch,
> ClusterSetup.html, MAPREDUCE-6704.0002.patch, MR-6704-branch2.8.tar.gz,
> MR-6704-trunk-tempPatch.tar.gz, MR-6704-trunk.tar.gz, SingleCluster.html,
> container-whitelist-env-wip.patch, temp.patch
>
>
> Container fail to launch for mapred application.
> As part for launch script {{HADOOP_MAPRED_HOME}} default value is not set
> .After
> https://github.com/apache/hadoop/commit/9d4d30243b0fc9630da51a2c17b543ef671d035c
>{{HADOOP_MAPRED_HOME}} is not able to get from {{builder.environment()}}
> since {{DefaultContainerExecutor#buildCommandExecutor}} sets inherit to false.
> {noformat}
> 16/05/02 09:16:05 INFO mapreduce.Job: Job job_1462155939310_0004 failed with
> state FAILED due to: Application application_1462155939310_0004 failed 2
> times due to AM Container for appattempt_1462155939310_0004_02 exited
> with exitCode: 1
> Failing this attempt.Diagnostics: Exception from container-launch.
> Container id: container_1462155939310_0004_02_01
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:946)
> at org.apache.hadoop.util.Shell.run(Shell.java:850)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:227)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:385)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:281)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:89)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier;
> support was removed in 8.0
> Error: Could not find or load main class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier;
> support was removed in 8.0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6704) Container fail to launch for mapred application
[
https://issues.apache.org/jira/browse/MAPREDUCE-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674297#comment-15674297
]
Daniel Templeton commented on MAPREDUCE-6704:
-
[~tangzhankun], you'll want to test with a branch-2 cluster so that you have
all the latest fixes, so you'll either need to update the Docker image or
create your own. Let us know how the testing goes.
> Container fail to launch for mapred application
> ---
>
> Key: MAPREDUCE-6704
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6704
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Blocker
> Attachments: 0001-MAPREDUCE-6704.patch, 0001-YARN-5026.patch,
> ClusterSetup.html, MAPREDUCE-6704.0002.patch, MR-6704-branch2.8.tar.gz,
> MR-6704-trunk-tempPatch.tar.gz, MR-6704-trunk.tar.gz, SingleCluster.html,
> container-whitelist-env-wip.patch, temp.patch
>
>
> Container fail to launch for mapred application.
> As part for launch script {{HADOOP_MAPRED_HOME}} default value is not set
> .After
> https://github.com/apache/hadoop/commit/9d4d30243b0fc9630da51a2c17b543ef671d035c
>{{HADOOP_MAPRED_HOME}} is not able to get from {{builder.environment()}}
> since {{DefaultContainerExecutor#buildCommandExecutor}} sets inherit to false.
> {noformat}
> 16/05/02 09:16:05 INFO mapreduce.Job: Job job_1462155939310_0004 failed with
> state FAILED due to: Application application_1462155939310_0004 failed 2
> times due to AM Container for appattempt_1462155939310_0004_02 exited
> with exitCode: 1
> Failing this attempt.Diagnostics: Exception from container-launch.
> Container id: container_1462155939310_0004_02_01
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:946)
> at org.apache.hadoop.util.Shell.run(Shell.java:850)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:227)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:385)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:281)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:89)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier;
> support was removed in 8.0
> Error: Could not find or load main class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier;
> support was removed in 8.0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6704) Container fail to launch for mapred application
[
https://issues.apache.org/jira/browse/MAPREDUCE-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15654241#comment-15654241
]
Daniel Templeton commented on MAPREDUCE-6704:
-
[~tangzhankun], I'm pretty surprised by that. The Docker container should
inherit the env vars set for the container, which is what this whitelist
property does. Could you run a Docker job with delayed cleanup
({{yarn.nodemanager.delete.debug-delay-sec}}) set so that you can see the
launch script? That should give you a good idea what's going on.
[~bibinchundatt], I'm fine with the patch, but I would like to understand why
it's not working for Docker. As there are alternative solutions, we should
make sure we pick one that works across the board.
> Container fail to launch for mapred application
> ---
>
> Key: MAPREDUCE-6704
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6704
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Blocker
> Attachments: 0001-MAPREDUCE-6704.patch, 0001-YARN-5026.patch,
> ClusterSetup.html, MAPREDUCE-6704.0002.patch, SingleCluster.html,
> container-whitelist-env-wip.patch
>
>
> Container fail to launch for mapred application.
> As part for launch script {{HADOOP_MAPRED_HOME}} default value is not set
> .After
> https://github.com/apache/hadoop/commit/9d4d30243b0fc9630da51a2c17b543ef671d035c
>{{HADOOP_MAPRED_HOME}} is not able to get from {{builder.environment()}}
> since {{DefaultContainerExecutor#buildCommandExecutor}} sets inherit to false.
> {noformat}
> 16/05/02 09:16:05 INFO mapreduce.Job: Job job_1462155939310_0004 failed with
> state FAILED due to: Application application_1462155939310_0004 failed 2
> times due to AM Container for appattempt_1462155939310_0004_02 exited
> with exitCode: 1
> Failing this attempt.Diagnostics: Exception from container-launch.
> Container id: container_1462155939310_0004_02_01
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:946)
> at org.apache.hadoop.util.Shell.run(Shell.java:850)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:227)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:385)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:281)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:89)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier;
> support was removed in 8.0
> Error: Could not find or load main class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier;
> support was removed in 8.0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6704) Container fail to launch for mapred application
[
https://issues.apache.org/jira/browse/MAPREDUCE-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15638066#comment-15638066
]
Daniel Templeton commented on MAPREDUCE-6704:
-
The patch looks fine. I'm OK with this solution. After thinking it over, I
don't think we need to take it to the mailing list. As long as we're just
updating docs, this doesn't need to be a high profile issue.
Anyone have any concerns about this patch as the solution to the out-of-the-box
issue for MapReduce?
> Container fail to launch for mapred application
> ---
>
> Key: MAPREDUCE-6704
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6704
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Blocker
> Attachments: 0001-MAPREDUCE-6704.patch, 0001-YARN-5026.patch,
> MAPREDUCE-6704.0002.patch, container-whitelist-env-wip.patch
>
>
> Container fail to launch for mapred application.
> As part for launch script {{HADOOP_MAPRED_HOME}} default value is not set
> .After
> https://github.com/apache/hadoop/commit/9d4d30243b0fc9630da51a2c17b543ef671d035c
>{{HADOOP_MAPRED_HOME}} is not able to get from {{builder.environment()}}
> since {{DefaultContainerExecutor#buildCommandExecutor}} sets inherit to false.
> {noformat}
> 16/05/02 09:16:05 INFO mapreduce.Job: Job job_1462155939310_0004 failed with
> state FAILED due to: Application application_1462155939310_0004 failed 2
> times due to AM Container for appattempt_1462155939310_0004_02 exited
> with exitCode: 1
> Failing this attempt.Diagnostics: Exception from container-launch.
> Container id: container_1462155939310_0004_02_01
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:946)
> at org.apache.hadoop.util.Shell.run(Shell.java:850)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:227)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:385)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:281)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:89)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier;
> support was removed in 8.0
> Error: Could not find or load main class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier;
> support was removed in 8.0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6704) Container fail to launch for mapred application
[
https://issues.apache.org/jira/browse/MAPREDUCE-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15629641#comment-15629641
]
Daniel Templeton commented on MAPREDUCE-6704:
-
Yep. I was planning to submit a JIRA this morning to make exactly that change.
> Container fail to launch for mapred application
> ---
>
> Key: MAPREDUCE-6704
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6704
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Blocker
> Attachments: 0001-MAPREDUCE-6704.patch, 0001-YARN-5026.patch,
> container-whitelist-env-wip.patch
>
>
> Container fail to launch for mapred application.
> As part for launch script {{HADOOP_MAPRED_HOME}} default value is not set
> .After
> https://github.com/apache/hadoop/commit/9d4d30243b0fc9630da51a2c17b543ef671d035c
>{{HADOOP_MAPRED_HOME}} is not able to get from {{builder.environment()}}
> since {{DefaultContainerExecutor#buildCommandExecutor}} sets inherit to false.
> {noformat}
> 16/05/02 09:16:05 INFO mapreduce.Job: Job job_1462155939310_0004 failed with
> state FAILED due to: Application application_1462155939310_0004 failed 2
> times due to AM Container for appattempt_1462155939310_0004_02 exited
> with exitCode: 1
> Failing this attempt.Diagnostics: Exception from container-launch.
> Container id: container_1462155939310_0004_02_01
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:946)
> at org.apache.hadoop.util.Shell.run(Shell.java:850)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:227)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:385)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:281)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:89)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier;
> support was removed in 8.0
> Error: Could not find or load main class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier;
> support was removed in 8.0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6704) Container fail to launch for mapred application
[
https://issues.apache.org/jira/browse/MAPREDUCE-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15625447#comment-15625447
]
Daniel Templeton commented on MAPREDUCE-6704:
-
Since this is a rather core issue, I'd love to see some additional opinions
here. [~andrew.wang], [~varun_saxena], [~ajisakaa], [~sjlee0], [~sunilg],
[~Naganarasimha], [~vinodkv], any thoughts? Should we take this to the mailing
list for more visibility?
> Container fail to launch for mapred application
> ---
>
> Key: MAPREDUCE-6704
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6704
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Priority: Blocker
> Attachments: 0001-MAPREDUCE-6704.patch, 0001-YARN-5026.patch,
> container-whitelist-env-wip.patch
>
>
> Container fail to launch for mapred application.
> As part for launch script {{HADOOP_MAPRED_HOME}} default value is not set
> .After
> https://github.com/apache/hadoop/commit/9d4d30243b0fc9630da51a2c17b543ef671d035c
>{{HADOOP_MAPRED_HOME}} is not able to get from {{builder.environment()}}
> since {{DefaultContainerExecutor#buildCommandExecutor}} sets inherit to false.
> {noformat}
> 16/05/02 09:16:05 INFO mapreduce.Job: Job job_1462155939310_0004 failed with
> state FAILED due to: Application application_1462155939310_0004 failed 2
> times due to AM Container for appattempt_1462155939310_0004_02 exited
> with exitCode: 1
> Failing this attempt.Diagnostics: Exception from container-launch.
> Container id: container_1462155939310_0004_02_01
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:946)
> at org.apache.hadoop.util.Shell.run(Shell.java:850)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:227)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:385)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:281)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:89)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier;
> support was removed in 8.0
> Error: Could not find or load main class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier;
> support was removed in 8.0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6704) Container fail to launch for mapred application
[
https://issues.apache.org/jira/browse/MAPREDUCE-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15615461#comment-15615461
]
Daniel Templeton commented on MAPREDUCE-6704:
-
As if we're not split enough on this decision, let me throw in one more option.
The final and correct answer is that users should set up distributed cache
deployment (per
[https://hadoop.apache.org/docs/r3.0.0-alpha1/hadoop-mapreduce-client/hadoop-mapreduce-client-core/DistributedCacheDeploy.html]).
At that point things just work, and upgrades become simpler. The problem is
that distributed cache deployment isn't an option for out of the box.
Spark does not have the problem we're trying to solve here. Every time you
submit a Spark application, they ship the assembly JAR via the distributed
cache. If we want to push users to distributed cache deployment, maybe the way
to solve the out-of-box problem is the light version of dist cache deployment:
-libjars.
Option #5) Add a MapReduce property that controls whether the MapReduce JARs
are automatically shipped with the job via -libjars, and turn it on by default.
Yes, it's inefficient (in both time and space), but it works out of the box
and is a natural segue into dist cache deployment.
> Container fail to launch for mapred application
> ---
>
> Key: MAPREDUCE-6704
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6704
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Priority: Blocker
> Attachments: 0001-MAPREDUCE-6704.patch, 0001-YARN-5026.patch,
> container-whitelist-env-wip.patch
>
>
> Container fail to launch for mapred application.
> As part for launch script {{HADOOP_MAPRED_HOME}} default value is not set
> .After
> https://github.com/apache/hadoop/commit/9d4d30243b0fc9630da51a2c17b543ef671d035c
>{{HADOOP_MAPRED_HOME}} is not able to get from {{builder.environment()}}
> since {{DefaultContainerExecutor#buildCommandExecutor}} sets inherit to false.
> {noformat}
> 16/05/02 09:16:05 INFO mapreduce.Job: Job job_1462155939310_0004 failed with
> state FAILED due to: Application application_1462155939310_0004 failed 2
> times due to AM Container for appattempt_1462155939310_0004_02 exited
> with exitCode: 1
> Failing this attempt.Diagnostics: Exception from container-launch.
> Container id: container_1462155939310_0004_02_01
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:946)
> at org.apache.hadoop.util.Shell.run(Shell.java:850)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:227)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:385)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:281)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:89)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier;
> support was removed in 8.0
> Error: Could not find or load main class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier;
> support was removed in 8.0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-6704) Container fail to launch for mapred application
[
https://issues.apache.org/jira/browse/MAPREDUCE-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15612421#comment-15612421
]
Daniel Templeton edited comment on MAPREDUCE-6704 at 10/28/16 1:39 PM:
---
+1 to #3, but *making it the default*, not just documenting it. I agree with
[~aw] that a little pollution is OK if it makes the out-of-box experience
significantly better. And I don't think spilling over an environment variable
is so bad anyway.
#1 has several issues. In addition to what was already stated, the vars are
evaluated on the client, so if the client doesn't have the same paths as the NM
hosts, it won't work.
#2 is a code-level pollution, which is a more serious thing than #3.
#4 doesn't sound like the right idea to me. Users don't need to control the
whitelisting on a per-job basis, so it's solving a problem that doesn't exist.
It also has some security implications that need to be carefully considered.
was (Author: templedf):
+1 to #3. I agree with [~aw] that a little pollution is OK if it makes the
out-of-box experience significantly better. And I don't think spilling over an
environment variable is so bad anyway.
#1 has several issues. In addition to what was already stated, the vars are
evaluated on the client, so if the client doesn't have the same paths as the NM
hosts, it won't work.
#2 is a code-level pollution, which is a more serious thing than #3.
#4 doesn't sound like the right idea to me. Users don't need to control the
whitelisting on a per-job basis, so it's solving a problem that doesn't exist.
It also has some security implications that need to be carefully considered.
> Container fail to launch for mapred application
> ---
>
> Key: MAPREDUCE-6704
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6704
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Priority: Blocker
> Attachments: 0001-MAPREDUCE-6704.patch, 0001-YARN-5026.patch,
> container-whitelist-env-wip.patch
>
>
> Container fail to launch for mapred application.
> As part for launch script {{HADOOP_MAPRED_HOME}} default value is not set
> .After
> https://github.com/apache/hadoop/commit/9d4d30243b0fc9630da51a2c17b543ef671d035c
>{{HADOOP_MAPRED_HOME}} is not able to get from {{builder.environment()}}
> since {{DefaultContainerExecutor#buildCommandExecutor}} sets inherit to false.
> {noformat}
> 16/05/02 09:16:05 INFO mapreduce.Job: Job job_1462155939310_0004 failed with
> state FAILED due to: Application application_1462155939310_0004 failed 2
> times due to AM Container for appattempt_1462155939310_0004_02 exited
> with exitCode: 1
> Failing this attempt.Diagnostics: Exception from container-launch.
> Container id: container_1462155939310_0004_02_01
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:946)
> at org.apache.hadoop.util.Shell.run(Shell.java:850)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:227)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:385)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:281)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:89)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier;
> support was removed in 8.0
> Error: Could not find or load main class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier;
> support was removed in 8.0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-6704) Container fail to launch for mapred application
[
https://issues.apache.org/jira/browse/MAPREDUCE-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15612421#comment-15612421
]
Daniel Templeton edited comment on MAPREDUCE-6704 at 10/27/16 4:30 PM:
---
+1 to #3. I agree with [~aw] that a little pollution is OK if it makes the
out-of-box experience significantly better. And I don't think spilling over an
environment variable is so bad anyway.
#1 has several issues. In addition to what was already stated, the vars are
evaluated on the client, so if the client doesn't have the same paths as the NM
hosts, it won't work.
#2 is a code-level pollution, which is a more serious thing than #3.
#4 doesn't sound like the right idea to me. Users don't need to control the
whitelisting on a per-job basis, so it's solving a problem that doesn't exist.
It also has some security implications that need to be carefully considered.
was (Author: templedf):
+1 to #3. A agree with [~aw] that a little pollution is OK if it makes the
out-of-box experience significantly better. And I don't think spilling over an
environment variable is so bad anyway.
#1 has several issues. In addition to what was already stated, the vars are
evaluated on the client, so if the client doesn't have the same paths as the NM
hosts, it won't work.
#2 is a code-level pollution, which is a more serious thing than #3.
#4 doesn't sound like the right idea to me. Users don't need to control the
whitelisting on a per-job basis, so it's solving a problem that doesn't exist.
It also has some security implications that need to be carefully considered.
> Container fail to launch for mapred application
> ---
>
> Key: MAPREDUCE-6704
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6704
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Priority: Blocker
> Attachments: 0001-MAPREDUCE-6704.patch, 0001-YARN-5026.patch,
> container-whitelist-env-wip.patch
>
>
> Container fail to launch for mapred application.
> As part for launch script {{HADOOP_MAPRED_HOME}} default value is not set
> .After
> https://github.com/apache/hadoop/commit/9d4d30243b0fc9630da51a2c17b543ef671d035c
>{{HADOOP_MAPRED_HOME}} is not able to get from {{builder.environment()}}
> since {{DefaultContainerExecutor#buildCommandExecutor}} sets inherit to false.
> {noformat}
> 16/05/02 09:16:05 INFO mapreduce.Job: Job job_1462155939310_0004 failed with
> state FAILED due to: Application application_1462155939310_0004 failed 2
> times due to AM Container for appattempt_1462155939310_0004_02 exited
> with exitCode: 1
> Failing this attempt.Diagnostics: Exception from container-launch.
> Container id: container_1462155939310_0004_02_01
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:946)
> at org.apache.hadoop.util.Shell.run(Shell.java:850)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:227)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:385)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:281)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:89)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier;
> support was removed in 8.0
> Error: Could not find or load main class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier;
> support was removed in 8.0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6704) Container fail to launch for mapred application
[
https://issues.apache.org/jira/browse/MAPREDUCE-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15612421#comment-15612421
]
Daniel Templeton commented on MAPREDUCE-6704:
-
+1 to #3. A agree with [~aw] that a little pollution is OK if it makes the
out-of-box experience significantly better. And I don't think spilling over an
environment variable is so bad anyway.
#1 has several issues. In addition to what was already stated, the vars are
evaluated on the client, so if the client doesn't have the same paths as the NM
hosts, it won't work.
#2 is a code-level pollution, which is a more serious thing than #3.
#4 doesn't sound like the right idea to me. Users don't need to control the
whitelisting on a per-job basis, so it's solving a problem that doesn't exist.
It also has some security implications that need to be carefully considered.
> Container fail to launch for mapred application
> ---
>
> Key: MAPREDUCE-6704
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6704
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Priority: Blocker
> Attachments: 0001-MAPREDUCE-6704.patch, 0001-YARN-5026.patch,
> container-whitelist-env-wip.patch
>
>
> Container fail to launch for mapred application.
> As part for launch script {{HADOOP_MAPRED_HOME}} default value is not set
> .After
> https://github.com/apache/hadoop/commit/9d4d30243b0fc9630da51a2c17b543ef671d035c
>{{HADOOP_MAPRED_HOME}} is not able to get from {{builder.environment()}}
> since {{DefaultContainerExecutor#buildCommandExecutor}} sets inherit to false.
> {noformat}
> 16/05/02 09:16:05 INFO mapreduce.Job: Job job_1462155939310_0004 failed with
> state FAILED due to: Application application_1462155939310_0004 failed 2
> times due to AM Container for appattempt_1462155939310_0004_02 exited
> with exitCode: 1
> Failing this attempt.Diagnostics: Exception from container-launch.
> Container id: container_1462155939310_0004_02_01
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:946)
> at org.apache.hadoop.util.Shell.run(Shell.java:850)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:227)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:385)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:281)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:89)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier;
> support was removed in 8.0
> Error: Could not find or load main class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier;
> support was removed in 8.0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6728) Give fetchers hint when ShuffleHandler rejects a shuffling connection
[ https://issues.apache.org/jira/browse/MAPREDUCE-6728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15596254#comment-15596254 ] Daniel Templeton commented on MAPREDUCE-6728: - Thanks, [~haibochen]. Latest patch looks good to me. +1 (non-binding) > Give fetchers hint when ShuffleHandler rejects a shuffling connection > - > > Key: MAPREDUCE-6728 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6728 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: mapreduce6728.001.patch, mapreduce6728.002.patch, > mapreduce6728.003.patch, mapreduce6728.004.patch, mapreduce6728.005.patch, > mapreduce6728.prelim.patch > > > If # of open shuffle connection to a node goes over the max, ShuffleHandler > closes the connection immediately without giving fetchers any hint of the > reason, which causes fetchers to fail due to exceptions > java.net.SocketException: Unexpected end of file from server > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:772) > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633) > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:769) > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633) > at > sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323) > at > java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.verifyConnection(Fetcher.java:430) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.setupConnectionsWithRetry(Fetcher.java:395) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.openShuffleUrl(Fetcher.java:266) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:323) > at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:193) > OR > java.net.SocketException: Connection reset > at java.net.SocketInputStream.read(SocketInputStream.java:196) > at java.net.SocketInputStream.read(SocketInputStream.java:122) > at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) > at java.io.BufferedInputStream.read1(BufferedInputStream.java:275) > at java.io.BufferedInputStream.read(BufferedInputStream.java:334) > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:687) > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633) > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:769) > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633) > at > sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323) > at > java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.verifyConnection(Fetcher.java:430) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.setupConnectionsWithRetry(Fetcher.java:395) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.openShuffleUrl(Fetcher.java:266) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java > Such failures are counted as fetcher failures -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6728) Give fetchers hint when ShuffleHandler rejects a shuffling connection
[
https://issues.apache.org/jira/browse/MAPREDUCE-6728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15552572#comment-15552572
]
Daniel Templeton commented on MAPREDUCE-6728:
-
Sorry to string this out. I just caught two more items that should be
addressed:
# This comment: {code} /**
* This should be kept in sync with ShuffleHandler.FETCH_RETRY_DELAY.
*/
private static final long FETCH_RETRY_DELAY_DEFAULT = 1000L;{code} either
should not be a javadoc comment or should start with a sentence that is an
appropriate javadoc comment.
# The {{TestShuffleHandler}} should test that the Retry-after header is set to
a non-negative integer.
> Give fetchers hint when ShuffleHandler rejects a shuffling connection
> -
>
> Key: MAPREDUCE-6728
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6728
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Attachments: mapreduce6728.001.patch, mapreduce6728.002.patch,
> mapreduce6728.003.patch, mapreduce6728.004.patch, mapreduce6728.prelim.patch
>
>
> If # of open shuffle connection to a node goes over the max, ShuffleHandler
> closes the connection immediately without giving fetchers any hint of the
> reason, which causes fetchers to fail due to exceptions
> java.net.SocketException: Unexpected end of file from server
> at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:772)
> at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
> at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:769)
> at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323)
> at
> java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)
> at
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.verifyConnection(Fetcher.java:430)
> at
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.setupConnectionsWithRetry(Fetcher.java:395)
> at
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.openShuffleUrl(Fetcher.java:266)
> at
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:323)
> at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:193)
> OR
> java.net.SocketException: Connection reset
> at java.net.SocketInputStream.read(SocketInputStream.java:196)
> at java.net.SocketInputStream.read(SocketInputStream.java:122)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:687)
> at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
> at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:769)
> at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323)
> at
> java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)
> at
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.verifyConnection(Fetcher.java:430)
> at
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.setupConnectionsWithRetry(Fetcher.java:395)
> at
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.openShuffleUrl(Fetcher.java:266)
> at
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java
> Such failures are counted as fetcher failures
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6728) Give fetchers hint when ShuffleHandler rejects a shuffling connection
[ https://issues.apache.org/jira/browse/MAPREDUCE-6728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated MAPREDUCE-6728: Status: Open (was: Patch Available) > Give fetchers hint when ShuffleHandler rejects a shuffling connection > - > > Key: MAPREDUCE-6728 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6728 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: mapreduce6728.001.patch, mapreduce6728.002.patch, > mapreduce6728.003.patch, mapreduce6728.004.patch, mapreduce6728.prelim.patch > > > If # of open shuffle connection to a node goes over the max, ShuffleHandler > closes the connection immediately without giving fetchers any hint of the > reason, which causes fetchers to fail due to exceptions > java.net.SocketException: Unexpected end of file from server > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:772) > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633) > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:769) > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633) > at > sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323) > at > java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.verifyConnection(Fetcher.java:430) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.setupConnectionsWithRetry(Fetcher.java:395) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.openShuffleUrl(Fetcher.java:266) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:323) > at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:193) > OR > java.net.SocketException: Connection reset > at java.net.SocketInputStream.read(SocketInputStream.java:196) > at java.net.SocketInputStream.read(SocketInputStream.java:122) > at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) > at java.io.BufferedInputStream.read1(BufferedInputStream.java:275) > at java.io.BufferedInputStream.read(BufferedInputStream.java:334) > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:687) > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633) > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:769) > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633) > at > sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323) > at > java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.verifyConnection(Fetcher.java:430) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.setupConnectionsWithRetry(Fetcher.java:395) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.openShuffleUrl(Fetcher.java:266) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java > Such failures are counted as fetcher failures -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6789) TestAMWebApp fails
[
https://issues.apache.org/jira/browse/MAPREDUCE-6789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated MAPREDUCE-6789:
Status: Patch Available (was: Open)
> TestAMWebApp fails
> --
>
> Key: MAPREDUCE-6789
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6789
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Akira Ajisaka
>Assignee: Daniel Templeton
> Attachments: MAPREDUCE-6789.001.patch
>
>
> TestAMWebApp.testMRWebAppRedirection fails.
> {noformat}
> Running org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebApp
> Tests run: 12, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 27.337 sec
> <<< FAILURE! - in org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebApp
> testMRWebAppRedirection(org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebApp)
> Time elapsed: 0.854 sec <<< FAILURE!
> org.junit.ComparisonFailure:
> expected:<...ttp://9.9.9.9/proxy/[]application_0_/m...> but
> was:<...ttp://9.9.9.9/proxy/[redirect/]application_0_/m...>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebApp.testMRWebAppRedirection(TestAMWebApp.java:253)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6789) TestAMWebApp fails
[
https://issues.apache.org/jira/browse/MAPREDUCE-6789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated MAPREDUCE-6789:
Attachment: MAPREDUCE-6789.001.patch
This should resolve the issue.
> TestAMWebApp fails
> --
>
> Key: MAPREDUCE-6789
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6789
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Akira Ajisaka
>Assignee: Daniel Templeton
> Attachments: MAPREDUCE-6789.001.patch
>
>
> TestAMWebApp.testMRWebAppRedirection fails.
> {noformat}
> Running org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebApp
> Tests run: 12, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 27.337 sec
> <<< FAILURE! - in org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebApp
> testMRWebAppRedirection(org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebApp)
> Time elapsed: 0.854 sec <<< FAILURE!
> org.junit.ComparisonFailure:
> expected:<...ttp://9.9.9.9/proxy/[]application_0_/m...> but
> was:<...ttp://9.9.9.9/proxy/[redirect/]application_0_/m...>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebApp.testMRWebAppRedirection(TestAMWebApp.java:253)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Assigned] (MAPREDUCE-6789) TestAMWebApp fails
[
https://issues.apache.org/jira/browse/MAPREDUCE-6789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton reassigned MAPREDUCE-6789:
---
Assignee: Daniel Templeton
> TestAMWebApp fails
> --
>
> Key: MAPREDUCE-6789
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6789
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Akira Ajisaka
>Assignee: Daniel Templeton
>
> TestAMWebApp.testMRWebAppRedirection fails.
> {noformat}
> Running org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebApp
> Tests run: 12, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 27.337 sec
> <<< FAILURE! - in org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebApp
> testMRWebAppRedirection(org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebApp)
> Time elapsed: 0.854 sec <<< FAILURE!
> org.junit.ComparisonFailure:
> expected:<...ttp://9.9.9.9/proxy/[]application_0_/m...> but
> was:<...ttp://9.9.9.9/proxy/[redirect/]application_0_/m...>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebApp.testMRWebAppRedirection(TestAMWebApp.java:253)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6789) TestAMWebApp fails
[
https://issues.apache.org/jira/browse/MAPREDUCE-6789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15545498#comment-15545498
]
Daniel Templeton commented on MAPREDUCE-6789:
-
This is from YARN-4767. I'll look into it.
> TestAMWebApp fails
> --
>
> Key: MAPREDUCE-6789
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6789
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Akira Ajisaka
>
> TestAMWebApp.testMRWebAppRedirection fails.
> {noformat}
> Running org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebApp
> Tests run: 12, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 27.337 sec
> <<< FAILURE! - in org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebApp
> testMRWebAppRedirection(org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebApp)
> Time elapsed: 0.854 sec <<< FAILURE!
> org.junit.ComparisonFailure:
> expected:<...ttp://9.9.9.9/proxy/[]application_0_/m...> but
> was:<...ttp://9.9.9.9/proxy/[redirect/]application_0_/m...>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebApp.testMRWebAppRedirection(TestAMWebApp.java:253)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6728) Give fetchers hint when ShuffleHandler rejects a shuffling connection
[
https://issues.apache.org/jira/browse/MAPREDUCE-6728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15542927#comment-15542927
]
Daniel Templeton commented on MAPREDUCE-6728:
-
Thanks, [~haibochen]! The patch looks great. Two tiny nits:
# The colon in {{ShuffleHandler}} needs a space before it: {code} for
(Map.Entry header: headers.entrySet()) {{code}
# It would be nice to add a comment where you send the error code to say what
the header means and why you're sending it.
> Give fetchers hint when ShuffleHandler rejects a shuffling connection
> -
>
> Key: MAPREDUCE-6728
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6728
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Attachments: mapreduce6728.001.patch, mapreduce6728.002.patch,
> mapreduce6728.003.patch, mapreduce6728.prelim.patch
>
>
> If # of open shuffle connection to a node goes over the max, ShuffleHandler
> closes the connection immediately without giving fetchers any hint of the
> reason, which causes fetchers to fail due to exceptions
> java.net.SocketException: Unexpected end of file from server
> at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:772)
> at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
> at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:769)
> at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323)
> at
> java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)
> at
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.verifyConnection(Fetcher.java:430)
> at
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.setupConnectionsWithRetry(Fetcher.java:395)
> at
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.openShuffleUrl(Fetcher.java:266)
> at
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:323)
> at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:193)
> OR
> java.net.SocketException: Connection reset
> at java.net.SocketInputStream.read(SocketInputStream.java:196)
> at java.net.SocketInputStream.read(SocketInputStream.java:122)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:687)
> at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
> at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:769)
> at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323)
> at
> java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)
> at
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.verifyConnection(Fetcher.java:430)
> at
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.setupConnectionsWithRetry(Fetcher.java:395)
> at
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.openShuffleUrl(Fetcher.java:266)
> at
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java
> Such failures are counted as fetcher failures
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6783) Default setting 'inheritParentEnv=false' caused mr executor failed to initialize
[ https://issues.apache.org/jira/browse/MAPREDUCE-6783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15529736#comment-15529736 ] Daniel Templeton commented on MAPREDUCE-6783: - Looks like it to me. > Default setting 'inheritParentEnv=false' caused mr executor failed to > initialize > > > Key: MAPREDUCE-6783 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6783 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: nodemanager >Affects Versions: 3.0.0-alpha1 >Reporter: DENG FEI > > MR default classpath depend on the "HADOOP_MAPRED_HOME" , defined at > MRJobConfig#DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH. > On 3.0.0.alpha1 , contain "Remove parent's env vars from child processes " > which commit by [#Robert Kanter] ,but it's seem not found in jira or release > note. DefaultContainerExecutor & DefaultLinuxContinerExecutor default the > 'inheritParentEnv' as false,there is no any approach to set the > "HADOOP_MAPRED_HOME" ,the AM failed by "Could not find or load main class > org.apache.hadoop.mapreduce.v2.app.MRAppMaster" -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6728) Give fetchers hint when ShuffleHandler rejects a shuffling connection
[
https://issues.apache.org/jira/browse/MAPREDUCE-6728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510648#comment-15510648
]
Daniel Templeton commented on MAPREDUCE-6728:
-
Thanks for the patch, [~haibochen].
Comments:
* Instead of reusing the {{INITIAL_DELAY}}, you should define a retry delay
instead. You might also want to consider some kind of backoff. The most
correct approach would be to define the delay in the {{ShuffleHandler}} and
pass it back in the {{Retry-after}} header.
* I don't love defining an inner exception, but it appears to be the best
option. Can we call it something like {{TryAgainLaterException}} so that it's
really clear what it means? Should it be _static_? It should probably be
_private_.
* In this line the whitespace is wrong: {code}for(TaskAttemptID left:
remaining) {{code}
* Is there a clever way to not duplicate the code to put back the remaining
attempts? It appears in both _catch_ clauses.
* Please include javadoc for your {{ShuffleSchedulerImpl.penalize()}} method.
* In the {{ShuffleHandler.Shuffle.channelOpen()}} method, instead of writing
the error code there, you should probably call {{sendError()}}.
* In the {{TestFetcher}} test, watching the calls to {{hostFailed}} and
{{copyFailed}} seems brittle. Maybe instead watch the {{ioErrs}} counter?
* Please include a message in your new assert in {{TestShuffleHandler}}.
> Give fetchers hint when ShuffleHandler rejects a shuffling connection
> -
>
> Key: MAPREDUCE-6728
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6728
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Attachments: mapreduce6728.001.patch, mapreduce6728.prelim.patch
>
>
> If # of open shuffle connection to a node goes over the max, ShuffleHandler
> closes the connection immediately without giving fetchers any hint of the
> reason, which causes fetchers to fail due to exceptions
> java.net.SocketException: Unexpected end of file from server
> at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:772)
> at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
> at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:769)
> at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323)
> at
> java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)
> at
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.verifyConnection(Fetcher.java:430)
> at
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.setupConnectionsWithRetry(Fetcher.java:395)
> at
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.openShuffleUrl(Fetcher.java:266)
> at
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:323)
> at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:193)
> OR
> java.net.SocketException: Connection reset
> at java.net.SocketInputStream.read(SocketInputStream.java:196)
> at java.net.SocketInputStream.read(SocketInputStream.java:122)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:687)
> at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
> at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:769)
> at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323)
> at
> java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)
> at
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.verifyConnection(Fetcher.java:430)
> at
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.setupConnectionsWithRetry(Fetcher.java:395)
> at
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.openShuffleUrl(Fetcher.java:266)
> at
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java
> Such failures are counted as fetcher failures
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6765) MR should not schedule container requests in cases where reducer or mapper containers demand resource larger than the maximum supported
[ https://issues.apache.org/jira/browse/MAPREDUCE-6765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496819#comment-15496819 ] Daniel Templeton commented on MAPREDUCE-6765: - LGTM. +1 (non-binding) > MR should not schedule container requests in cases where reducer or mapper > containers demand resource larger than the maximum supported > --- > > Key: MAPREDUCE-6765 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6765 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.7.2 >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Minor > Fix For: 2.9.0 > > Attachments: mapreduce6765.001.patch, mapreduce6765.002.patch > > > When mapper or reducer containers request resource larger than the > maxResourceRequest in the cluster, job is to be killed. In such cases, it is > unnecessary to still schedule container requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6776) yarn.app.mapreduce.client.job.max-retries should have a more useful default
[ https://issues.apache.org/jira/browse/MAPREDUCE-6776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated MAPREDUCE-6776: Description: The default is 0, so any communication failure results in a client failure. Oozie doesn't like that. If the RM is failing over and Oozie gets a communication failure, it assumes the target job has failed. I propose raising the default to something modest like 3 or 5. The default retry interval is 2s. (was: The default is 0, so any communication results in a client failure. Oozie doesn't like that. If the RM is failing over and Oozie gets a communication failure, it assumes the target job has failed. I propose raising the default to something modest like 3 or 5. The default retry interval is 2s.) > yarn.app.mapreduce.client.job.max-retries should have a more useful default > --- > > Key: MAPREDUCE-6776 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6776 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client >Affects Versions: 2.8.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton > > The default is 0, so any communication failure results in a client failure. > Oozie doesn't like that. If the RM is failing over and Oozie gets a > communication failure, it assumes the target job has failed. I propose > raising the default to something modest like 3 or 5. The default retry > interval is 2s. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6776) yarn.app.mapreduce.client.job.max-retries should have a more useful default
Daniel Templeton created MAPREDUCE-6776: --- Summary: yarn.app.mapreduce.client.job.max-retries should have a more useful default Key: MAPREDUCE-6776 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6776 Project: Hadoop Map/Reduce Issue Type: Improvement Components: client Affects Versions: 2.8.0 Reporter: Daniel Templeton Assignee: Daniel Templeton The default is 0, so any communication results in a client failure. Oozie doesn't like that. If the RM is failing over and Oozie gets a communication failure, it assumes the target job has failed. I propose raising the default to something modest like 3 or 5. The default retry interval is 2s. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6765) MR should not schedule container requests in cases where reducer or mapper containers demand resource larger than the maximum supported
[ https://issues.apache.org/jira/browse/MAPREDUCE-6765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15471227#comment-15471227 ] Daniel Templeton commented on MAPREDUCE-6765: - I share Djikstra's dislike of multiple exit points from a function, especially in this case where the code is explicitly structured as a series of conditionals to avoid having multiple exit points. In both cases, could you please put the subsequent few lines of code in an _else_ instead of adding the returns? > MR should not schedule container requests in cases where reducer or mapper > containers demand resource larger than the maximum supported > --- > > Key: MAPREDUCE-6765 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6765 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.7.2 >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Minor > Fix For: 2.9.0 > > Attachments: mapreduce6765.001.patch > > > When mapper or reducer containers request resource larger than the > maxResourceRequest in the cluster, job is to be killed. In such cases, it is > unnecessary to still schedule container requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Resolved] (MAPREDUCE-6560) ClientServiceDelegate doesn't handle retries during AM restart as intended
[
https://issues.apache.org/jira/browse/MAPREDUCE-6560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton resolved MAPREDUCE-6560.
-
Resolution: Invalid
Looks like I was just wrong.
> ClientServiceDelegate doesn't handle retries during AM restart as intended
> --
>
> Key: MAPREDUCE-6560
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6560
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>
> In the {{invoke()}} method, I found the following code:
> {code}
> private AtomicBoolean usingAMProxy = new AtomicBoolean(false);
> ...
> // if it's AM shut down, do not decrement maxClientRetry as we wait
> for
> // AM to be restarted.
> if (!usingAMProxy.get()) {
> maxClientRetry--;
> }
> usingAMProxy.set(false);
> {code}
> When we create the AM proxy, we set the flag to true. If we fail to connect,
> the impact of the flag being true is that the code will try one extra time,
> giving it 400ms instead of just 300ms. I can't imagine that's the intended
> behavior. After any failure, the flag will forever more be false, but
> fortunately (?!?) the flag is otherwise unused.
> Looks like I need to do some archeology to figure out how we ended up here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6632) Master.getMasterAddress() should be updated to use YARN-4629
[
https://issues.apache.org/jira/browse/MAPREDUCE-6632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated MAPREDUCE-6632:
Status: Patch Available (was: Open)
> Master.getMasterAddress() should be updated to use YARN-4629
>
>
> Key: MAPREDUCE-6632
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6632
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: applicationmaster
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Minor
> Attachments: MAPREDUCE-6632.001.patch
>
>
> The new {{YarnClientUtil.getRmPrincipal()}} method can replace most of the
> {{Master.getMasterAddress()}} method and should to reduce redundancy and
> improve servicability.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6632) Master.getMasterAddress() should be updated to use YARN-4629
[
https://issues.apache.org/jira/browse/MAPREDUCE-6632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated MAPREDUCE-6632:
Attachment: MAPREDUCE-6632.001.patch
This patch adds a dependency from hadoop-mapreduce-client-core on
hadoop-yarn-client, but it removes a decent chunk of duplicated code.
> Master.getMasterAddress() should be updated to use YARN-4629
>
>
> Key: MAPREDUCE-6632
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6632
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: applicationmaster
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Minor
> Attachments: MAPREDUCE-6632.001.patch
>
>
> The new {{YarnClientUtil.getRmPrincipal()}} method can replace most of the
> {{Master.getMasterAddress()}} method and should to reduce redundancy and
> improve servicability.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6767) TestSlive fails after a common change
[ https://issues.apache.org/jira/browse/MAPREDUCE-6767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15435883#comment-15435883 ] Daniel Templeton commented on MAPREDUCE-6767: - Oops. Thanks, [~chris.douglas]. > TestSlive fails after a common change > - > > Key: MAPREDUCE-6767 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6767 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Kihwal Lee >Assignee: Daniel Templeton > Attachments: MAPREDUCE-6767.001.patch > > > It looks like this was broken after HADOOP-12726. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6767) TestSlive fails after a common change
[ https://issues.apache.org/jira/browse/MAPREDUCE-6767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated MAPREDUCE-6767: Attachment: MAPREDUCE-6767.001.patch This resolves the Slive failures. > TestSlive fails after a common change > - > > Key: MAPREDUCE-6767 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6767 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Kihwal Lee >Assignee: Daniel Templeton > Attachments: MAPREDUCE-6767.001.patch > > > It looks like this was broken after HADOOP-12726. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Assigned] (MAPREDUCE-6767) TestSlive fails after a common change
[ https://issues.apache.org/jira/browse/MAPREDUCE-6767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton reassigned MAPREDUCE-6767: --- Assignee: Daniel Templeton > TestSlive fails after a common change > - > > Key: MAPREDUCE-6767 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6767 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Kihwal Lee >Assignee: Daniel Templeton > > It looks like this was broken after HADOOP-12726. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6767) TestSlive fails after a common change
[ https://issues.apache.org/jira/browse/MAPREDUCE-6767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15435619#comment-15435619 ] Daniel Templeton commented on MAPREDUCE-6767: - Any details you can give? > TestSlive fails after a common change > - > > Key: MAPREDUCE-6767 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6767 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Kihwal Lee > > It looks like this was broken after HADOOP-12726. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6761) Regression when handling providers - invalid configuration ServiceConfiguration causes Cluster initialization failure
[
https://issues.apache.org/jira/browse/MAPREDUCE-6761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15427613#comment-15427613
]
Daniel Templeton commented on MAPREDUCE-6761:
-
Thanks for the patch, [~pvary]!
[~rchiang], using the JDK5 foreach is exactly the error [~kshukla] made that
created the issue. The {{Iterator}} that {{ServiceLoader.iterator()}} returns
can throw a {{ServiceConfigurationError}} on the {{next()}} and {{hasNext()}}
calls. [~pvary], looks like you should also guard the {{hasNext()}} call.
I agree that the error message should be more prescriptive. I also agree that
adding a test would be a good thing. You don't have to make it public, though.
Default visibility (with {{@VisibleForTesting}}) should be enough.
> Regression when handling providers - invalid configuration
> ServiceConfiguration causes Cluster initialization failure
> -
>
> Key: MAPREDUCE-6761
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6761
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 3.0.0-alpha2
>Reporter: Peter Vary
>Assignee: Peter Vary
> Labels: supportability
> Attachments: MAPREDUCE-6761.patch
>
>
> When a rouge org.apache.hadoop.mapreduce.protocol.ClientProtocolProvider
> defines a provider that is not on classpath, then the initialization is
> failed with the following exception:
> java.util.ServiceConfigurationError:
> org.apache.hadoop.mapreduce.protocol.ClientProtocolProvider: Provider
> org.apache.hadoop.mapred.YarnClientProtocolProvider not found
> at java.util.ServiceLoader.fail(ServiceLoader.java:239)
> at java.util.ServiceLoader.access$300(ServiceLoader.java:185)
> at
> java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:372)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
> at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
> at org.apache.hadoop.mapreduce.Cluster.initProviderList(Cluster.java:84)
> at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:114)
> at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:108)
> at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:101)
> at org.apache.hadoop.mapred.JobClient.init(JobClient.java:477)
> at org.apache.hadoop.mapred.JobClient.(JobClient.java:455)
> This regression is caused by MAPREDUCE-6473
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6724) Unsafe conversion from long to int in MergeManagerImpl.unconditionalReserve()
[ https://issues.apache.org/jira/browse/MAPREDUCE-6724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383221#comment-15383221 ] Daniel Templeton commented on MAPREDUCE-6724: - LGTM. +1 > Unsafe conversion from long to int in MergeManagerImpl.unconditionalReserve() > - > > Key: MAPREDUCE-6724 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6724 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: mapreduce6724.001.patch, mapreduce6724.002.patch, > mapreduce6724.003.patch, mapreduce6724.004.patch, mapreduce6724.005.patch > > > When shuffle is done in memory, MergeManagerImpl converts the requested size > to an int to allocate an instance of InMemoryMapOutput. This results in an > overflow if the requested size is bigger than Integer.MAX_VALUE and > eventually causes the reducer to fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6724) Unsafe conversion from long to int in MergeManagerImpl.unconditionalReserve()
[ https://issues.apache.org/jira/browse/MAPREDUCE-6724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383085#comment-15383085 ] Daniel Templeton commented on MAPREDUCE-6724: - Works for me. I'd not say that the value is irrelevant, though. It's just not part of that test. (If you set it to some terrible value, I guarantee it would be relevant.) Maybe just say it's not needed in this test. > Unsafe conversion from long to int in MergeManagerImpl.unconditionalReserve() > - > > Key: MAPREDUCE-6724 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6724 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: mapreduce6724.001.patch, mapreduce6724.002.patch, > mapreduce6724.003.patch, mapreduce6724.004.patch > > > When shuffle is done in memory, MergeManagerImpl converts the requested size > to an int to allocate an instance of InMemoryMapOutput. This results in an > overflow if the requested size is bigger than Integer.MAX_VALUE and > eventually causes the reducer to fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6724) Unsafe conversion from long to int in MergeManagerImpl.unconditionalReserve()
[
https://issues.apache.org/jira/browse/MAPREDUCE-6724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15375783#comment-15375783
]
Daniel Templeton commented on MAPREDUCE-6724:
-
Thanks for the patch, [~haibochen]! A couple comments:
{code}
final long totalReduceMem = 8L * 1024 * 1024 * 1024;
final float shuffleInputBuf = 1.0f;
final float reduceInputBuf = 1.0f;
final MergeManagerImpl mgr = createMergeManager(
totalReduceMem, shuffleInputBuf, 0.95f, reduceInputBuf);
{code}
It would be better to be consistent in the parameters. Everything is a
variable except for the shuffle memory limit.
{code}
final long totalReduceMem = 1L;
final float shuffleInputBuf = 1.0f;
final float shuffleMemLimit =
singleShuffleLimitConfiged / (float) totalReduceMem;
final MergeManagerImpl mgr = createMergeManager(
totalReduceMem, shuffleInputBuf, shuffleMemLimit, 1.0f);
{code}
Same comment here, except with reducer input buffer.
> Unsafe conversion from long to int in MergeManagerImpl.unconditionalReserve()
> -
>
> Key: MAPREDUCE-6724
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6724
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Attachments: mapreduce6724.001.patch, mapreduce6724.002.patch,
> mapreduce6724.003.patch
>
>
> When shuffle is done in memory, MergeManagerImpl converts the requested size
> to an int to allocate an instance of InMemoryMapOutput. This results in an
> overflow if the requested size is bigger than Integer.MAX_VALUE and
> eventually causes the reducer to fail.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6725) Comment for CLI#listEvents() contains no-existent param
[ https://issues.apache.org/jira/browse/MAPREDUCE-6725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15348289#comment-15348289 ] Daniel Templeton commented on MAPREDUCE-6725: - [~ajisakaa], got time for a quickie? > Comment for CLI#listEvents() contains no-existent param > --- > > Key: MAPREDUCE-6725 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6725 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 2.6.0 > Environment: centos x64 > hadoop2.6.0 >Reporter: shenyinjie >Priority: Minor > Attachments: MAPREDUCE-6725.patch, MAPREDUCE-6725_1.patch > > > Comment for CLI#listEvents() contains @param jobid, but this method doesn't > has the parameter,it may be some jira changed the method and forgot to > change the comment > So I made a simple fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6725) Comment for CLI#listEvents() contains no-existing param
[ https://issues.apache.org/jira/browse/MAPREDUCE-6725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15347599#comment-15347599 ] Daniel Templeton commented on MAPREDUCE-6725: - Works for me. +1 (non-binding) > Comment for CLI#listEvents() contains no-existing param > --- > > Key: MAPREDUCE-6725 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6725 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 2.6.0 > Environment: centos x64 > hadoop2.6.0 >Reporter: shenyinjie >Priority: Minor > Attachments: MAPREDUCE-6725.patch, MAPREDUCE-6725_1.patch > > > Comment for CLI#listEvents() contains @param jobid, but this method doesn't > has the parameter,it may be some jira changed the method and forgot to > change the comment > So I made a simple fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6725) Comment for CLI#listEvents() contains no-existing param
[ https://issues.apache.org/jira/browse/MAPREDUCE-6725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15346452#comment-15346452 ] Daniel Templeton commented on MAPREDUCE-6725: - Thanks, [~shenyinjie]! Would you mind also adding the two missing parameters? > Comment for CLI#listEvents() contains no-existing param > --- > > Key: MAPREDUCE-6725 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6725 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 2.6.0 > Environment: centos x64 > hadoop2.6.0 >Reporter: shenyinjie >Priority: Minor > Attachments: MAPREDUCE-6725.patch > > > Comment for CLI#listEvents() contains @param jobid, but this method doesn't > has the parameter,it may be some jira changed the method and forgot to > change the comment > So I made a simple fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
