[jira] [Updated] (MAPREDUCE-5728) Check NPE for serializer/deserializer in MapTask
[
https://issues.apache.org/jira/browse/MAPREDUCE-5728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-5728:
-
Fix Version/s: (was: 2.6.0)
2.7.0
Check NPE for serializer/deserializer in MapTask
Key: MAPREDUCE-5728
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5728
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: client
Affects Versions: 2.2.0
Reporter: Jerry He
Assignee: Jerry He
Priority: Minor
Fix For: 2.4.0, 2.7.0
Attachments: MAPREDUCE-5728-trunk.patch
Currently we will get NPE if the serializer/deserializer is not configured
correctly.
{code}
14/01/14 11:52:35 INFO mapred.JobClient: Task Id :
attempt_201401072154_0027_m_02_2, Status : FAILED
java.lang.NullPointerException
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.init(MapTask.java:944)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.init(MapTask.java:672)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:740)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:368)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at
java.security.AccessController.doPrivileged(AccessController.java:362)
at javax.security.auth.Subject.doAs(Subject.java:573)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1502)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
{code}
serializationFactory.getSerializer and serializationFactory.getDeserializer
returns NULL in this case.
Let's check NPE for serializer/deserializer in MapTask so that we don't get
meaningless NPE.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6156) Fetcher - connect() doesn't handle connection refused correctly
[
https://issues.apache.org/jira/browse/MAPREDUCE-6156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14210145#comment-14210145
]
Arun C Murthy commented on MAPREDUCE-6156:
--
[~jlowe] Thanks. I've merged this into branch-2.6.0 also for hadoop-2.6.0-rc1.
Fetcher - connect() doesn't handle connection refused correctly
Key: MAPREDUCE-6156
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6156
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Sidharta Seethana
Assignee: Junping Du
Priority: Blocker
Fix For: 2.6.0
Attachments: MAPREDUCE-6156-v2.patch, MAPREDUCE-6156-v3.patch,
MAPREDUCE-6156.patch
The connect() function in the fetcher assumes that whenever an IOException is
thrown, the amount of time passed equals connectionTimeout ( see code
snippet below ). This is incorrect. For example, in case the NM is down, an
ConnectException is thrown immediately - and the catch block assumes a minute
has passed when it is not the case.
{code}
if (connectionTimeout 0) {
throw new IOException(Invalid timeout
+ [timeout = + connectionTimeout + ms]);
} else if (connectionTimeout 0) {
unit = Math.min(UNIT_CONNECT_TIMEOUT, connectionTimeout);
}
// set the connect timeout to the unit-connect-timeout
connection.setConnectTimeout(unit);
while (true) {
try {
connection.connect();
break;
} catch (IOException ioe) {
// update the total remaining connect-timeout
connectionTimeout -= unit;
// throw an exception if we have waited for timeout amount of time
// note that the updated value if timeout is used here
if (connectionTimeout == 0) {
throw ioe;
}
// reset the connect timeout for the last try
if (connectionTimeout unit) {
unit = connectionTimeout;
// reset the connect time out for the final connect
connection.setConnectTimeout(unit);
}
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6083) Map/Reduce dangerously adds Guava @Beta class to CryptoUtils
[ https://issues.apache.org/jira/browse/MAPREDUCE-6083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14198360#comment-14198360 ] Arun C Murthy commented on MAPREDUCE-6083: -- [~ctubbsii] - Apologies for the late response. Unfortunately, we can't change guava versions in 2.6 since it would be incompatible. Map/Reduce dangerously adds Guava @Beta class to CryptoUtils Key: MAPREDUCE-6083 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6083 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.6.0 Reporter: Christopher Tubbs Priority: Blocker Labels: beta, deprecated, guava Attachments: 0001-MAPREDUCE-6083-Avoid-client-use-of-deprecated-LimitI.patch See HDFS-7040 for more background/details. In recent 2.6.0-SNAPSHOTs, the use of LimitInputStream was added to CryptoUtils. This is part of the API components of Hadoop, which severely impacts users who were utilizing newer versions of Guava, where the @Beta and @Deprecated class, LimitInputStream, has been removed (removed in version 15 and later), beyond the impact already experienced in 2.4.0 as identified in HDFS-7040. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6083) Map/Reduce dangerously adds Guava @Beta class to CryptoUtils
[ https://issues.apache.org/jira/browse/MAPREDUCE-6083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-6083: - Status: Open (was: Patch Available) Yes, I'm cool with that. Tx! Map/Reduce dangerously adds Guava @Beta class to CryptoUtils Key: MAPREDUCE-6083 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6083 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.6.0 Reporter: Christopher Tubbs Priority: Blocker Labels: beta, deprecated, guava Attachments: 0001-MAPREDUCE-6083-Avoid-client-use-of-deprecated-LimitI.patch See HDFS-7040 for more background/details. In recent 2.6.0-SNAPSHOTs, the use of LimitInputStream was added to CryptoUtils. This is part of the API components of Hadoop, which severely impacts users who were utilizing newer versions of Guava, where the @Beta and @Deprecated class, LimitInputStream, has been removed (removed in version 15 and later), beyond the impact already experienced in 2.4.0 as identified in HDFS-7040. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-2841) Task level native optimization
[ https://issues.apache.org/jira/browse/MAPREDUCE-2841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14041754#comment-14041754 ] Arun C Murthy commented on MAPREDUCE-2841: -- I also noticed that the github has a bunch of code related to Pig, Hive etc. - I think we'd all agree that they need to be in respective projects eventually. Task level native optimization -- Key: MAPREDUCE-2841 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2841 Project: Hadoop Map/Reduce Issue Type: Improvement Components: task Environment: x86-64 Linux/Unix Reporter: Binglin Chang Assignee: Sean Zhong Attachments: DESIGN.html, MAPREDUCE-2841.v1.patch, MAPREDUCE-2841.v2.patch, dualpivot-0.patch, dualpivotv20-0.patch, fb-shuffle.patch I'm recently working on native optimization for MapTask based on JNI. The basic idea is that, add a NativeMapOutputCollector to handle k/v pairs emitted by mapper, therefore sort, spill, IFile serialization can all be done in native code, preliminary test(on Xeon E5410, jdk6u24) showed promising results: 1. Sort is about 3x-10x as fast as java(only binary string compare is supported) 2. IFile serialization speed is about 3x of java, about 500MB/s, if hardware CRC32C is used, things can get much faster(1G/ 3. Merge code is not completed yet, so the test use enough io.sort.mb to prevent mid-spill This leads to a total speed up of 2x~3x for the whole MapTask, if IdentityMapper(mapper does nothing) is used There are limitations of course, currently only Text and BytesWritable is supported, and I have not think through many things right now, such as how to support map side combine. I had some discussion with somebody familiar with hive, it seems that these limitations won't be much problem for Hive to benefit from those optimizations, at least. Advices or discussions about improving compatibility are most welcome:) Currently NativeMapOutputCollector has a static method called canEnable(), which checks if key/value type, comparator type, combiner are all compatible, then MapTask can choose to enable NativeMapOutputCollector. This is only a preliminary test, more work need to be done. I expect better final results, and I believe similar optimization can be adopt to reduce task and shuffle too. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-2841) Task level native optimization
[
https://issues.apache.org/jira/browse/MAPREDUCE-2841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14041752#comment-14041752
]
Arun C Murthy commented on MAPREDUCE-2841:
--
{quote}
If the MR developer community generally agrees this belongs in the core, I'd
like to start a feature branch for it in order to import the current code, sort
out the build/integration issues, and take care of the remaining items that
Sean mentioned above.
{quote}
[~tlipcon] Thanks for starting this discussion. I have a few thoughts I'd like
to run by you.
I think the eventual goal of this (looking at
https://github.com/intel-hadoop/nativetask/blob/master/README.md) is a
full-native runtime for MapReduce including sort, shuffle, merge etc.
Hence, it does look like we will achieve a compatible, but alternate
implementation of MapReduce runtime. Hence, this is similar to other alternate
runtimes for MapReduce such as Apache Tez.
Furthermore, this is implemented in C++ - which is, frankly, a concern for the
poor job C++ has done with ABI. I'm glad to see that it doesn't rely on boost -
the worst affender. This is the same reason the native Hadoop client
(HADOOP-10388) is being done purely in C. Also, the MR development community is
pre-dominantly Java, which is something to keep in mind. This is a big concern
for me.
In all, it seems to me we could consider having this not in Apache Hadoop, but
as an incubator project to develop a native, MR compatible runtime.
This will allow it to develop a like-minded community (C++ skills etc.) and not
be bogged down by *all* of Hadoop's requirements such as security (how/when
will this allow for secure shuffle or encrypted shuffle etc.), compatibility
with several OSes (flavours of Linux, MacOSX, Windows) etc. It will also allow
them to ship independently and get user feedback more quickly.
Similarly, I am wary of importing a nearly 75K LOC codebase into a stable
project and it's impact on our releases on breakage - particularly given the
difference in skills of the community i.e. Java v/s C++ etc.
What do you think Todd Sean? I'm more than happy to help with incubator
process if required.
Task level native optimization
--
Key: MAPREDUCE-2841
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2841
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: task
Environment: x86-64 Linux/Unix
Reporter: Binglin Chang
Assignee: Sean Zhong
Attachments: DESIGN.html, MAPREDUCE-2841.v1.patch,
MAPREDUCE-2841.v2.patch, dualpivot-0.patch, dualpivotv20-0.patch,
fb-shuffle.patch
I'm recently working on native optimization for MapTask based on JNI.
The basic idea is that, add a NativeMapOutputCollector to handle k/v pairs
emitted by mapper, therefore sort, spill, IFile serialization can all be done
in native code, preliminary test(on Xeon E5410, jdk6u24) showed promising
results:
1. Sort is about 3x-10x as fast as java(only binary string compare is
supported)
2. IFile serialization speed is about 3x of java, about 500MB/s, if hardware
CRC32C is used, things can get much faster(1G/
3. Merge code is not completed yet, so the test use enough io.sort.mb to
prevent mid-spill
This leads to a total speed up of 2x~3x for the whole MapTask, if
IdentityMapper(mapper does nothing) is used
There are limitations of course, currently only Text and BytesWritable is
supported, and I have not think through many things right now, such as how to
support map side combine. I had some discussion with somebody familiar with
hive, it seems that these limitations won't be much problem for Hive to
benefit from those optimizations, at least. Advices or discussions about
improving compatibility are most welcome:)
Currently NativeMapOutputCollector has a static method called canEnable(),
which checks if key/value type, comparator type, combiner are all compatible,
then MapTask can choose to enable NativeMapOutputCollector.
This is only a preliminary test, more work need to be done. I expect better
final results, and I believe similar optimization can be adopt to reduce task
and shuffle too.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-2841) Task level native optimization
[ https://issues.apache.org/jira/browse/MAPREDUCE-2841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14041757#comment-14041757 ] Arun C Murthy commented on MAPREDUCE-2841: -- On related thought to Pig/Hive etc. - I see Hadoop MapReduce fading away fast particularly since projects using MR such as Pig, Hive, Cascading etc. re-vector on other projects like Apache Tez or Apache Spark. For e.g. # Hive-on-Tez (https://issues.apache.org/jira/browse/HIVE-4660) - The hive community has already moved it's major investments away from MR to Tez. # Pig-on-Tez (https://issues.apache.org/jira/browse/PIG-3446) - The pig community is very close to shipping this in pig-0.14 and again is investing heavily on Tez. Given that, Sean/Todd, would it be useful to discuss contributing this to Tez instead? This way the work here would continue to stay relevant in the context of the majority users of MapReduce who use Pig, Hive, Cascading etc. Of course, I'm sure another option is Apache Spark, but given that Tez is much more closer (code-base wise) to MR, it would be much easier to contribute to Tez. Happy to help if that makes sense too. Thanks. Task level native optimization -- Key: MAPREDUCE-2841 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2841 Project: Hadoop Map/Reduce Issue Type: Improvement Components: task Environment: x86-64 Linux/Unix Reporter: Binglin Chang Assignee: Sean Zhong Attachments: DESIGN.html, MAPREDUCE-2841.v1.patch, MAPREDUCE-2841.v2.patch, dualpivot-0.patch, dualpivotv20-0.patch, fb-shuffle.patch I'm recently working on native optimization for MapTask based on JNI. The basic idea is that, add a NativeMapOutputCollector to handle k/v pairs emitted by mapper, therefore sort, spill, IFile serialization can all be done in native code, preliminary test(on Xeon E5410, jdk6u24) showed promising results: 1. Sort is about 3x-10x as fast as java(only binary string compare is supported) 2. IFile serialization speed is about 3x of java, about 500MB/s, if hardware CRC32C is used, things can get much faster(1G/ 3. Merge code is not completed yet, so the test use enough io.sort.mb to prevent mid-spill This leads to a total speed up of 2x~3x for the whole MapTask, if IdentityMapper(mapper does nothing) is used There are limitations of course, currently only Text and BytesWritable is supported, and I have not think through many things right now, such as how to support map side combine. I had some discussion with somebody familiar with hive, it seems that these limitations won't be much problem for Hive to benefit from those optimizations, at least. Advices or discussions about improving compatibility are most welcome:) Currently NativeMapOutputCollector has a static method called canEnable(), which checks if key/value type, comparator type, combiner are all compatible, then MapTask can choose to enable NativeMapOutputCollector. This is only a preliminary test, more work need to be done. I expect better final results, and I believe similar optimization can be adopt to reduce task and shuffle too. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-2841) Task level native optimization
[
https://issues.apache.org/jira/browse/MAPREDUCE-2841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14041816#comment-14041816
]
Arun C Murthy commented on MAPREDUCE-2841:
--
Todd,
bq. I agree that building a completely parallel C++ MR runtime is a much larger
project that should not be part of Hadoop.
I'm confused. There already exists large amounts of code on the github for a
the full task runtime. Is that abandoned? Are you saying there no intention to
contribute that to Hadoop, ever? Why would that be? Would that be a separate
project?
With or without ABI, C++ still is a major problem w.r.t different compiler
versions, different platforms we support etc. That is precisely why
HADOOP-10388 chose to use pure-C only. A similar switch makes me *much* more
comfortable, aside from the disparity in skills in the Hadoop community.
Furthermore, there are considerably more security issues which open up in C++
land such as buffer overflow etc.
bq. I think the 75k you're counting may include the auto-generated shell
scripts.
From the github:
{noformat}
$ find . -name *.java | xargs wc -l
11988 total
$ find . -name *.h | xargs wc -l
27269 total
$ find . -name *.cc | xargs wc -l
26276 total
{noformat}
Whether it's test or non-test, we are still importing a *lot* of code - code
for which the Hadoop community does need to maintain?
bq. So, it's not a tiny import by any means, but for 2x improvement on terasort
wallclock, my opinion is that the maintenance burden is worth it.
Todd, as we both know, there are many, many ways to get 2x improvement on
terasort...
... nor is it worth a lot in real-world outside of benchmarks.
I'm sure we both would take 2x on Pig/Hive anyday... *smile*
bq. As for importing to Tez, I don't think the community has generally agreed
to EOL MapReduce
Regardless of whether or not we pull this into MR, it would be useful to pull
it into Tez too - if Sean wants to do it. Let's not discourage them.
I'm sure we both agree, and want to see real world workloads improve and that
Hive/Pig/Cascading etc. represent that.
IAC, hopefully we can stop this meme that I'm trying to *preclude* you from
doing anything regardless of my religious beliefs. IAC, we both realize MR is
reasonably stable and won't get a lot of investment, and so do our employers:
http://vision.cloudera.com/mapreduce-spark/
http://hortonworks.com/hadoop/tez/
So, I'd appreciate if we don't misinterpret each others' technical opinions and
concerns during this discussion. Thanks.
FTR: I'll restate my concerns about C++, roadmap for C++ runtime,
maintainability, support for all of Hadoop (security, platforms etc.).
Furthermore, this jira was opened nearly 3 years ago and only has sporadic
bursts of activity - not a good sign for long-term maintainability.
I've stated my concerns, let's try get through them by focussing on those
aspects.
Finally, what is the concern you see with starting this as an incubator project
and allowing folks to develop a community around it? We can certainly help on
our end by making it easy for them to plug in via interfaces etc.
Thanks.
Task level native optimization
--
Key: MAPREDUCE-2841
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2841
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: task
Environment: x86-64 Linux/Unix
Reporter: Binglin Chang
Assignee: Sean Zhong
Attachments: DESIGN.html, MAPREDUCE-2841.v1.patch,
MAPREDUCE-2841.v2.patch, dualpivot-0.patch, dualpivotv20-0.patch,
fb-shuffle.patch
I'm recently working on native optimization for MapTask based on JNI.
The basic idea is that, add a NativeMapOutputCollector to handle k/v pairs
emitted by mapper, therefore sort, spill, IFile serialization can all be done
in native code, preliminary test(on Xeon E5410, jdk6u24) showed promising
results:
1. Sort is about 3x-10x as fast as java(only binary string compare is
supported)
2. IFile serialization speed is about 3x of java, about 500MB/s, if hardware
CRC32C is used, things can get much faster(1G/
3. Merge code is not completed yet, so the test use enough io.sort.mb to
prevent mid-spill
This leads to a total speed up of 2x~3x for the whole MapTask, if
IdentityMapper(mapper does nothing) is used
There are limitations of course, currently only Text and BytesWritable is
supported, and I have not think through many things right now, such as how to
support map side combine. I had some discussion with somebody familiar with
hive, it seems that these limitations won't be much problem for Hive to
benefit from those optimizations, at least. Advices or discussions about
improving compatibility are most welcome:)
Currently NativeMapOutputCollector has a
[jira] [Comment Edited] (MAPREDUCE-2841) Task level native optimization
[
https://issues.apache.org/jira/browse/MAPREDUCE-2841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14041816#comment-14041816
]
Arun C Murthy edited comment on MAPREDUCE-2841 at 6/24/14 8:30 AM:
---
Todd,
bq. I agree that building a completely parallel C++ MR runtime is a much larger
project that should not be part of Hadoop.
I'm confused. There already exists large amounts of code on the github for a
the full task runtime. Is that abandoned? Are you saying there no intention to
contribute that to Hadoop, ever? Why would that be? Would that be a separate
project?
With or without ABI, C++ still is a major problem w.r.t different compiler
versions, different platforms we support etc. That is precisely why
HADOOP-10388 chose to use pure-C only. A similar switch makes me *much* more
comfortable, aside from the disparity in skills in the Hadoop community.
Furthermore, there are considerably more security issues which open up in C++
land such as buffer overflow etc.
bq. I think the 75k you're counting may include the auto-generated shell
scripts.
From the github:
{noformat}
$ find . -name *.java | xargs wc -l
11988 total
$ find . -name *.h | xargs wc -l
27269 total
$ find . -name *.cc | xargs wc -l
26276 total
{noformat}
Whether it's test or non-test, we are still importing a *lot* of code - code
for which the Hadoop community does need to maintain?
bq. So, it's not a tiny import by any means, but for 2x improvement on terasort
wallclock, my opinion is that the maintenance burden is worth it.
Todd, as we both know, there are many, many ways to get 2x improvement on
terasort...
... nor is it worth a lot in real-world outside of benchmarks.
I'm sure we both would take 2x on Pig/Hive anyday... *smile*
bq. As for importing to Tez, I don't think the community has generally agreed
to EOL MapReduce
Regardless of whether or not we pull this into MR, it would be useful to pull
it into Tez too - if Sean wants to do it. Let's not discourage them.
I'm sure we both agree, and want to see real world workloads improve and that
Hive/Pig/Cascading etc. represent that.
IAC, hopefully we can stop this meme that I'm trying to *preclude* you from
doing anything regardless of my beliefs. IAC, we both realize MR is reasonably
stable and won't get a lot of investment, and so do our employers:
http://vision.cloudera.com/mapreduce-spark/
http://hortonworks.com/hadoop/tez/
Essentially, you asked for feedback from the MapReduce community; and this is
my honest feedback - as someone who has actively helped maintain this codebase
for more than 8 years now. So, I'd appreciate if we don't misinterpret each
others' technical opinions and concerns during this discussion. Thanks in
advance.
FTR: I'll restate my concerns about C++, roadmap for C++ runtime,
maintainability, support for all of Hadoop (new security bugs, future security
features, platforms etc.).
Furthermore, this jira was opened nearly 3 years ago and only has sporadic
bursts of activity - not a good sign for long-term maintainability.
I've stated my concerns, let's try get through them by focussing on those
aspects.
Finally, what is the concern you see with starting this as an incubator project
and allowing folks to develop a community around it? We can certainly help on
our end by making it easy for them to plug in via interfaces etc.
Thanks.
was (Author: acmurthy):
Todd,
bq. I agree that building a completely parallel C++ MR runtime is a much larger
project that should not be part of Hadoop.
I'm confused. There already exists large amounts of code on the github for a
the full task runtime. Is that abandoned? Are you saying there no intention to
contribute that to Hadoop, ever? Why would that be? Would that be a separate
project?
With or without ABI, C++ still is a major problem w.r.t different compiler
versions, different platforms we support etc. That is precisely why
HADOOP-10388 chose to use pure-C only. A similar switch makes me *much* more
comfortable, aside from the disparity in skills in the Hadoop community.
Furthermore, there are considerably more security issues which open up in C++
land such as buffer overflow etc.
bq. I think the 75k you're counting may include the auto-generated shell
scripts.
From the github:
{noformat}
$ find . -name *.java | xargs wc -l
11988 total
$ find . -name *.h | xargs wc -l
27269 total
$ find . -name *.cc | xargs wc -l
26276 total
{noformat}
Whether it's test or non-test, we are still importing a *lot* of code - code
for which the Hadoop community does need to maintain?
bq. So, it's not a tiny import by any means, but for 2x improvement on terasort
wallclock, my opinion is that the maintenance burden is worth it.
Todd, as we both know, there are many, many ways to get 2x improvement on
[jira] [Commented] (MAPREDUCE-5831) Old MR client is not compatible with new MR application
[
https://issues.apache.org/jira/browse/MAPREDUCE-5831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14039136#comment-14039136
]
Arun C Murthy commented on MAPREDUCE-5831:
--
I think the real way, going fwd, to fix this is to ensure the MR client and the
application are on the same version - MAPREDUCE-4421 should solve that; and be
the default deployment model.
Old MR client is not compatible with new MR application
---
Key: MAPREDUCE-5831
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5831
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client, mr-am
Affects Versions: 2.2.0, 2.3.0
Reporter: Zhijie Shen
Assignee: Tan, Wangda
Priority: Critical
Recently, we saw the following scenario:
1. The user setup a cluster of hadoop 2.3., which contains YARN 2.3 and MR
2.3.
2. The user client on a machine that MR 2.2 is installed and in the classpath.
Then, when the user submitted a simple wordcount job, he saw the following
message:
{code}
16:00:41,027 INFO main mapreduce.Job:1345 - map 100% reduce 100%
16:00:41,036 INFO main mapreduce.Job:1356 - Job job_1396468045458_0006
completed successfully
16:02:20,535 WARN main mapreduce.JobRunner:212 - Cannot start job
[wordcountJob]
java.lang.IllegalArgumentException: No enum constant
org.apache.hadoop.mapreduce.JobCounter.MB_MILLIS_REDUCES
at java.lang.Enum.valueOf(Enum.java:236)
at
org.apache.hadoop.mapreduce.counters.FrameworkCounterGroup.valueOf(FrameworkCounterGroup.java:148)
at
org.apache.hadoop.mapreduce.counters.FrameworkCounterGroup.findCounter(FrameworkCounterGroup.java:182)
at
org.apache.hadoop.mapreduce.counters.AbstractCounters.findCounter(AbstractCounters.java:154)
at
org.apache.hadoop.mapreduce.TypeConverter.fromYarn(TypeConverter.java:240)
at
org.apache.hadoop.mapred.ClientServiceDelegate.getJobCounters(ClientServiceDelegate.java:370)
at
org.apache.hadoop.mapred.YARNRunner.getJobCounters(YARNRunner.java:511)
at org.apache.hadoop.mapreduce.Job$7.run(Job.java:756)
at org.apache.hadoop.mapreduce.Job$7.run(Job.java:753)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapreduce.Job.getCounters(Job.java:753)
at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1361)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1289)
. . .
{code}
The problem is that the wordcount job was running on one or more than one
nodes of the YARN cluster, where MR 2.3 libs were installed, and
JobCounter.MB_MILLIS_REDUCES is available in the counters. On the other side,
due to the classpath setting, the client was likely to run with MR 2.2 libs.
After the client retrieved the counters from MR AM, it tried to construct the
Counter object with the received counter name. Unfortunately, the enum didn't
exist in the client's classpath. Therefore, No enum constant exception is
thrown here.
JobCounter.MB_MILLIS_REDUCES is brought to MR2 via MAPREDUCE-5464 since
Hadoop 2.3.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-5728) Check NPE for serializer/deserializer in MapTask
[
https://issues.apache.org/jira/browse/MAPREDUCE-5728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-5728:
-
Fix Version/s: (was: 2.2.1)
2.5.0
Check NPE for serializer/deserializer in MapTask
Key: MAPREDUCE-5728
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5728
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: client
Affects Versions: 2.2.0
Reporter: Jerry He
Assignee: Jerry He
Priority: Minor
Fix For: 2.4.0, 2.5.0
Attachments: MAPREDUCE-5728-trunk.patch
Currently we will get NPE if the serializer/deserializer is not configured
correctly.
{code}
14/01/14 11:52:35 INFO mapred.JobClient: Task Id :
attempt_201401072154_0027_m_02_2, Status : FAILED
java.lang.NullPointerException
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.init(MapTask.java:944)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.init(MapTask.java:672)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:740)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:368)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at
java.security.AccessController.doPrivileged(AccessController.java:362)
at javax.security.auth.Subject.doAs(Subject.java:573)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1502)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
{code}
serializationFactory.getSerializer and serializationFactory.getDeserializer
returns NULL in this case.
Let's check NPE for serializer/deserializer in MapTask so that we don't get
meaningless NPE.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-5815) Fix NPE in TestMRAppMaster
[ https://issues.apache.org/jira/browse/MAPREDUCE-5815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-5815: - Priority: Blocker (was: Major) Fix NPE in TestMRAppMaster -- Key: MAPREDUCE-5815 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5815 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, mrv2 Reporter: Gera Shegalov Assignee: Gera Shegalov Priority: Blocker Attachments: MAPREDUCE-5815.v01.patch Working MAPREDUCE-5813 I stumbled on NPE's in TestMRAppMaster. They seem to be introduced by MAPREDUCE-5805. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5815) Fix NPE in TestMRAppMaster
[ https://issues.apache.org/jira/browse/MAPREDUCE-5815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13954377#comment-13954377 ] Arun C Murthy commented on MAPREDUCE-5815: -- [~jira.shegalov] - I'm not sure we should quietly ignore the queue-name... maybe the right fix is to ensure TestMRAppMaster actually sets a proper queue name? [~vinodkv] - Thoughts? Fix NPE in TestMRAppMaster -- Key: MAPREDUCE-5815 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5815 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, mrv2 Reporter: Gera Shegalov Assignee: Gera Shegalov Priority: Blocker Attachments: MAPREDUCE-5815.v01.patch Working MAPREDUCE-5813 I stumbled on NPE's in TestMRAppMaster. They seem to be introduced by MAPREDUCE-5805. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5815) Fix NPE in TestMRAppMaster
[ https://issues.apache.org/jira/browse/MAPREDUCE-5815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-5815: - Status: Open (was: Patch Available) Fix NPE in TestMRAppMaster -- Key: MAPREDUCE-5815 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5815 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, mrv2 Reporter: Gera Shegalov Assignee: Gera Shegalov Priority: Blocker Attachments: MAPREDUCE-5815.v01.patch Working MAPREDUCE-5813 I stumbled on NPE's in TestMRAppMaster. They seem to be introduced by MAPREDUCE-5805. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5813) YarnChild does not load job.xml with mapreduce.job.classloader=true
[
https://issues.apache.org/jira/browse/MAPREDUCE-5813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-5813:
-
Resolution: Fixed
Fix Version/s: 2.4.0
Status: Resolved (was: Patch Available)
I just committed this. Thanks [~jira.shegalov]!
YarnChild does not load job.xml with mapreduce.job.classloader=true
Key: MAPREDUCE-5813
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5813
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2, task
Affects Versions: 2.3.0
Reporter: Gera Shegalov
Assignee: Gera Shegalov
Priority: Blocker
Fix For: 2.4.0
Attachments: MAPREDUCE-5813.v01.patch, MAPREDUCE-5813.v02.patch
{{YarnChild.main}} uses {{JobConf.addResource(String)}} to load {{job.xml}}
that relies on class loading. When {{mapreduce.job.classloader=true}} the
job-speicific part of the class path is separated from {{CLASSPATH}} into
{{APP_CLASSPATH}}. Therefore {{job.xml}} is inaccessible for the default
class loader. Later {{writeLocalJobFile}} overwrites the correct localized
{{job.xml}} on disk as well.
This problem is easily avoided by using {{JobConf.addResource(Path)}} to
read the localized {{job.xml}} without relying on class loading.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-5813) YarnChild does not load job.xml with mapreduce.job.classloader=true
[
https://issues.apache.org/jira/browse/MAPREDUCE-5813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-5813:
-
Status: Open (was: Patch Available)
Agree w/ [~sjlee0].
[~jira.shegalov] - mind updating the patch? Tx.
YarnChild does not load job.xml with mapreduce.job.classloader=true
Key: MAPREDUCE-5813
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5813
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2, task
Affects Versions: 2.3.0
Reporter: Gera Shegalov
Assignee: Gera Shegalov
Priority: Blocker
Attachments: MAPREDUCE-5813.v01.patch
{{YarnChild.main}} uses {{JobConf.addResource(String)}} to load {{job.xml}}
that relies on class loading. When {{mapreduce.job.classloader=true}} the
job-speicific part of the class path is separated from {{CLASSPATH}} into
{{APP_CLASSPATH}}. Therefore {{job.xml}} is inaccessible for the default
class loader. Later {{writeLocalJobFile}} overwrites the correct localized
{{job.xml}} on disk as well.
This problem is easily avoided by using {{JobConf.addResource(Path)}} to
read the localized {{job.xml}} without relying on class loading.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-5553) Add task state filters on Application/MRJob page for MR Application master
[ https://issues.apache.org/jira/browse/MAPREDUCE-5553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-5553: - Assignee: Paul Han Add task state filters on Application/MRJob page for MR Application master --- Key: MAPREDUCE-5553 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5553 Project: Hadoop Map/Reduce Issue Type: Improvement Components: applicationmaster Affects Versions: 2.0.5-alpha Reporter: Paul Han Assignee: Paul Han Priority: Minor Attachments: FilteredTaskPage.png, MAPREDUCE-5553.patch, MRJobWithTaskStateFilter.png, MRJobWithoutTaskStateFilters.png On Job page of MR application master, the task attempts have a nice breakdown of different state: running, failed But for map/reduce task level, there's only one links to all tasks without breakdown for different states: pending, running, complete, etc. With Hadoop 1, the similar functionality exists: user can go thru jobtracker- job- map/reduce tasks - attempts. This ticket will improve the usability to allow user to go job- total, pending, running, complete Map/Reduce task. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5553) Add task state filters on Application/MRJob page for MR Application master
[ https://issues.apache.org/jira/browse/MAPREDUCE-5553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13932480#comment-13932480 ] Arun C Murthy commented on MAPREDUCE-5553: -- I just committed this. Apologies for taking so long to get back to you - thanks [~paulhan]! Add task state filters on Application/MRJob page for MR Application master --- Key: MAPREDUCE-5553 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5553 Project: Hadoop Map/Reduce Issue Type: Improvement Components: applicationmaster Affects Versions: 2.0.5-alpha Reporter: Paul Han Assignee: Paul Han Priority: Minor Fix For: 2.4.0 Attachments: FilteredTaskPage.png, MAPREDUCE-5553.patch, MRJobWithTaskStateFilter.png, MRJobWithoutTaskStateFilters.png On Job page of MR application master, the task attempts have a nice breakdown of different state: running, failed But for map/reduce task level, there's only one links to all tasks without breakdown for different states: pending, running, complete, etc. With Hadoop 1, the similar functionality exists: user can go thru jobtracker- job- map/reduce tasks - attempts. This ticket will improve the usability to allow user to go job- total, pending, running, complete Map/Reduce task. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5553) Add task state filters on Application/MRJob page for MR Application master
[ https://issues.apache.org/jira/browse/MAPREDUCE-5553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-5553: - Resolution: Fixed Fix Version/s: 2.4.0 Status: Resolved (was: Patch Available) Add task state filters on Application/MRJob page for MR Application master --- Key: MAPREDUCE-5553 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5553 Project: Hadoop Map/Reduce Issue Type: Improvement Components: applicationmaster Affects Versions: 2.0.5-alpha Reporter: Paul Han Assignee: Paul Han Priority: Minor Fix For: 2.4.0 Attachments: FilteredTaskPage.png, MAPREDUCE-5553.patch, MRJobWithTaskStateFilter.png, MRJobWithoutTaskStateFilters.png On Job page of MR application master, the task attempts have a nice breakdown of different state: running, failed But for map/reduce task level, there's only one links to all tasks without breakdown for different states: pending, running, complete, etc. With Hadoop 1, the similar functionality exists: user can go thru jobtracker- job- map/reduce tasks - attempts. This ticket will improve the usability to allow user to go job- total, pending, running, complete Map/Reduce task. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-4868) Allow multiple iteration for map
[ https://issues.apache.org/jira/browse/MAPREDUCE-4868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-4868: - Fix Version/s: (was: 2.3.0) 2.4.0 Allow multiple iteration for map Key: MAPREDUCE-4868 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4868 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 3.0.0, 2.0.3-alpha Reporter: Jerry Chen Fix For: 3.0.0, 2.4.0 Original Estimate: 168h Remaining Estimate: 168h Currently, the Mapper class allows advanced users to override public void run(Context context) method for more control over the execution of the mapper, while Context interface limit the operations over the data which is the foundation of more control. One of use cases is that when I am considering a hive optimziation problem, I want to go two passes over the input data instead of using a another job or task ( which may slower the whole process). Each pass do the same thing but with a different parameters. This is a new paradigm of Map Reduce usage and can be archived easily by extend Context interface a little with the more control over the data such as reset the input. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (MAPREDUCE-5728) Check NPE for serializer/deserializer in MapTask
[
https://issues.apache.org/jira/browse/MAPREDUCE-5728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-5728:
-
Fix Version/s: (was: 2.3.0)
2.4.0
Check NPE for serializer/deserializer in MapTask
Key: MAPREDUCE-5728
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5728
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: client
Affects Versions: 2.2.0
Reporter: Jerry He
Assignee: Jerry He
Priority: Minor
Fix For: 2.2.1, 2.4.0
Attachments: MAPREDUCE-5728-trunk.patch
Currently we will get NPE if the serializer/deserializer is not configured
correctly.
{code}
14/01/14 11:52:35 INFO mapred.JobClient: Task Id :
attempt_201401072154_0027_m_02_2, Status : FAILED
java.lang.NullPointerException
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.init(MapTask.java:944)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.init(MapTask.java:672)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:740)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:368)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at
java.security.AccessController.doPrivileged(AccessController.java:362)
at javax.security.auth.Subject.doAs(Subject.java:573)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1502)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
{code}
serializationFactory.getSerializer and serializationFactory.getDeserializer
returns NULL in this case.
Let's check NPE for serializer/deserializer in MapTask so that we don't get
meaningless NPE.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (MAPREDUCE-4462) Enhance readability of TestFairScheduler.java
[ https://issues.apache.org/jira/browse/MAPREDUCE-4462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-4462: - Fix Version/s: (was: 2.3.0) 2.4.0 Enhance readability of TestFairScheduler.java - Key: MAPREDUCE-4462 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4462 Project: Hadoop Map/Reduce Issue Type: Bug Components: scheduler, test Reporter: Ryan Hennig Priority: Minor Labels: comments, test Fix For: 2.4.0 Attachments: MAPREDUCE-4462.patch Original Estimate: 2h Remaining Estimate: 2h While reading over the unit tests for the Fair Scheduler introduced by MAPREDUCE-3451, I added comments to make the logic of the test easier to grok quickly. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (MAPREDUCE-5559) Reconsidering the policy of ignoring the blacklist after reaching the threshold
[ https://issues.apache.org/jira/browse/MAPREDUCE-5559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-5559: - Fix Version/s: (was: 2.3.0) 2.4.0 Reconsidering the policy of ignoring the blacklist after reaching the threshold --- Key: MAPREDUCE-5559 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5559 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.1.1-beta Reporter: Zhijie Shen Assignee: Zhijie Shen Fix For: 2.4.0 Nowadays, when MR AM find the number of blacklisted nodes reaches one threshold, the blacklist will be totally ignored. The newly assigned containers on the blacklisted nodes will be allocated. This may be not the best practice. We need to reconsider of it. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (MAPREDUCE-5028) Maps fail when io.sort.mb is set to high value
[
https://issues.apache.org/jira/browse/MAPREDUCE-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-5028:
-
Fix Version/s: (was: 2.3.0)
2.4.0
Maps fail when io.sort.mb is set to high value
--
Key: MAPREDUCE-5028
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5028
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 1.1.1, 2.0.3-alpha, 0.23.5
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Priority: Critical
Fix For: 1.2.0, 2.4.0
Attachments: MR-5028_testapp.patch, mr-5028-branch1.patch,
mr-5028-branch1.patch, mr-5028-branch1.patch, mr-5028-trunk.patch,
mr-5028-trunk.patch, mr-5028-trunk.patch, repro-mr-5028.patch
Verified the problem exists on branch-1 with the following configuration:
Pseudo-dist mode: 2 maps/ 1 reduce, mapred.child.java.opts=-Xmx2048m,
io.sort.mb=1280, dfs.block.size=2147483648
Run teragen to generate 4 GB data
Maps fail when you run wordcount on this configuration with the following
error:
{noformat}
java.io.IOException: Spill failed
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1031)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:692)
at
org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
at
org.apache.hadoop.examples.WordCount$TokenizerMapper.map(WordCount.java:45)
at
org.apache.hadoop.examples.WordCount$TokenizerMapper.map(WordCount.java:34)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:766)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:375)
at org.apache.hadoop.io.IntWritable.readFields(IntWritable.java:38)
at
org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:67)
at
org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40)
at
org.apache.hadoop.mapreduce.ReduceContext.nextKeyValue(ReduceContext.java:116)
at
org.apache.hadoop.mapreduce.ReduceContext.nextKey(ReduceContext.java:92)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:175)
at
org.apache.hadoop.mapred.Task$NewCombinerRunner.combine(Task.java:1505)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1438)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$1800(MapTask.java:855)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:1346)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (MAPREDUCE-5036) Default shuffle handler port should not be 8080
[ https://issues.apache.org/jira/browse/MAPREDUCE-5036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-5036: - Fix Version/s: (was: 2.3.0) 2.4.0 Default shuffle handler port should not be 8080 --- Key: MAPREDUCE-5036 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5036 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 2.4.0 Attachments: MAPREDUCE-5036-13562.patch, MAPREDUCE-5036-2.patch, MAPREDUCE-5036.patch The shuffle handler port (mapreduce.shuffle.port) defaults to 8080. This is a pretty common port for web services, and is likely to cause unnecessary port conflicts. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (MAPREDUCE-4253) Tests for mapreduce-client-core are lying under mapreduce-client-jobclient
[ https://issues.apache.org/jira/browse/MAPREDUCE-4253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-4253: - Fix Version/s: (was: 2.3.0) 2.4.0 Tests for mapreduce-client-core are lying under mapreduce-client-jobclient -- Key: MAPREDUCE-4253 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4253 Project: Hadoop Map/Reduce Issue Type: Task Components: client Affects Versions: 2.0.0-alpha Reporter: Harsh J Assignee: Tsuyoshi OZAWA Fix For: 2.4.0 Attachments: MR-4253.1.patch, MR-4253.2.patch, crossing_project_checker.rb, result.txt Many of the tests for client libs from mapreduce-client-core are lying under mapreduce-client-jobclient. We should investigate if this is the right thing to do and if not, move the tests back into client-core. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (MAPREDUCE-4468) Encapsulate FairScheduler preemption logic into helper class
[ https://issues.apache.org/jira/browse/MAPREDUCE-4468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-4468: - Fix Version/s: (was: 2.3.0) 2.4.0 Encapsulate FairScheduler preemption logic into helper class Key: MAPREDUCE-4468 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4468 Project: Hadoop Map/Reduce Issue Type: Improvement Components: scheduler Reporter: Ryan Hennig Priority: Minor Labels: refactoring, scheduler Fix For: 2.4.0 Attachments: MAPREDUCE-4468.patch Original Estimate: 4h Remaining Estimate: 4h I've extracted the preemption logic from the Fair Scheduler into a helper class so that FairScheduler is closer to following the Single Responsibility Principle. This may eventually evolve into a generalized preemption module which could be leveraged by other schedulers. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (MAPREDUCE-5663) Add an interface to Input/Ouput Formats to obtain delegation tokens
[ https://issues.apache.org/jira/browse/MAPREDUCE-5663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-5663: - Status: Open (was: Patch Available) +1, mostly lgtm. One nit: I'd like to change the api to 'obtainCredentials' rather than 'addCredentials'. That threw me off a little. Last thing - a more descriptive javadoc would help too. Thanks. Add an interface to Input/Ouput Formats to obtain delegation tokens --- Key: MAPREDUCE-5663 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5663 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Siddharth Seth Assignee: Michael Weng Attachments: MAPREDUCE-5663.4.txt, MAPREDUCE-5663.5.txt, MAPREDUCE-5663.patch.txt, MAPREDUCE-5663.patch.txt2, MAPREDUCE-5663.patch.txt3 Currently, delegation tokens are obtained as part of the getSplits / checkOutputSpecs calls to the InputFormat / OutputFormat respectively. This works as long as the splits are generated on a node with kerberos credentials. For split generation elsewhere (AM for example), an explicit interface is required. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Issue Comment Deleted] (MAPREDUCE-5541) Improved algorithm for whether need speculative task
[ https://issues.apache.org/jira/browse/MAPREDUCE-5541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-5541: - Comment: was deleted (was: Committed to hadoop-longwing/condor-branch-1.) Improved algorithm for whether need speculative task Key: MAPREDUCE-5541 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5541 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv1 Affects Versions: 1.2.1 Reporter: zhaoyunjiong Assignee: zhaoyunjiong Fix For: 1.2.2 Attachments: MAPREDUCE-5541-branch-1.2.patch, MAPREDUCE-5541-branch-1.2.patch Most of time, tasks won't start running at same time. In this case hasSpeculativeTask in TaskInProgress not working very well. Some times, some tasks just start running, and scheduler already decide it need speculative task to run. And this waste a lot of resource. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5541) Improved algorithm for whether need speculative task
[ https://issues.apache.org/jira/browse/MAPREDUCE-5541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13801360#comment-13801360 ] Arun C Murthy commented on MAPREDUCE-5541: -- [~zhaoyunjiong] The patch looks good. Can you pls share your experience running with this patch? What kind of tests have you done? Tx. Improved algorithm for whether need speculative task Key: MAPREDUCE-5541 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5541 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv1 Affects Versions: 1.2.1 Reporter: zhaoyunjiong Assignee: zhaoyunjiong Fix For: 1.2.2 Attachments: MAPREDUCE-5541-branch-1.2.patch, MAPREDUCE-5541-branch-1.2.patch Most of time, tasks won't start running at same time. In this case hasSpeculativeTask in TaskInProgress not working very well. Some times, some tasks just start running, and scheduler already decide it need speculative task to run. And this waste a lot of resource. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5541) Improved algorithm for whether need speculative task
[ https://issues.apache.org/jira/browse/MAPREDUCE-5541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13801488#comment-13801488 ] Arun C Murthy commented on MAPREDUCE-5541: -- Thanks. Meanwhile, can you please do a full run of 'ant test' and report results? I'm almost sure this will break some unit tests. Thanks. Improved algorithm for whether need speculative task Key: MAPREDUCE-5541 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5541 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv1 Affects Versions: 1.2.1 Reporter: zhaoyunjiong Assignee: zhaoyunjiong Fix For: 1.2.2 Attachments: MAPREDUCE-5541-branch-1.2.patch, MAPREDUCE-5541-branch-1.2.patch Most of time, tasks won't start running at same time. In this case hasSpeculativeTask in TaskInProgress not working very well. Some times, some tasks just start running, and scheduler already decide it need speculative task to run. And this waste a lot of resource. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5583) Ability to limit running map and reduce tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-5583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13795429#comment-13795429 ] Arun C Murthy commented on MAPREDUCE-5583: -- Looks like we are rehashing the discussion on HADOOP-5170. Ability to limit running map and reduce tasks - Key: MAPREDUCE-5583 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5583 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mr-am, mrv2 Affects Versions: 0.23.9, 2.1.1-beta Reporter: Jason Lowe It would be nice if users could specify a limit to the number of map or reduce tasks that are running simultaneously. Occasionally users are performing operations in tasks that can lead to DDoS scenarios if too many tasks run simultaneously (e.g.: accessing a database, web service, etc.). Having the ability to throttle the number of tasks simultaneously running would provide users a way to mitigate issues with too many tasks on a large cluster attempting to access a serivce at any one time. This is similar to the functionality requested by MAPREDUCE-224 and implemented by HADOOP-3412 but was dropped in mrv2. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5583) Ability to limit running map and reduce tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-5583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13795427#comment-13795427 ] Arun C Murthy commented on MAPREDUCE-5583: -- I'm very concerned about allowing this sort of control to the end user without checks/balances. If every one does the same I want max of n tasks, all resources in the cluster can deadlock. This is one of the main reasons we haven't supported this. I'm still against it. Ability to limit running map and reduce tasks - Key: MAPREDUCE-5583 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5583 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mr-am, mrv2 Affects Versions: 0.23.9, 2.1.1-beta Reporter: Jason Lowe It would be nice if users could specify a limit to the number of map or reduce tasks that are running simultaneously. Occasionally users are performing operations in tasks that can lead to DDoS scenarios if too many tasks run simultaneously (e.g.: accessing a database, web service, etc.). Having the ability to throttle the number of tasks simultaneously running would provide users a way to mitigate issues with too many tasks on a large cluster attempting to access a serivce at any one time. This is similar to the functionality requested by MAPREDUCE-224 and implemented by HADOOP-3412 but was dropped in mrv2. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5583) Ability to limit running map and reduce tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-5583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13795443#comment-13795443 ] Arun C Murthy commented on MAPREDUCE-5583: -- Cluster with 100,000 containers, 1,000 jobs, each with 10 tasks, and specifies that they can only run 5 tasks. So, you are now only using 5% of the cluster and no one makes progress leading to very poor utilization and peanut-buttering effect. Admittedly it's a contrived example and yes, I agree a user can hack his own AM to do this - but let's not make this trivial for normal users. This leads to all sorts of bad side-effects by supporting it out of the box. Some form of admin control (e.g. queue with a max-cap) for a small number of use-cases where you *actually* need this feature is much safer. Ability to limit running map and reduce tasks - Key: MAPREDUCE-5583 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5583 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mr-am, mrv2 Affects Versions: 0.23.9, 2.1.1-beta Reporter: Jason Lowe It would be nice if users could specify a limit to the number of map or reduce tasks that are running simultaneously. Occasionally users are performing operations in tasks that can lead to DDoS scenarios if too many tasks run simultaneously (e.g.: accessing a database, web service, etc.). Having the ability to throttle the number of tasks simultaneously running would provide users a way to mitigate issues with too many tasks on a large cluster attempting to access a serivce at any one time. This is similar to the functionality requested by MAPREDUCE-224 and implemented by HADOOP-3412 but was dropped in mrv2. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5583) Ability to limit running map and reduce tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-5583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794782#comment-13794782 ] Arun C Murthy commented on MAPREDUCE-5583: -- [~jlowe] We can already accomplish this with queue/user limits? Ability to limit running map and reduce tasks - Key: MAPREDUCE-5583 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5583 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mr-am, mrv2 Affects Versions: 0.23.9, 2.1.1-beta Reporter: Jason Lowe It would be nice if users could specify a limit to the number of map or reduce tasks that are running simultaneously. Occasionally users are performing operations in tasks that can lead to DDoS scenarios if too many tasks run simultaneously (e.g.: accessing a database, web service, etc.). Having the ability to throttle the number of tasks simultaneously running would provide users a way to mitigate issues with too many tasks on a large cluster attempting to access a serivce at any one time. This is similar to the functionality requested by MAPREDUCE-224 and implemented by HADOOP-3412 but was dropped in mrv2. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5565) job clean up fails on secure cluster as the file system is not created in the context of the ugi running the job
[
https://issues.apache.org/jira/browse/MAPREDUCE-5565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-5565:
-
Resolution: Duplicate
Status: Resolved (was: Patch Available)
Duplicate of MAPREDUCE-5508.
job clean up fails on secure cluster as the file system is not created in the
context of the ugi running the job
Key: MAPREDUCE-5565
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5565
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobtracker
Affects Versions: 1.2.1
Reporter: Arpit Gupta
Assignee: Arun C Murthy
Priority: Critical
Fix For: 1.3.0
Attachments: MAPREDUCE-5565.patch, MAPREDUCE-5565.patch
On secure clusters we see the following exceptions in the jt log
{code}
2013-10-04 04:52:31,753 ERROR
org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:tt/host@REALM cause:javax.security.sasl.SaslException: GSS
initiate failed [Caused by GSSException: No valid credentials provided
(Mechanism level: Failed to find any Kerberos tgt)]
{code}
And after the job finishes the staging dir is not cleaned up. While debugging
with [~acmurthy] we determined that file system object needs to be created in
the the context of the user who ran the job.
Job however successfully completes
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (MAPREDUCE-5432) JobHistoryParser does not fetch clockSplits, cpuUsages, vMemKbytes, physMemKbytes from history file
[ https://issues.apache.org/jira/browse/MAPREDUCE-5432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-5432: - Status: Open (was: Patch Available) [~ozawa] Patch looks good. Can you pls modify existing tests to check for the new values? Tx. JobHistoryParser does not fetch clockSplits, cpuUsages, vMemKbytes, physMemKbytes from history file --- Key: MAPREDUCE-5432 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5432 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 2.0.4-alpha Reporter: Vrushali C Assignee: Tsuyoshi OZAWA Attachments: MAPREDUCE-5432.1.patch, MAPREDUCE-5432.2.patch JobHistoryParser's handleMapAttemptFinishedEvent() function does not look at MapAttemptFinishedEvent's int[] clockSplits; int[] cpuUsages; int[] vMemKbytes; int[] physMemKbytes; JobHistoryParser's inner class TaskAttemptInfo also needs to be enhanced to have these as members so that handleMapAttemptFinishedEvent() can get them and store them. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5565) job clean up fails on secure cluster as the file system is not created in the context of the ugi running the job
[
https://issues.apache.org/jira/browse/MAPREDUCE-5565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-5565:
-
Fix Version/s: 1.3.0
job clean up fails on secure cluster as the file system is not created in the
context of the ugi running the job
Key: MAPREDUCE-5565
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5565
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobtracker
Affects Versions: 1.2.1
Reporter: Arpit Gupta
Assignee: Arun C Murthy
Priority: Critical
Fix For: 1.3.0
Attachments: MAPREDUCE-5565.patch
On secure clusters we see the following exceptions in the jt log
{code}
2013-10-04 04:52:31,753 ERROR
org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:tt/host@REALM cause:javax.security.sasl.SaslException: GSS
initiate failed [Caused by GSSException: No valid credentials provided
(Mechanism level: Failed to find any Kerberos tgt)]
{code}
And after the job finishes the staging dir is not cleaned up. While debugging
with [~acmurthy] we determined that file system object needs to be created in
the the context of the user who ran the job.
Job however successfully completes
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (MAPREDUCE-5565) job clean up fails on secure cluster as the file system is not created in the context of the ugi running the job
[
https://issues.apache.org/jira/browse/MAPREDUCE-5565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-5565:
-
Status: Patch Available (was: Open)
job clean up fails on secure cluster as the file system is not created in the
context of the ugi running the job
Key: MAPREDUCE-5565
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5565
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobtracker
Affects Versions: 1.2.1
Reporter: Arpit Gupta
Assignee: Arun C Murthy
Priority: Critical
Attachments: MAPREDUCE-5565.patch
On secure clusters we see the following exceptions in the jt log
{code}
2013-10-04 04:52:31,753 ERROR
org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:tt/host@REALM cause:javax.security.sasl.SaslException: GSS
initiate failed [Caused by GSSException: No valid credentials provided
(Mechanism level: Failed to find any Kerberos tgt)]
{code}
And after the job finishes the staging dir is not cleaned up. While debugging
with [~acmurthy] we determined that file system object needs to be created in
the the context of the user who ran the job.
Job however successfully completes
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (MAPREDUCE-5565) job clean up fails on secure cluster as the file system is not created in the context of the ugi running the job
[
https://issues.apache.org/jira/browse/MAPREDUCE-5565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-5565:
-
Attachment: MAPREDUCE-5565.patch
Thanks for helping to track this down [~arpitgupta]!
Here is a straight-fwd patch which should fix the regression due to
MAPREDUCE-5351.
job clean up fails on secure cluster as the file system is not created in the
context of the ugi running the job
Key: MAPREDUCE-5565
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5565
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobtracker
Affects Versions: 1.2.1
Reporter: Arpit Gupta
Assignee: Arun C Murthy
Priority: Critical
Attachments: MAPREDUCE-5565.patch
On secure clusters we see the following exceptions in the jt log
{code}
2013-10-04 04:52:31,753 ERROR
org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:tt/host@REALM cause:javax.security.sasl.SaslException: GSS
initiate failed [Caused by GSSException: No valid credentials provided
(Mechanism level: Failed to find any Kerberos tgt)]
{code}
And after the job finishes the staging dir is not cleaned up. While debugging
with [~acmurthy] we determined that file system object needs to be created in
the the context of the user who ran the job.
Job however successfully completes
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (MAPREDUCE-5565) job clean up fails on secure cluster as the file system is not created in the context of the ugi running the job
[
https://issues.apache.org/jira/browse/MAPREDUCE-5565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-5565:
-
Attachment: MAPREDUCE-5565.patch
Slightly more conservative patch to reset tempDirFs in catch blocks.
job clean up fails on secure cluster as the file system is not created in the
context of the ugi running the job
Key: MAPREDUCE-5565
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5565
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobtracker
Affects Versions: 1.2.1
Reporter: Arpit Gupta
Assignee: Arun C Murthy
Priority: Critical
Fix For: 1.3.0
Attachments: MAPREDUCE-5565.patch, MAPREDUCE-5565.patch
On secure clusters we see the following exceptions in the jt log
{code}
2013-10-04 04:52:31,753 ERROR
org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:tt/host@REALM cause:javax.security.sasl.SaslException: GSS
initiate failed [Caused by GSSException: No valid credentials provided
(Mechanism level: Failed to find any Kerberos tgt)]
{code}
And after the job finishes the staging dir is not cleaned up. While debugging
with [~acmurthy] we determined that file system object needs to be created in
the the context of the user who ran the job.
Job however successfully completes
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (MAPREDUCE-5530) Binary and source incompatibility in mapred.lib.CombineFileInputFormat between branch-1 and branch-2
[
https://issues.apache.org/jira/browse/MAPREDUCE-5530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-5530:
-
Resolution: Fixed
Fix Version/s: 2.1.2-beta
Status: Resolved (was: Patch Available)
I just committed this. Thanks [~rkanter]!
Also, thanks to [~sandyr] and [~zjshen] for reviews and feedback.
Binary and source incompatibility in mapred.lib.CombineFileInputFormat
between branch-1 and branch-2
Key: MAPREDUCE-5530
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5530
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: mrv1, mrv2
Affects Versions: 2.1.1-beta
Reporter: Robert Kanter
Assignee: Robert Kanter
Priority: Blocker
Fix For: 2.1.2-beta
Attachments: MAPREDUCE-5530.patch, MAPREDUCE-5530.patch,
MAPREDUCE-5530.patch, MAPREDUCE-5530.patch
{{mapred.lib.CombineFileInputFormat}} in branch-1 has this method:
{code:java}
protected boolean isSplitable(FileSystem fs, Path file)
{code}
In branch-2, {{mapred.lib.CombineFileInputFormat}} is now a subclass of
{{mapreduce.lib.input.CombineFileInputFormat}}, from which it inherits the
similar method:
{code:java}
protected boolean isSplitable(JobContext context, Path file)
{code}
This means that any code that subclasses
{{mapred.lib.CombineFileInputFormat}} and does not provide its own
implementation of {{protected boolean isSplitable(FileSystem fs, Path file)}}
will not be binary or source compatible if it tries to call {{isSplitable}}
with a {{FileSystem}} argument anywhere (that is, if compiled against
branch-1, it will throw a {{NoSuchMethodError}} if run against branch-2;
also, it won't even compile against branch-2).
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (MAPREDUCE-5459) Update the doc of running MRv1 examples jar on YARN
[
https://issues.apache.org/jira/browse/MAPREDUCE-5459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-5459:
-
Resolution: Fixed
Fix Version/s: 2.1.2-beta
Status: Resolved (was: Patch Available)
I just committed this. Thanks [~zjshen]!
Update the doc of running MRv1 examples jar on YARN
---
Key: MAPREDUCE-5459
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5459
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Zhijie Shen
Assignee: Zhijie Shen
Fix For: 2.1.2-beta
Attachments: MAPREDUCE-5459.1.patch
In addition to adding two env vars: HADOOP_USER_CLASSPATH_FIRST and
HADOOP_CLASSPATH, we still need to add
{code}
property
namemapreduce.job.user.classpath.first/name
valuetrue/value
/property
{code}
in mapred-site.xml to make sure that the MRv1 examples jar runs correctly on
YARN. Some examples will use Java reflection to find the classes in the
examples jar dynamically when they are running. With this configuration, the
MRv1 examples jar will appear before the MRv2 examples jar in CLASSPATH of
the processes in YARN containers. Therefore, the classes found via reflection
will be picked from MRv1 examples jar instead of MRv2 examples jar as well.
MapReduce_Compatibility_Hadoop1_Hadoop2.apt.vm needs to be updated to
document this.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (MAPREDUCE-5530) Binary and source incompatibility in mapred.lib.CombineFileInputFormat between branch-1 and branch-2
[
https://issues.apache.org/jira/browse/MAPREDUCE-5530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13783099#comment-13783099
]
Arun C Murthy commented on MAPREDUCE-5530:
--
[~rkanter]: did my comment make sense, is there a chance you can quickly
incorporate them so we can resolve this last MR issue? Thanks!
Binary and source incompatibility in mapred.lib.CombineFileInputFormat
between branch-1 and branch-2
Key: MAPREDUCE-5530
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5530
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: mrv1, mrv2
Affects Versions: 2.1.1-beta
Reporter: Robert Kanter
Assignee: Robert Kanter
Priority: Blocker
Attachments: MAPREDUCE-5530.patch, MAPREDUCE-5530.patch
{{mapred.lib.CombineFileInputFormat}} in branch-1 has this method:
{code:java}
protected boolean isSplitable(FileSystem fs, Path file)
{code}
In branch-2, {{mapred.lib.CombineFileInputFormat}} is now a subclass of
{{mapreduce.lib.input.CombineFileInputFormat}}, from which it inherits the
similar method:
{code:java}
protected boolean isSplitable(JobContext context, Path file)
{code}
This means that any code that subclasses
{{mapred.lib.CombineFileInputFormat}} and does not provide its own
implementation of {{protected boolean isSplitable(FileSystem fs, Path file)}}
will not be binary or source compatible if it tries to call {{isSplitable}}
with a {{FileSystem}} argument anywhere (that is, if compiled against
branch-1, it will throw a {{NoSuchMethodError}} if run against branch-2;
also, it won't even compile against branch-2).
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (MAPREDUCE-5536) mapreduce.jobhistory.webapp.https.address property is not respected
[ https://issues.apache.org/jira/browse/MAPREDUCE-5536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-5536: - Target Version/s: 2.1.2-beta mapreduce.jobhistory.webapp.https.address property is not respected --- Key: MAPREDUCE-5536 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5536 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.1.1-beta Reporter: Yesha Vora Assignee: Omkar Vinit Joshi Priority: Blocker Attachments: MAPREDUCE-5536.20131027.1.patch, MAPREDUCE-5536.20131030.1.patch, MAPREDUCE-5536.20131030.1.patch, MAPREDUCE-5536.20131030.3.patch, YARN-1240.20131025.1.patch The jobhistory server starts on port defined by mapreduce.jobhistory.webapp.address property instead mapreduce.jobhistory.webapp.https.address when hadoop.ssl.enabled=true. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5530) Binary and source incompatibility in mapred.lib.CombineFileInputFormat between branch-1 and branch-2
[
https://issues.apache.org/jira/browse/MAPREDUCE-5530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13783447#comment-13783447
]
Arun C Murthy commented on MAPREDUCE-5530:
--
[~rkanter] People *really* should NOT be overriding
mapred.lib.CombineFileInputFormat.isSplitable(JobContext, Path) because
JobContext does not exist in the mapred api... hence we really should mark it
as @Private.
Binary and source incompatibility in mapred.lib.CombineFileInputFormat
between branch-1 and branch-2
Key: MAPREDUCE-5530
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5530
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: mrv1, mrv2
Affects Versions: 2.1.1-beta
Reporter: Robert Kanter
Assignee: Robert Kanter
Priority: Blocker
Attachments: MAPREDUCE-5530.patch, MAPREDUCE-5530.patch,
MAPREDUCE-5530.patch
{{mapred.lib.CombineFileInputFormat}} in branch-1 has this method:
{code:java}
protected boolean isSplitable(FileSystem fs, Path file)
{code}
In branch-2, {{mapred.lib.CombineFileInputFormat}} is now a subclass of
{{mapreduce.lib.input.CombineFileInputFormat}}, from which it inherits the
similar method:
{code:java}
protected boolean isSplitable(JobContext context, Path file)
{code}
This means that any code that subclasses
{{mapred.lib.CombineFileInputFormat}} and does not provide its own
implementation of {{protected boolean isSplitable(FileSystem fs, Path file)}}
will not be binary or source compatible if it tries to call {{isSplitable}}
with a {{FileSystem}} argument anywhere (that is, if compiled against
branch-1, it will throw a {{NoSuchMethodError}} if run against branch-2;
also, it won't even compile against branch-2).
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (MAPREDUCE-5530) Binary and source incompatibility in mapred.lib.CombineFileInputFormat between branch-1 and branch-2
[
https://issues.apache.org/jira/browse/MAPREDUCE-5530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-5530:
-
Status: Patch Available (was: Open)
Thanks [~rkanter]. Submitting patch for hudson on your behalf so that we can
commit this asap.
Binary and source incompatibility in mapred.lib.CombineFileInputFormat
between branch-1 and branch-2
Key: MAPREDUCE-5530
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5530
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: mrv1, mrv2
Affects Versions: 2.1.1-beta
Reporter: Robert Kanter
Assignee: Robert Kanter
Priority: Blocker
Attachments: MAPREDUCE-5530.patch, MAPREDUCE-5530.patch,
MAPREDUCE-5530.patch, MAPREDUCE-5530.patch
{{mapred.lib.CombineFileInputFormat}} in branch-1 has this method:
{code:java}
protected boolean isSplitable(FileSystem fs, Path file)
{code}
In branch-2, {{mapred.lib.CombineFileInputFormat}} is now a subclass of
{{mapreduce.lib.input.CombineFileInputFormat}}, from which it inherits the
similar method:
{code:java}
protected boolean isSplitable(JobContext context, Path file)
{code}
This means that any code that subclasses
{{mapred.lib.CombineFileInputFormat}} and does not provide its own
implementation of {{protected boolean isSplitable(FileSystem fs, Path file)}}
will not be binary or source compatible if it tries to call {{isSplitable}}
with a {{FileSystem}} argument anywhere (that is, if compiled against
branch-1, it will throw a {{NoSuchMethodError}} if run against branch-2;
also, it won't even compile against branch-2).
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (MAPREDUCE-5551) Binary Incompatibility of O.A.H.U.mapred.SequenceFileAsBinaryOutputFormat.WritableValueBytes
[ https://issues.apache.org/jira/browse/MAPREDUCE-5551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-5551: - Resolution: Fixed Fix Version/s: 2.1.2-beta Status: Resolved (was: Patch Available) I just committed this. Thanks [~zjshen]! Binary Incompatibility of O.A.H.U.mapred.SequenceFileAsBinaryOutputFormat.WritableValueBytes Key: MAPREDUCE-5551 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5551 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Priority: Blocker Fix For: 2.1.2-beta Attachments: MAPREDUCE-5551.1.patch The non-default constructor is moved to the super class, but it cannot be inherited. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5530) Binary and source incompatibility in mapred.lib.CombineFileInputFormat between branch-1 and branch-2
[
https://issues.apache.org/jira/browse/MAPREDUCE-5530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-5530:
-
Status: Open (was: Patch Available)
Binary and source incompatibility in mapred.lib.CombineFileInputFormat
between branch-1 and branch-2
Key: MAPREDUCE-5530
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5530
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: mrv1, mrv2
Affects Versions: 2.1.1-beta
Reporter: Robert Kanter
Assignee: Robert Kanter
Priority: Blocker
Attachments: MAPREDUCE-5530.patch, MAPREDUCE-5530.patch
{{mapred.lib.CombineFileInputFormat}} in branch-1 has this method:
{code:java}
protected boolean isSplitable(FileSystem fs, Path file)
{code}
In branch-2, {{mapred.lib.CombineFileInputFormat}} is now a subclass of
{{mapreduce.lib.input.CombineFileInputFormat}}, from which it inherits the
similar method:
{code:java}
protected boolean isSplitable(JobContext context, Path file)
{code}
This means that any code that subclasses
{{mapred.lib.CombineFileInputFormat}} and does not provide its own
implementation of {{protected boolean isSplitable(FileSystem fs, Path file)}}
will not be binary or source compatible if it tries to call {{isSplitable}}
with a {{FileSystem}} argument anywhere (that is, if compiled against
branch-1, it will throw a {{NoSuchMethodError}} if run against branch-2;
also, it won't even compile against branch-2).
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (MAPREDUCE-5530) Binary and source incompatibility in mapred.lib.CombineFileInputFormat between branch-1 and branch-2
[
https://issues.apache.org/jira/browse/MAPREDUCE-5530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13782567#comment-13782567
]
Arun C Murthy commented on MAPREDUCE-5530:
--
[~rkanter] I agree the patch isn't perfect, but that's the only option we have
now... thought I wonder if we should just document this and not fix it...
thoughts?
IAC, couple of comments:
# Please mark
org.apache.hadoop.mapred.lib.CombineFileInputFormat.isSplitable(JobContext,
Path) as @Private
# Also add some documentation to the method with a pointer to this jira
explaining why end-users should never override
org.apache.hadoop.mapred.lib.CombineFileInputFormat.isSplitable(JobContext,
Path).
Thanks.
Binary and source incompatibility in mapred.lib.CombineFileInputFormat
between branch-1 and branch-2
Key: MAPREDUCE-5530
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5530
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: mrv1, mrv2
Affects Versions: 2.1.1-beta
Reporter: Robert Kanter
Assignee: Robert Kanter
Priority: Blocker
Attachments: MAPREDUCE-5530.patch, MAPREDUCE-5530.patch
{{mapred.lib.CombineFileInputFormat}} in branch-1 has this method:
{code:java}
protected boolean isSplitable(FileSystem fs, Path file)
{code}
In branch-2, {{mapred.lib.CombineFileInputFormat}} is now a subclass of
{{mapreduce.lib.input.CombineFileInputFormat}}, from which it inherits the
similar method:
{code:java}
protected boolean isSplitable(JobContext context, Path file)
{code}
This means that any code that subclasses
{{mapred.lib.CombineFileInputFormat}} and does not provide its own
implementation of {{protected boolean isSplitable(FileSystem fs, Path file)}}
will not be binary or source compatible if it tries to call {{isSplitable}}
with a {{FileSystem}} argument anywhere (that is, if compiled against
branch-1, it will throw a {{NoSuchMethodError}} if run against branch-2;
also, it won't even compile against branch-2).
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (MAPREDUCE-5531) Binary and source incompatibility in mapreduce.TaskID and mapreduce.TaskAttemptID between branch-1 and branch-2
[
https://issues.apache.org/jira/browse/MAPREDUCE-5531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-5531:
-
Resolution: Fixed
Fix Version/s: 2.1.2-beta
Status: Resolved (was: Patch Available)
+1 from me too.
I just committed this. Thanks [~rkanter]!
Also, [~zjshen] thanks for the review.
Binary and source incompatibility in mapreduce.TaskID and
mapreduce.TaskAttemptID between branch-1 and branch-2
---
Key: MAPREDUCE-5531
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5531
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: mrv1, mrv2
Affects Versions: 2.1.1-beta
Reporter: Robert Kanter
Assignee: Robert Kanter
Priority: Blocker
Fix For: 2.1.2-beta
Attachments: MAPREDUCE-5531.patch, MAPREDUCE-5531.patch
{{mapreduce.TaskID}} in branch-1 has these two constructors:
{code:java}
public TaskID(JobID jobId, boolean isMap, int id)
public TaskID(String jtIdentifier, int jobId, boolean isMap, int id)
{code}
In branch-2, {{mapreduce.TaskID}} no longer has either of the above two
constructors.
Also, {{mapreduce.TaskAttemptID}} in branch-1 has this constructor:
{code:java}
public TaskAttemptID(String jtIdentifier, int jobId, boolean isMap, int
taskId, int id)
{code}
In branch-2, {{mapreduce.TaskAttemptID}} no longer his this constructor.
It looks like these constructors were probably removed because the {{boolean
isMap}} was replaced by an enum, {{TaskType}}.
This means that any code that tries to use any of those constructors will not
be binary or source compatible (in fact, the missing {{TaskAttemptID}}
constructor calls one of the missing {{TaskID}} constructors).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5529) Binary incompatibilities in mapred.lib.TotalOrderPartitioner between branch-1 and branch-2
[
https://issues.apache.org/jira/browse/MAPREDUCE-5529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-5529:
-
Attachment: MAPREDUCE-5529_branch_2.patch
Sigh, I needed to modify the branch-2 patch since MAPREDUCE-4594 is only on
trunk
[~qwertymaniac]: in future, please don't commit simple enhancements such as
MAPREDU-4594 only to trunk. Thanks.
Binary incompatibilities in mapred.lib.TotalOrderPartitioner between branch-1
and branch-2
--
Key: MAPREDUCE-5529
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5529
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: mrv1, mrv2
Affects Versions: 2.1.1-beta
Reporter: Robert Kanter
Assignee: Robert Kanter
Priority: Blocker
Attachments: MAPREDUCE-5529_branch_2.patch, MAPREDUCE-5529.patch,
MAPREDUCE-5529.patch
{{mapred.lib.TotalOrderPartitioner}} in branch-1 has these two methods:
{code:java}
public static String getPartitionFile(JobConf job)
public static void setPartitionFile(JobConf job, Path p)
{code}
In branch-2, {{mapred.lib.TotalOrderPartitioner}} is now a subclass of
{{mapred.lib.TotalOrderPartitioner}}, from which it inherits the similar
methods:
{code:java}
public static String getPartitionFile(Configuration conf)
public static void setPartitionFile(Configuration conf, Path p)
{code}
This means that any code that does either of the following:
{code:java}
TotalOrderPartitioner.setPartitionFile(new JobConf(), new Path(/));
String str = TotalOrderPartitioner.getPartitionFile(new JobConf());
{code}
will not be binary compatible (that is, if compiled against branch-1, it will
throw a {{NoSuchMethodError}} if run against branch-2).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5529) Binary incompatibilities in mapred.lib.TotalOrderPartitioner between branch-1 and branch-2
[
https://issues.apache.org/jira/browse/MAPREDUCE-5529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-5529:
-
Resolution: Fixed
Fix Version/s: 2.1.2-beta
Status: Resolved (was: Patch Available)
+1. I just committed this, thanks [~rkanter]!
Thanks for the reviews [~zjshen]!
Binary incompatibilities in mapred.lib.TotalOrderPartitioner between branch-1
and branch-2
--
Key: MAPREDUCE-5529
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5529
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: mrv1, mrv2
Affects Versions: 2.1.1-beta
Reporter: Robert Kanter
Assignee: Robert Kanter
Priority: Blocker
Fix For: 2.1.2-beta
Attachments: MAPREDUCE-5529_branch_2.patch, MAPREDUCE-5529.patch,
MAPREDUCE-5529.patch
{{mapred.lib.TotalOrderPartitioner}} in branch-1 has these two methods:
{code:java}
public static String getPartitionFile(JobConf job)
public static void setPartitionFile(JobConf job, Path p)
{code}
In branch-2, {{mapred.lib.TotalOrderPartitioner}} is now a subclass of
{{mapred.lib.TotalOrderPartitioner}}, from which it inherits the similar
methods:
{code:java}
public static String getPartitionFile(Configuration conf)
public static void setPartitionFile(Configuration conf, Path p)
{code}
This means that any code that does either of the following:
{code:java}
TotalOrderPartitioner.setPartitionFile(new JobConf(), new Path(/));
String str = TotalOrderPartitioner.getPartitionFile(new JobConf());
{code}
will not be binary compatible (that is, if compiled against branch-1, it will
throw a {{NoSuchMethodError}} if run against branch-2).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5514) TestRMContainerAllocator fails on trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-5514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-5514: - Resolution: Fixed Fix Version/s: 2.3.0 Status: Resolved (was: Patch Available) TestRMContainerAllocator fails on trunk --- Key: MAPREDUCE-5514 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5514 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Zhijie Shen Assignee: Zhijie Shen Priority: Blocker Fix For: 2.3.0 Attachments: MAPREDUCE-5514.1.patch, MAPREDUCE-5514.2.patch, MAPREDUCE-5514.3.patch, org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator-output.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5514) TestRMContainerAllocator fails on trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-5514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13779419#comment-13779419 ] Arun C Murthy commented on MAPREDUCE-5514: -- I just committed this. Thanks [~zjshen]! TestRMContainerAllocator fails on trunk --- Key: MAPREDUCE-5514 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5514 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Zhijie Shen Assignee: Zhijie Shen Priority: Blocker Fix For: 2.3.0 Attachments: MAPREDUCE-5514.1.patch, MAPREDUCE-5514.2.patch, MAPREDUCE-5514.3.patch, org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator-output.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5529) Binary incompatibilities in mapred.lib.TotalOrderPartitioner between branch-1 and branch-2
[
https://issues.apache.org/jira/browse/MAPREDUCE-5529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-5529:
-
Target Version/s: 2.1.2-beta
Binary incompatibilities in mapred.lib.TotalOrderPartitioner between branch-1
and branch-2
--
Key: MAPREDUCE-5529
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5529
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: mrv1, mrv2
Affects Versions: 2.1.1-beta
Reporter: Robert Kanter
Assignee: Robert Kanter
Priority: Blocker
Attachments: MAPREDUCE-5529.patch
{{mapred.lib.TotalOrderPartitioner}} in branch-1 has these two methods:
{code:java}
public static String getPartitionFile(JobConf job)
public static void setPartitionFile(JobConf job, Path p)
{code}
In branch-2, {{mapred.lib.TotalOrderPartitioner}} is now a subclass of
{{mapred.lib.TotalOrderPartitioner}}, from which it inherits the similar
methods:
{code:java}
public static String getPartitionFile(Configuration conf)
public static void setPartitionFile(Configuration conf, Path p)
{code}
This means that any code that does either of the following:
{code:java}
TotalOrderPartitioner.setPartitionFile(new JobConf(), new Path(/));
String str = TotalOrderPartitioner.getPartitionFile(new JobConf());
{code}
will not be binary compatible (that is, if compiled against branch-1, it will
throw a {{NoSuchMethodError}} if run against branch-2).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5503) TestMRJobClient.testJobClient is failing
[
https://issues.apache.org/jira/browse/MAPREDUCE-5503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-5503:
-
Target Version/s: 3.0.0, 2.1.2-beta (was: 3.0.0, 2.1.1-beta)
TestMRJobClient.testJobClient is failing
Key: MAPREDUCE-5503
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5503
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Jason Lowe
Assignee: Jian He
Priority: Blocker
Attachments: MAPREDUCE-5503.patch, MAPREDUCE-5503.patch
TestMRJobClient.testJobClient is failing on trunk and causing precommit
builds to complain:
{noformat}
testJobClient(org.apache.hadoop.mapreduce.TestMRJobClient) Time elapsed:
26.361 sec FAILURE!
junit.framework.AssertionFailedError: expected:1 but was:0
at junit.framework.Assert.fail(Assert.java:50)
at junit.framework.Assert.failNotEquals(Assert.java:287)
at junit.framework.Assert.assertEquals(Assert.java:67)
at junit.framework.Assert.assertEquals(Assert.java:199)
at junit.framework.Assert.assertEquals(Assert.java:205)
at
org.apache.hadoop.mapreduce.TestMRJobClient.testJobList(TestMRJobClient.java:474)
at
org.apache.hadoop.mapreduce.TestMRJobClient.testJobClient(TestMRJobClient.java:112)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5448) MapFileOutputFormat#getReaders bug with invisible files/folders
[
https://issues.apache.org/jira/browse/MAPREDUCE-5448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-5448:
-
Status: Patch Available (was: Open)
MapFileOutputFormat#getReaders bug with invisible files/folders
---
Key: MAPREDUCE-5448
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5448
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Maysam Yabandeh
Assignee: Maysam Yabandeh
Priority: Minor
Attachments: MAPREDUCE-5448.patch
Original Estimate: 1h
Remaining Estimate: 1h
MapReduce jobs also produce some invisible files such as _SUCCESS, even when
the output format is MapFileOutputFormat. MapFileOutputFormat#getReaders
however reads the entire content of the job output, assming that they are
MapFiles.
{code}
Path[] names = FileUtil.stat2Paths(fs.listStatus(dir));
{code}
It should use a filter to skip the files that start with . or _.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5493) In-memory map outputs can be leaked after shuffle completes
[ https://issues.apache.org/jira/browse/MAPREDUCE-5493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-5493: - Resolution: Fixed Fix Version/s: 2.1.1-beta Status: Resolved (was: Patch Available) +1, looks good - thanks for tracking this down [~jlowe]! I just committed this. In-memory map outputs can be leaked after shuffle completes --- Key: MAPREDUCE-5493 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5493 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.1.0-beta, 0.23.9 Reporter: Jason Lowe Assignee: Jason Lowe Priority: Blocker Fix For: 2.1.1-beta Attachments: MAPREDUCE-5493.patch MergeManagerImpl#close adds the contents of inMemoryMergedMapOutputs and inMemoryMapOutputs to a list of map outputs that is subsequently processed, but it does not clear those sets. This prevents some of the map outputs from being garbage collected and significantly reduces the memory available for the subsequent reduce phase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5480) TestJSHSecurity.testDelegationToken is breaking after YARN-1085
[
https://issues.apache.org/jira/browse/MAPREDUCE-5480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-5480:
-
Resolution: Duplicate
Status: Resolved (was: Patch Available)
Fixed via YARN-1085
TestJSHSecurity.testDelegationToken is breaking after YARN-1085
---
Key: MAPREDUCE-5480
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5480
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.1.1-beta
Reporter: Vinod Kumar Vavilapalli
Assignee: Omkar Vinit Joshi
Priority: Blocker
Attachments: MAPREDUCE-5480.20130824.1.patch
See https://builds.apache.org/job/PreCommit-YARN-Build/1755//testReport/.
{code}
org.apache.hadoop.yarn.webapp.WebAppException: Error starting http server
at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:251)
at
org.apache.hadoop.mapreduce.v2.hs.HistoryClientService.initializeWebApp(HistoryClientService.java:152)
---
Caused by: javax.servlet.ServletException: javax.servlet.ServletException:
Principal not defined in configuration
at
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.init(KerberosAuthenticationHandler.java:203)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.init(AuthenticationFilter.java:146)
---
Caused by: javax.servlet.ServletException: Principal not defined in
configuration
at
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.init(KerberosAuthenticationHandler.java:164)
... 53 more
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5439) mapred-default.xml has missing properties
[ https://issues.apache.org/jira/browse/MAPREDUCE-5439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-5439: - Fix Version/s: (was: 2.1.0-beta) 2.3.0 mapred-default.xml has missing properties - Key: MAPREDUCE-5439 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5439 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.1.0-beta Reporter: Siddharth Wagle Fix For: 2.3.0 Properties that need to be added: mapreduce.map.memory.mb mapreduce.map.java.opts mapreduce.reduce.memory.mb mapreduce.reduce.java.opts Properties that need to be fixed: mapred.child.java.opts should not be in mapred-default. yarn.app.mapreduce.am.command-opts description needs fixing -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4868) Allow multiple iteration for map
[ https://issues.apache.org/jira/browse/MAPREDUCE-4868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-4868: - Fix Version/s: (was: 2.1.0-beta) 2.3.0 Allow multiple iteration for map Key: MAPREDUCE-4868 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4868 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 3.0.0, 2.0.3-alpha Reporter: Jerry Chen Fix For: 3.0.0, 2.3.0 Original Estimate: 168h Remaining Estimate: 168h Currently, the Mapper class allows advanced users to override public void run(Context context) method for more control over the execution of the mapper, while Context interface limit the operations over the data which is the foundation of more control. One of use cases is that when I am considering a hive optimziation problem, I want to go two passes over the input data instead of using a another job or task ( which may slower the whole process). Each pass do the same thing but with a different parameters. This is a new paradigm of Map Reduce usage and can be archived easily by extend Context interface a little with the more control over the data such as reset the input. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5036) Default shuffle handler port should not be 8080
[ https://issues.apache.org/jira/browse/MAPREDUCE-5036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-5036: - Fix Version/s: (was: 2.1.0-beta) 2.3.0 Default shuffle handler port should not be 8080 --- Key: MAPREDUCE-5036 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5036 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 2.3.0 Attachments: MAPREDUCE-5036-13562.patch, MAPREDUCE-5036-2.patch, MAPREDUCE-5036.patch The shuffle handler port (mapreduce.shuffle.port) defaults to 8080. This is a pretty common port for web services, and is likely to cause unnecessary port conflicts. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5028) Maps fail when io.sort.mb is set to high value
[
https://issues.apache.org/jira/browse/MAPREDUCE-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-5028:
-
Fix Version/s: (was: 2.1.0-beta)
2.3.0
Maps fail when io.sort.mb is set to high value
--
Key: MAPREDUCE-5028
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5028
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 1.1.1, 2.0.3-alpha, 0.23.5
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Priority: Critical
Fix For: 1.2.0, 2.3.0
Attachments: mr-5028-branch1.patch, mr-5028-branch1.patch,
mr-5028-branch1.patch, MR-5028_testapp.patch, mr-5028-trunk.patch,
mr-5028-trunk.patch, mr-5028-trunk.patch, repro-mr-5028.patch
Verified the problem exists on branch-1 with the following configuration:
Pseudo-dist mode: 2 maps/ 1 reduce, mapred.child.java.opts=-Xmx2048m,
io.sort.mb=1280, dfs.block.size=2147483648
Run teragen to generate 4 GB data
Maps fail when you run wordcount on this configuration with the following
error:
{noformat}
java.io.IOException: Spill failed
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1031)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:692)
at
org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
at
org.apache.hadoop.examples.WordCount$TokenizerMapper.map(WordCount.java:45)
at
org.apache.hadoop.examples.WordCount$TokenizerMapper.map(WordCount.java:34)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:766)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:375)
at org.apache.hadoop.io.IntWritable.readFields(IntWritable.java:38)
at
org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:67)
at
org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40)
at
org.apache.hadoop.mapreduce.ReduceContext.nextKeyValue(ReduceContext.java:116)
at
org.apache.hadoop.mapreduce.ReduceContext.nextKey(ReduceContext.java:92)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:175)
at
org.apache.hadoop.mapred.Task$NewCombinerRunner.combine(Task.java:1505)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1438)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$1800(MapTask.java:855)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:1346)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4468) Encapsulate FairScheduler preemption logic into helper class
[ https://issues.apache.org/jira/browse/MAPREDUCE-4468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-4468: - Fix Version/s: (was: 2.1.0-beta) 2.3.0 Encapsulate FairScheduler preemption logic into helper class Key: MAPREDUCE-4468 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4468 Project: Hadoop Map/Reduce Issue Type: Improvement Components: scheduler Reporter: Ryan Hennig Priority: Minor Labels: refactoring, scheduler Fix For: 2.3.0 Attachments: MAPREDUCE-4468.patch Original Estimate: 4h Remaining Estimate: 4h I've extracted the preemption logic from the Fair Scheduler into a helper class so that FairScheduler is closer to following the Single Responsibility Principle. This may eventually evolve into a generalized preemption module which could be leveraged by other schedulers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5170) incorrect exception message if min node size min rack size
[ https://issues.apache.org/jira/browse/MAPREDUCE-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-5170: - Fix Version/s: (was: 2.1.0-beta) 2.3.0 incorrect exception message if min node size min rack size Key: MAPREDUCE-5170 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5170 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.0.3-alpha Reporter: Sangjin Lee Priority: Trivial Fix For: 2.3.0 Attachments: MAPREDUCE-5170.patch The exception message for CombineFileInputFormat if min node size min rack size is worded backwards. Currently it reads Minimum split size per node... cannot be smaller than the minimum split size per rack... It should be Minimum split size per node... cannot be LARGER than the minimum split size per rack... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4253) Tests for mapreduce-client-core are lying under mapreduce-client-jobclient
[ https://issues.apache.org/jira/browse/MAPREDUCE-4253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-4253: - Fix Version/s: (was: 2.1.0-beta) 2.3.0 Tests for mapreduce-client-core are lying under mapreduce-client-jobclient -- Key: MAPREDUCE-4253 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4253 Project: Hadoop Map/Reduce Issue Type: Task Components: client Affects Versions: 2.0.0-alpha Reporter: Harsh J Assignee: Tsuyoshi OZAWA Fix For: 2.3.0 Attachments: crossing_project_checker.rb, MR-4253.1.patch, MR-4253.2.patch, result.txt Many of the tests for client libs from mapreduce-client-core are lying under mapreduce-client-jobclient. We should investigate if this is the right thing to do and if not, move the tests back into client-core. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4462) Enhance readability of TestFairScheduler.java
[ https://issues.apache.org/jira/browse/MAPREDUCE-4462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-4462: - Fix Version/s: (was: 2.1.0-beta) 2.3.0 Enhance readability of TestFairScheduler.java - Key: MAPREDUCE-4462 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4462 Project: Hadoop Map/Reduce Issue Type: Bug Components: scheduler, test Reporter: Ryan Hennig Priority: Minor Labels: comments, test Fix For: 2.3.0 Attachments: MAPREDUCE-4462.patch Original Estimate: 2h Remaining Estimate: 2h While reading over the unit tests for the Fair Scheduler introduced by MAPREDUCE-3451, I added comments to make the logic of the test easier to grok quickly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5480) TestJSHSecurity.testDelegationToken is breaking after YARN-1085
[
https://issues.apache.org/jira/browse/MAPREDUCE-5480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-5480:
-
Priority: Blocker (was: Major)
TestJSHSecurity.testDelegationToken is breaking after YARN-1085
---
Key: MAPREDUCE-5480
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5480
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.1.1-beta
Reporter: Vinod Kumar Vavilapalli
Assignee: Vinod Kumar Vavilapalli
Priority: Blocker
See https://builds.apache.org/job/PreCommit-YARN-Build/1755//testReport/.
{code}
org.apache.hadoop.yarn.webapp.WebAppException: Error starting http server
at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:251)
at
org.apache.hadoop.mapreduce.v2.hs.HistoryClientService.initializeWebApp(HistoryClientService.java:152)
---
Caused by: javax.servlet.ServletException: javax.servlet.ServletException:
Principal not defined in configuration
at
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.init(KerberosAuthenticationHandler.java:203)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.init(AuthenticationFilter.java:146)
---
Caused by: javax.servlet.ServletException: Principal not defined in
configuration
at
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.init(KerberosAuthenticationHandler.java:164)
... 53 more
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5475) MRClientService does not verify ACLs properly
[ https://issues.apache.org/jira/browse/MAPREDUCE-5475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13746965#comment-13746965 ] Arun C Murthy commented on MAPREDUCE-5475: -- +1 pending jenkins, lgtm. Thanks [~jlowe]! MRClientService does not verify ACLs properly - Key: MAPREDUCE-5475 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5475 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am, mrv2 Affects Versions: 2.0.4-alpha, 0.23.9 Reporter: Jason Lowe Assignee: Jason Lowe Priority: Blocker Attachments: MAPREDUCE-5475.patch When MRClientService receives requests, it calls verifyAndGetJob which does not actually validate that the current user has the proper access. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5476) Job can fail when RM restarts after staging dir is cleaned but before MR successfully unregister with RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-5476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-5476: - Target Version/s: 2.1.1-beta Job can fail when RM restarts after staging dir is cleaned but before MR successfully unregister with RM Key: MAPREDUCE-5476 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5476 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Jian He Assignee: Jian He Priority: Blocker Attachments: MAPREDUCE-5476.patch, YARN-917.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5476) Job can fail when RM restarts after staging dir is cleaned but before MR successfully unregister with RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-5476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-5476: - Priority: Blocker (was: Major) Job can fail when RM restarts after staging dir is cleaned but before MR successfully unregister with RM Key: MAPREDUCE-5476 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5476 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Jian He Assignee: Jian He Priority: Blocker Attachments: MAPREDUCE-5476.patch, YARN-917.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5468) AM recovery does not work for map only jobs
[ https://issues.apache.org/jira/browse/MAPREDUCE-5468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-5468: - Resolution: Fixed Fix Version/s: 2.1.1-beta Status: Resolved (was: Patch Available) +1 looks good. Good test too. I just committed this. Thanks [~vinodkv]! AM recovery does not work for map only jobs --- Key: MAPREDUCE-5468 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5468 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: yeshavora Assignee: Vinod Kumar Vavilapalli Priority: Blocker Fix For: 2.1.1-beta Attachments: MAPREDUCE-5468-20130821.txt Map only job (randomwriter, randomtextwriter) restarts from scratch [0% map 0% reduce] after RM restart. It should resume from the last state when AM restarted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5466) Historyserver does not refresh the result of restarted jobs after RM restart
[ https://issues.apache.org/jira/browse/MAPREDUCE-5466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-5466: - Priority: Blocker (was: Major) Historyserver does not refresh the result of restarted jobs after RM restart Key: MAPREDUCE-5466 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5466 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: yeshavora Assignee: Jian He Priority: Blocker Attachments: MAPREDUCE-5466.1.patch, MAPREDUCE-5466.patch Restart RM when sort job is running and verify that the job passes successfully after RM restarts. Once the job finishes successfully, run job status command for sort job. It shows Job state =FAILED. Job history server does not update the result for the job which restarted after RM restart. hadoop job -status job_1375923346354_0003 13/08/08 01:24:13 INFO mapred.ClientServiceDelegate: Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server Job: job_1375923346354_0003 Job File: hdfs://host1:port1/history/done/2013/08/08/00/job_1375923346354_0003_conf.xml Job Tracking URL : http://historyserver:port2/jobhistory/job/job_1375923346354_0003 Uber job : false Number of maps: 80 Number of reduces: 1 map() completion: 0.0 reduce() completion: 0.0 Job state: FAILED retired: false reason for failure: There are no failed tasks for the job. Job is failed due to some other reason and reason can be found in the logs. Counters not available. Job is retired. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5468) AM recovery does not work for map only jobs
[ https://issues.apache.org/jira/browse/MAPREDUCE-5468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-5468: - Priority: Blocker (was: Major) AM recovery does not work for map only jobs --- Key: MAPREDUCE-5468 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5468 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: yeshavora Assignee: Vinod Kumar Vavilapalli Priority: Blocker Map only job (randomwriter, randomtextwriter) restarts from scratch [0% map 0% reduce] after RM restart. It should resume from the last state when AM restarted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5466) Historyserver does not refresh the result of restarted jobs after RM restart
[ https://issues.apache.org/jira/browse/MAPREDUCE-5466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-5466: - Target Version/s: 2.1.1-beta (was: 2.1.0-beta) Historyserver does not refresh the result of restarted jobs after RM restart Key: MAPREDUCE-5466 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5466 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: yeshavora Assignee: Jian He Priority: Blocker Attachments: MAPREDUCE-5466.1.patch, MAPREDUCE-5466.patch Restart RM when sort job is running and verify that the job passes successfully after RM restarts. Once the job finishes successfully, run job status command for sort job. It shows Job state =FAILED. Job history server does not update the result for the job which restarted after RM restart. hadoop job -status job_1375923346354_0003 13/08/08 01:24:13 INFO mapred.ClientServiceDelegate: Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server Job: job_1375923346354_0003 Job File: hdfs://host1:port1/history/done/2013/08/08/00/job_1375923346354_0003_conf.xml Job Tracking URL : http://historyserver:port2/jobhistory/job/job_1375923346354_0003 Uber job : false Number of maps: 80 Number of reduces: 1 map() completion: 0.0 reduce() completion: 0.0 Job state: FAILED retired: false reason for failure: There are no failed tasks for the job. Job is failed due to some other reason and reason can be found in the logs. Counters not available. Job is retired. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5311) Replace SLOTS_MILLIS counters with MEM_MILLIS
[ https://issues.apache.org/jira/browse/MAPREDUCE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13739629#comment-13739629 ] Arun C Murthy commented on MAPREDUCE-5311: -- I'm glad we are coming around to understand we can break existing users of SLOT_MILLIS. I wasn't thrilled about using configs and I'm even more leery of adding new configs to track existing configs in RM. This can lead to all sorts of configuration-error hell with mismatch in configs b/w YarnConfiguration and MRConf. I'm against getting into that sort of situations. A much simpler approach is to re-introduce a LimitedPrivate RegisterApplicationMasterResponse.getMinimumResourceCapability as described here (http://bit.ly/18uA8zj). This would revert some part of YARN-787. Replace SLOTS_MILLIS counters with MEM_MILLIS - Key: MAPREDUCE-5311 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5311 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.0.4-alpha Reporter: Alejandro Abdelnur Assignee: Sandy Ryza Priority: Blocker Fix For: 2.1.0-beta Attachments: MAPREDUCE-5311-1.patch, MAPREDUCE-5311.patch, MAPREDUCE-5311.patch Per discussion in MAPREDUCE-5310 and comments in the code we should remove all the related logic and just leave the counter constant for backwards compatibility and deprecate the counter constants. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5311) Replace SLOTS_MILLIS counters with MEM_MILLIS
[ https://issues.apache.org/jira/browse/MAPREDUCE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13740390#comment-13740390 ] Arun C Murthy commented on MAPREDUCE-5311: -- I'll say this again, I'm -1 on adding a new config to track another one in the scheduler. Expecting admins to remember another config for the same value (one for YARN another for MR) is not kosher - our configuration is bad enough. We can use the same one in the scheduler (i.e. don't fix YARN-1004) or we can add back RegisterApplicationMasterResponse.getMinimumResourceCapability as LimitedPrivate(MapReduce). I'm ok with either of the above. Replace SLOTS_MILLIS counters with MEM_MILLIS - Key: MAPREDUCE-5311 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5311 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.0.4-alpha Reporter: Alejandro Abdelnur Assignee: Sandy Ryza Priority: Blocker Fix For: 2.1.0-beta Attachments: MAPREDUCE-5311-1.patch, MAPREDUCE-5311.patch, MAPREDUCE-5311.patch Per discussion in MAPREDUCE-5310 and comments in the code we should remove all the related logic and just leave the counter constant for backwards compatibility and deprecate the counter constants. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5311) Replace SLOTS_MILLIS counters with MEM_MILLIS
[ https://issues.apache.org/jira/browse/MAPREDUCE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13740577#comment-13740577 ] Arun C Murthy commented on MAPREDUCE-5311: -- This isn't a special case for 0.23. It will break tools built atop both hadoop-1.x hadoop-0.23.*. Replace SLOTS_MILLIS counters with MEM_MILLIS - Key: MAPREDUCE-5311 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5311 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.0.4-alpha Reporter: Alejandro Abdelnur Assignee: Sandy Ryza Priority: Blocker Fix For: 2.1.0-beta Attachments: MAPREDUCE-5311-1.patch, MAPREDUCE-5311.patch, MAPREDUCE-5311.patch Per discussion in MAPREDUCE-5310 and comments in the code we should remove all the related logic and just leave the counter constant for backwards compatibility and deprecate the counter constants. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5311) Replace SLOTS_MILLIS counters with MEM_MILLIS
[ https://issues.apache.org/jira/browse/MAPREDUCE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13740598#comment-13740598 ] Arun C Murthy commented on MAPREDUCE-5311: -- I'm ok with Vinod's suggestion too. Replace SLOTS_MILLIS counters with MEM_MILLIS - Key: MAPREDUCE-5311 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5311 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.0.4-alpha Reporter: Alejandro Abdelnur Assignee: Sandy Ryza Priority: Blocker Fix For: 2.1.0-beta Attachments: MAPREDUCE-5311-1.patch, MAPREDUCE-5311.patch, MAPREDUCE-5311.patch Per discussion in MAPREDUCE-5310 and comments in the code we should remove all the related logic and just leave the counter constant for backwards compatibility and deprecate the counter constants. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5311) Replace SLOTS_MILLIS counters with MEM_MILLIS
[ https://issues.apache.org/jira/browse/MAPREDUCE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13740613#comment-13740613 ] Arun C Murthy commented on MAPREDUCE-5311: -- [~tucu00] Can you please decide on one of the 3 options available which are reasonable? I'm ready to roll 2.1.0-rc2 ATM, this is the only issue I can think of. Thanks. Replace SLOTS_MILLIS counters with MEM_MILLIS - Key: MAPREDUCE-5311 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5311 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.0.4-alpha Reporter: Alejandro Abdelnur Assignee: Sandy Ryza Priority: Blocker Fix For: 2.1.0-beta Attachments: MAPREDUCE-5311-1.patch, MAPREDUCE-5311.patch, MAPREDUCE-5311.patch Per discussion in MAPREDUCE-5310 and comments in the code we should remove all the related logic and just leave the counter constant for backwards compatibility and deprecate the counter constants. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5311) Replace SLOTS_MILLIS counters with MEM_MILLIS
[ https://issues.apache.org/jira/browse/MAPREDUCE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13740671#comment-13740671 ] Arun C Murthy commented on MAPREDUCE-5311: -- I'll repeat that I'm against multiple configs for the same property which leads to operator errors (admins setting one but forgetting to set the other) and am willing to live with other alternatives which seem better from a technical standpoint. bq. Why we don't use the Hadoop 1 JT properties for the same, in the mapred-site.xml documenting how they have to be set to the MIN of the scheduler for SLOT_MILLIS counter to kick in? To me this seem a much more correct way of doing it. ATM, I don't see where this argument is going at all - frankly, we seem to be making YARN-1004 much bigger deal than it is. Several folks have expressed concerns about multiple configs and yet we don't seem to converge; this is in spite of, IMO, very reasonable alternatives. I'm inclined to go ahead with rc2 without this YARN-1004. We can revisit both for 2.1.1 via deprecation of properties in YARN-1004 etc. if necessary. My plan is to cut an rc2 by noon PST tmrw (9/15). Let's revisit this after if we can agree today. Thanks. Replace SLOTS_MILLIS counters with MEM_MILLIS - Key: MAPREDUCE-5311 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5311 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.0.4-alpha Reporter: Alejandro Abdelnur Assignee: Sandy Ryza Priority: Blocker Fix For: 2.1.0-beta Attachments: MAPREDUCE-5311-1.patch, MAPREDUCE-5311.patch, MAPREDUCE-5311.patch Per discussion in MAPREDUCE-5310 and comments in the code we should remove all the related logic and just leave the counter constant for backwards compatibility and deprecate the counter constants. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (MAPREDUCE-5311) Replace SLOTS_MILLIS counters with MEM_MILLIS
[ https://issues.apache.org/jira/browse/MAPREDUCE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13740671#comment-13740671 ] Arun C Murthy edited comment on MAPREDUCE-5311 at 8/15/13 5:04 AM: --- I'll repeat that I'm against multiple configs for the same property which leads to operator errors (admins setting one but forgetting to set the other) and am willing to live with other alternatives which seem better from a technical standpoint. bq. Why we don't use the Hadoop 1 JT properties for the same, in the mapred-site.xml documenting how they have to be set to the MIN of the scheduler for SLOT_MILLIS counter to kick in? To me this seem a much more correct way of doing it. ATM, I don't see where this argument is going at all - frankly, we seem to be making YARN-1004 much bigger deal than it is. Several folks have expressed concerns about multiple configs and yet we don't seem to converge; this is in spite of, IMO, very reasonable alternatives. I'm inclined to go ahead with rc2 without MAPREDUCE-5311 YARN-1004. Of course, we can revisit both for 2.1.1 via deprecation of yarn.scheduler.min being discussed in YARN-1004. I'll stress that we can continue talking and get both in once we all agree. My plan is to cut an rc2 by noon PST tmrw (9/15). Let's revisit this discussion after if we can't converge before then. Makes sense? Thanks. was (Author: acmurthy): I'll repeat that I'm against multiple configs for the same property which leads to operator errors (admins setting one but forgetting to set the other) and am willing to live with other alternatives which seem better from a technical standpoint. bq. Why we don't use the Hadoop 1 JT properties for the same, in the mapred-site.xml documenting how they have to be set to the MIN of the scheduler for SLOT_MILLIS counter to kick in? To me this seem a much more correct way of doing it. ATM, I don't see where this argument is going at all - frankly, we seem to be making YARN-1004 much bigger deal than it is. Several folks have expressed concerns about multiple configs and yet we don't seem to converge; this is in spite of, IMO, very reasonable alternatives. I'm inclined to go ahead with rc2 without this YARN-1004. We can revisit both for 2.1.1 via deprecation of properties in YARN-1004 etc. if necessary. My plan is to cut an rc2 by noon PST tmrw (9/15). Let's revisit this after if we can agree today. Thanks. Replace SLOTS_MILLIS counters with MEM_MILLIS - Key: MAPREDUCE-5311 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5311 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.0.4-alpha Reporter: Alejandro Abdelnur Assignee: Sandy Ryza Priority: Blocker Fix For: 2.1.0-beta Attachments: MAPREDUCE-5311-1.patch, MAPREDUCE-5311.patch, MAPREDUCE-5311.patch Per discussion in MAPREDUCE-5310 and comments in the code we should remove all the related logic and just leave the counter constant for backwards compatibility and deprecate the counter constants. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5311) Replace SLOTS_MILLIS counters with MEM_MILLIS
[ https://issues.apache.org/jira/browse/MAPREDUCE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738236#comment-13738236 ] Arun C Murthy commented on MAPREDUCE-5311: -- I've repeatedly pointed out that SLOT_MILLIS is something which is very useful and is used by a lot of end-users. I know of a lot of users who have built rudimentary reporting/charge-back systems using SLOT_MILLIS. Breaking compatibility (semantic or api) is not an option. We've done a lot of work on MR to keep it compatible so far. The more I think about it, the more strongly I believe YARN-787 went too far by removing 'minimum' from the API. We can/should keep RegisterApplicationMasterResponse.getMinimumResourceCapability as a LimitedPrivate api and use it only for calculation of SLOT_MILLIS. We don't have to expose it to users or other applications. Makes sense? Replace SLOTS_MILLIS counters with MEM_MILLIS - Key: MAPREDUCE-5311 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5311 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.0.4-alpha Reporter: Alejandro Abdelnur Assignee: Sandy Ryza Priority: Blocker Fix For: 2.1.0-beta Attachments: MAPREDUCE-5311-1.patch, MAPREDUCE-5311.patch, MAPREDUCE-5311.patch Per discussion in MAPREDUCE-5310 and comments in the code we should remove all the related logic and just leave the counter constant for backwards compatibility and deprecate the counter constants. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5311) Replace SLOTS_MILLIS counters with MEM_MILLIS
[ https://issues.apache.org/jira/browse/MAPREDUCE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738528#comment-13738528 ] Arun C Murthy commented on MAPREDUCE-5311: -- Let's up-level. We agree we can't break compatibility for existing applications ? If so, we can figure ways around how to fix YARN-1004, MAPREDUCE-5310 etc. without breaking compatibility. Replace SLOTS_MILLIS counters with MEM_MILLIS - Key: MAPREDUCE-5311 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5311 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.0.4-alpha Reporter: Alejandro Abdelnur Assignee: Sandy Ryza Priority: Blocker Fix For: 2.1.0-beta Attachments: MAPREDUCE-5311-1.patch, MAPREDUCE-5311.patch, MAPREDUCE-5311.patch Per discussion in MAPREDUCE-5310 and comments in the code we should remove all the related logic and just leave the counter constant for backwards compatibility and deprecate the counter constants. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5311) Replace SLOTS_MILLIS counters with MEM_MILLIS
[ https://issues.apache.org/jira/browse/MAPREDUCE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738663#comment-13738663 ] Arun C Murthy commented on MAPREDUCE-5311: -- bq. As slots no longer exist in MR2, we cannot avoid breaking compatibility for some existing applications. The original intention was to model the old notion of 'slots' as a minimum container. So, we have 2 cases: # mapred.cluster.(map,reduce).memory.mb set to -1 for the job. In this case, we just divide the 'timing' by 1. # mapred.cluster.(map,reduce).memory.mb set to non-zero then we normalize to yarn.scheduler.minimum-allocation-mb and divide 'timing' by that number. Makes sense? Replace SLOTS_MILLIS counters with MEM_MILLIS - Key: MAPREDUCE-5311 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5311 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.0.4-alpha Reporter: Alejandro Abdelnur Assignee: Sandy Ryza Priority: Blocker Fix For: 2.1.0-beta Attachments: MAPREDUCE-5311-1.patch, MAPREDUCE-5311.patch, MAPREDUCE-5311.patch Per discussion in MAPREDUCE-5310 and comments in the code we should remove all the related logic and just leave the counter constant for backwards compatibility and deprecate the counter constants. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5311) Replace SLOTS_MILLIS counters with MEM_MILLIS
[ https://issues.apache.org/jira/browse/MAPREDUCE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13735003#comment-13735003 ] Arun C Murthy commented on MAPREDUCE-5311: -- We seem to understand that we can't remove SLOTS_MILLIS_MAPS - that will break BC. For now, let's punt on this for 2.1.0-beta since we can add new counters (CONTAINERS_MILLIS_MAPS) after a bit more discussion. Replace SLOTS_MILLIS counters with MEM_MILLIS - Key: MAPREDUCE-5311 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5311 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.0.4-alpha Reporter: Alejandro Abdelnur Assignee: Sandy Ryza Priority: Blocker Fix For: 2.1.0-beta Attachments: MAPREDUCE-5311-1.patch, MAPREDUCE-5311.patch, MAPREDUCE-5311.patch Per discussion in MAPREDUCE-5310 and comments in the code we should remove all the related logic and just leave the counter constant for backwards compatibility and deprecate the counter constants. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5311) Replace SLOTS_MILLIS counters with MEM_MILLIS
[ https://issues.apache.org/jira/browse/MAPREDUCE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-5311: - Fix Version/s: (was: 2.1.0-beta) Replace SLOTS_MILLIS counters with MEM_MILLIS - Key: MAPREDUCE-5311 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5311 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.0.4-alpha Reporter: Alejandro Abdelnur Assignee: Sandy Ryza Priority: Blocker Attachments: MAPREDUCE-5311-1.patch, MAPREDUCE-5311.patch, MAPREDUCE-5311.patch Per discussion in MAPREDUCE-5310 and comments in the code we should remove all the related logic and just leave the counter constant for backwards compatibility and deprecate the counter constants. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5311) Replace SLOTS_MILLIS counters with MEM_MILLIS
[ https://issues.apache.org/jira/browse/MAPREDUCE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-5311: - Target Version/s: (was: 2.1.0-beta) Replace SLOTS_MILLIS counters with MEM_MILLIS - Key: MAPREDUCE-5311 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5311 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.0.4-alpha Reporter: Alejandro Abdelnur Assignee: Sandy Ryza Priority: Blocker Fix For: 2.1.0-beta Attachments: MAPREDUCE-5311-1.patch, MAPREDUCE-5311.patch, MAPREDUCE-5311.patch Per discussion in MAPREDUCE-5310 and comments in the code we should remove all the related logic and just leave the counter constant for backwards compatibility and deprecate the counter constants. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5450) Unnecessary Configuration instantiation in IFileInputStream slows down merge - Port to branch-1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13733826#comment-13733826 ] Arun C Murthy commented on MAPREDUCE-5450: -- Sorry, I meant for 1.2.2 (if necessary). Unnecessary Configuration instantiation in IFileInputStream slows down merge - Port to branch-1 --- Key: MAPREDUCE-5450 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5450 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Affects Versions: 1.1.0 Reporter: Stanislav Barton Assignee: Stanislav Barton Priority: Blocker Fix For: 1.3.0 Attachments: MAPREDUCE-5450-1.1.0.txt, mapreduce-5450.txt We are using hadoop-2.0.0+1357-1.cdh4.3.0.p0.21 with MRv1. After upgrade from 4.1.2 to 4.3.0, I have noticed some performance deterioration in our MR job in the Reduce phase. The MR job has usually 10 000 map tasks (10 000 files on input each about 100MB) and 6 000 reducers (one reducer per table region). I was trying to figure out what at which phase the slow down appears (firstly I suspected that the slow gathering of the 1 map output files is the culprit) and found out that the problem is not reading the map output (the shuffle) but the sort/merge phase that follows - the last and actual reduce phase is fast. I have tried to up the io.sort.factor because I thought the lots of small files are being merged on disk, but again upping that to 1000 didnt do any difference. I have then printed the stack trace and found out that the problem is initialization of the org.apache.hadoop.mapred.IFileInputStream namely the creation of the Configuration object which is not propagated along from earlier context, see the stack trace: Thread 13332: (state = IN_NATIVE) - java.io.UnixFileSystem.getBooleanAttributes0(java.io.File) @bci=0 (Compiled frame; information may be imprecise) - java.io.UnixFileSystem.getBooleanAttributes(java.io.File) @bci=2, line=228 (Compiled frame) - java.io.File.exists() @bci=20, line=733 (Compiled frame) - sun.misc.URLClassPath$FileLoader.getResource(java.lang.String, boolean) @bci=136, line=999 (Compiled frame) - sun.misc.URLClassPath$FileLoader.findResource(java.lang.String, boolean) @bci=3, line=966 (Compiled frame) - sun.misc.URLClassPath.findResource(java.lang.String, boolean) @bci=17, line=146 (Compiled frame) - java.net.URLClassLoader$2.run() @bci=12, line=385 (Compiled frame) - java.security.AccessController.doPrivileged(java.security.PrivilegedAction, java.security.AccessControlContext) @bci=0 (Compiled frame) - java.net.URLClassLoader.findResource(java.lang.String) @bci=13, line=382 (Compiled frame) - java.lang.ClassLoader.getResource(java.lang.String) @bci=30, line=1002 (Compiled frame) - java.lang.ClassLoader.getResourceAsStream(java.lang.String) @bci=2, line=1192 (Compiled frame) - javax.xml.parsers.SecuritySupport$4.run() @bci=26, line=96 (Compiled frame) - java.security.AccessController.doPrivileged(java.security.PrivilegedAction) @bci=0 (Compiled frame) - javax.xml.parsers.SecuritySupport.getResourceAsStream(java.lang.ClassLoader, java.lang.String) @bci=10, line=89 (Compiled frame) - javax.xml.parsers.FactoryFinder.findJarServiceProvider(java.lang.String) @bci=38, line=250 (Interpreted frame) - javax.xml.parsers.FactoryFinder.find(java.lang.String, java.lang.String) @bci=273, line=223 (Interpreted frame) - javax.xml.parsers.DocumentBuilderFactory.newInstance() @bci=4, line=123 (Compiled frame) - org.apache.hadoop.conf.Configuration.loadResource(java.util.Properties, org.apache.hadoop.conf.Configuration$Resource, boolean) @bci=16, line=1890 (Compiled frame) - org.apache.hadoop.conf.Configuration.loadResources(java.util.Properties, java.util.ArrayList, boolean) @bci=49, line=1867 (Compiled frame) - org.apache.hadoop.conf.Configuration.getProps() @bci=43, line=1785 (Compiled frame) - org.apache.hadoop.conf.Configuration.get(java.lang.String) @bci=35, line=712 (Compiled frame) - org.apache.hadoop.conf.Configuration.getTrimmed(java.lang.String) @bci=2, line=731 (Compiled frame) - org.apache.hadoop.conf.Configuration.getBoolean(java.lang.String, boolean) @bci=2, line=1047 (Interpreted frame) - org.apache.hadoop.mapred.IFileInputStream.init(java.io.InputStream, long, org.apache.hadoop.conf.Configuration) @bci=111, line=93 (Interpreted frame) - org.apache.hadoop.mapred.IFile$Reader.init(org.apache.hadoop.conf.Configuration, org.apache.hadoop.fs.FSDataInputStream, long, org.apache.hadoop.io.compress.CompressionCodec, org.apache.hadoop.mapred.Counters$Counter) @bci=60, line=303 (Interpreted frame) -
[jira] [Commented] (MAPREDUCE-5450) Unnecessary Configuration instantiation in IFileInputStream slows down merge - Port to branch-1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13733823#comment-13733823 ] Arun C Murthy commented on MAPREDUCE-5450: -- Let's get [~stanislav.barton]'s original patch into branch-1.2 for 1.2.1. Any objections? Unnecessary Configuration instantiation in IFileInputStream slows down merge - Port to branch-1 --- Key: MAPREDUCE-5450 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5450 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Affects Versions: 1.1.0 Reporter: Stanislav Barton Assignee: Stanislav Barton Priority: Blocker Fix For: 1.3.0 Attachments: MAPREDUCE-5450-1.1.0.txt, mapreduce-5450.txt We are using hadoop-2.0.0+1357-1.cdh4.3.0.p0.21 with MRv1. After upgrade from 4.1.2 to 4.3.0, I have noticed some performance deterioration in our MR job in the Reduce phase. The MR job has usually 10 000 map tasks (10 000 files on input each about 100MB) and 6 000 reducers (one reducer per table region). I was trying to figure out what at which phase the slow down appears (firstly I suspected that the slow gathering of the 1 map output files is the culprit) and found out that the problem is not reading the map output (the shuffle) but the sort/merge phase that follows - the last and actual reduce phase is fast. I have tried to up the io.sort.factor because I thought the lots of small files are being merged on disk, but again upping that to 1000 didnt do any difference. I have then printed the stack trace and found out that the problem is initialization of the org.apache.hadoop.mapred.IFileInputStream namely the creation of the Configuration object which is not propagated along from earlier context, see the stack trace: Thread 13332: (state = IN_NATIVE) - java.io.UnixFileSystem.getBooleanAttributes0(java.io.File) @bci=0 (Compiled frame; information may be imprecise) - java.io.UnixFileSystem.getBooleanAttributes(java.io.File) @bci=2, line=228 (Compiled frame) - java.io.File.exists() @bci=20, line=733 (Compiled frame) - sun.misc.URLClassPath$FileLoader.getResource(java.lang.String, boolean) @bci=136, line=999 (Compiled frame) - sun.misc.URLClassPath$FileLoader.findResource(java.lang.String, boolean) @bci=3, line=966 (Compiled frame) - sun.misc.URLClassPath.findResource(java.lang.String, boolean) @bci=17, line=146 (Compiled frame) - java.net.URLClassLoader$2.run() @bci=12, line=385 (Compiled frame) - java.security.AccessController.doPrivileged(java.security.PrivilegedAction, java.security.AccessControlContext) @bci=0 (Compiled frame) - java.net.URLClassLoader.findResource(java.lang.String) @bci=13, line=382 (Compiled frame) - java.lang.ClassLoader.getResource(java.lang.String) @bci=30, line=1002 (Compiled frame) - java.lang.ClassLoader.getResourceAsStream(java.lang.String) @bci=2, line=1192 (Compiled frame) - javax.xml.parsers.SecuritySupport$4.run() @bci=26, line=96 (Compiled frame) - java.security.AccessController.doPrivileged(java.security.PrivilegedAction) @bci=0 (Compiled frame) - javax.xml.parsers.SecuritySupport.getResourceAsStream(java.lang.ClassLoader, java.lang.String) @bci=10, line=89 (Compiled frame) - javax.xml.parsers.FactoryFinder.findJarServiceProvider(java.lang.String) @bci=38, line=250 (Interpreted frame) - javax.xml.parsers.FactoryFinder.find(java.lang.String, java.lang.String) @bci=273, line=223 (Interpreted frame) - javax.xml.parsers.DocumentBuilderFactory.newInstance() @bci=4, line=123 (Compiled frame) - org.apache.hadoop.conf.Configuration.loadResource(java.util.Properties, org.apache.hadoop.conf.Configuration$Resource, boolean) @bci=16, line=1890 (Compiled frame) - org.apache.hadoop.conf.Configuration.loadResources(java.util.Properties, java.util.ArrayList, boolean) @bci=49, line=1867 (Compiled frame) - org.apache.hadoop.conf.Configuration.getProps() @bci=43, line=1785 (Compiled frame) - org.apache.hadoop.conf.Configuration.get(java.lang.String) @bci=35, line=712 (Compiled frame) - org.apache.hadoop.conf.Configuration.getTrimmed(java.lang.String) @bci=2, line=731 (Compiled frame) - org.apache.hadoop.conf.Configuration.getBoolean(java.lang.String, boolean) @bci=2, line=1047 (Interpreted frame) - org.apache.hadoop.mapred.IFileInputStream.init(java.io.InputStream, long, org.apache.hadoop.conf.Configuration) @bci=111, line=93 (Interpreted frame) - org.apache.hadoop.mapred.IFile$Reader.init(org.apache.hadoop.conf.Configuration, org.apache.hadoop.fs.FSDataInputStream, long, org.apache.hadoop.io.compress.CompressionCodec,
[jira] [Updated] (MAPREDUCE-5450) Unnecessary Configuration instantiation in IFileInputStream slows down merge - Port to branch-1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-5450: - Fix Version/s: (was: 1.3.0) 1.2.1 Unnecessary Configuration instantiation in IFileInputStream slows down merge - Port to branch-1 --- Key: MAPREDUCE-5450 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5450 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Affects Versions: 1.1.0 Reporter: Stanislav Barton Assignee: Stanislav Barton Priority: Blocker Fix For: 1.2.1 Attachments: MAPREDUCE-5450-1.1.0.txt, mapreduce-5450.txt We are using hadoop-2.0.0+1357-1.cdh4.3.0.p0.21 with MRv1. After upgrade from 4.1.2 to 4.3.0, I have noticed some performance deterioration in our MR job in the Reduce phase. The MR job has usually 10 000 map tasks (10 000 files on input each about 100MB) and 6 000 reducers (one reducer per table region). I was trying to figure out what at which phase the slow down appears (firstly I suspected that the slow gathering of the 1 map output files is the culprit) and found out that the problem is not reading the map output (the shuffle) but the sort/merge phase that follows - the last and actual reduce phase is fast. I have tried to up the io.sort.factor because I thought the lots of small files are being merged on disk, but again upping that to 1000 didnt do any difference. I have then printed the stack trace and found out that the problem is initialization of the org.apache.hadoop.mapred.IFileInputStream namely the creation of the Configuration object which is not propagated along from earlier context, see the stack trace: Thread 13332: (state = IN_NATIVE) - java.io.UnixFileSystem.getBooleanAttributes0(java.io.File) @bci=0 (Compiled frame; information may be imprecise) - java.io.UnixFileSystem.getBooleanAttributes(java.io.File) @bci=2, line=228 (Compiled frame) - java.io.File.exists() @bci=20, line=733 (Compiled frame) - sun.misc.URLClassPath$FileLoader.getResource(java.lang.String, boolean) @bci=136, line=999 (Compiled frame) - sun.misc.URLClassPath$FileLoader.findResource(java.lang.String, boolean) @bci=3, line=966 (Compiled frame) - sun.misc.URLClassPath.findResource(java.lang.String, boolean) @bci=17, line=146 (Compiled frame) - java.net.URLClassLoader$2.run() @bci=12, line=385 (Compiled frame) - java.security.AccessController.doPrivileged(java.security.PrivilegedAction, java.security.AccessControlContext) @bci=0 (Compiled frame) - java.net.URLClassLoader.findResource(java.lang.String) @bci=13, line=382 (Compiled frame) - java.lang.ClassLoader.getResource(java.lang.String) @bci=30, line=1002 (Compiled frame) - java.lang.ClassLoader.getResourceAsStream(java.lang.String) @bci=2, line=1192 (Compiled frame) - javax.xml.parsers.SecuritySupport$4.run() @bci=26, line=96 (Compiled frame) - java.security.AccessController.doPrivileged(java.security.PrivilegedAction) @bci=0 (Compiled frame) - javax.xml.parsers.SecuritySupport.getResourceAsStream(java.lang.ClassLoader, java.lang.String) @bci=10, line=89 (Compiled frame) - javax.xml.parsers.FactoryFinder.findJarServiceProvider(java.lang.String) @bci=38, line=250 (Interpreted frame) - javax.xml.parsers.FactoryFinder.find(java.lang.String, java.lang.String) @bci=273, line=223 (Interpreted frame) - javax.xml.parsers.DocumentBuilderFactory.newInstance() @bci=4, line=123 (Compiled frame) - org.apache.hadoop.conf.Configuration.loadResource(java.util.Properties, org.apache.hadoop.conf.Configuration$Resource, boolean) @bci=16, line=1890 (Compiled frame) - org.apache.hadoop.conf.Configuration.loadResources(java.util.Properties, java.util.ArrayList, boolean) @bci=49, line=1867 (Compiled frame) - org.apache.hadoop.conf.Configuration.getProps() @bci=43, line=1785 (Compiled frame) - org.apache.hadoop.conf.Configuration.get(java.lang.String) @bci=35, line=712 (Compiled frame) - org.apache.hadoop.conf.Configuration.getTrimmed(java.lang.String) @bci=2, line=731 (Compiled frame) - org.apache.hadoop.conf.Configuration.getBoolean(java.lang.String, boolean) @bci=2, line=1047 (Interpreted frame) - org.apache.hadoop.mapred.IFileInputStream.init(java.io.InputStream, long, org.apache.hadoop.conf.Configuration) @bci=111, line=93 (Interpreted frame) - org.apache.hadoop.mapred.IFile$Reader.init(org.apache.hadoop.conf.Configuration, org.apache.hadoop.fs.FSDataInputStream, long, org.apache.hadoop.io.compress.CompressionCodec, org.apache.hadoop.mapred.Counters$Counter) @bci=60, line=303 (Interpreted frame) -
[jira] [Commented] (MAPREDUCE-5450) Unnecessary Configuration instantiation in IFileInputStream slows down merge - Port to branch-1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13733834#comment-13733834 ] Arun C Murthy commented on MAPREDUCE-5450: -- I merged this to branch-1.2 too. Thanks guys! Unnecessary Configuration instantiation in IFileInputStream slows down merge - Port to branch-1 --- Key: MAPREDUCE-5450 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5450 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Affects Versions: 1.1.0 Reporter: Stanislav Barton Assignee: Stanislav Barton Priority: Blocker Fix For: 1.2.1 Attachments: MAPREDUCE-5450-1.1.0.txt, mapreduce-5450.txt We are using hadoop-2.0.0+1357-1.cdh4.3.0.p0.21 with MRv1. After upgrade from 4.1.2 to 4.3.0, I have noticed some performance deterioration in our MR job in the Reduce phase. The MR job has usually 10 000 map tasks (10 000 files on input each about 100MB) and 6 000 reducers (one reducer per table region). I was trying to figure out what at which phase the slow down appears (firstly I suspected that the slow gathering of the 1 map output files is the culprit) and found out that the problem is not reading the map output (the shuffle) but the sort/merge phase that follows - the last and actual reduce phase is fast. I have tried to up the io.sort.factor because I thought the lots of small files are being merged on disk, but again upping that to 1000 didnt do any difference. I have then printed the stack trace and found out that the problem is initialization of the org.apache.hadoop.mapred.IFileInputStream namely the creation of the Configuration object which is not propagated along from earlier context, see the stack trace: Thread 13332: (state = IN_NATIVE) - java.io.UnixFileSystem.getBooleanAttributes0(java.io.File) @bci=0 (Compiled frame; information may be imprecise) - java.io.UnixFileSystem.getBooleanAttributes(java.io.File) @bci=2, line=228 (Compiled frame) - java.io.File.exists() @bci=20, line=733 (Compiled frame) - sun.misc.URLClassPath$FileLoader.getResource(java.lang.String, boolean) @bci=136, line=999 (Compiled frame) - sun.misc.URLClassPath$FileLoader.findResource(java.lang.String, boolean) @bci=3, line=966 (Compiled frame) - sun.misc.URLClassPath.findResource(java.lang.String, boolean) @bci=17, line=146 (Compiled frame) - java.net.URLClassLoader$2.run() @bci=12, line=385 (Compiled frame) - java.security.AccessController.doPrivileged(java.security.PrivilegedAction, java.security.AccessControlContext) @bci=0 (Compiled frame) - java.net.URLClassLoader.findResource(java.lang.String) @bci=13, line=382 (Compiled frame) - java.lang.ClassLoader.getResource(java.lang.String) @bci=30, line=1002 (Compiled frame) - java.lang.ClassLoader.getResourceAsStream(java.lang.String) @bci=2, line=1192 (Compiled frame) - javax.xml.parsers.SecuritySupport$4.run() @bci=26, line=96 (Compiled frame) - java.security.AccessController.doPrivileged(java.security.PrivilegedAction) @bci=0 (Compiled frame) - javax.xml.parsers.SecuritySupport.getResourceAsStream(java.lang.ClassLoader, java.lang.String) @bci=10, line=89 (Compiled frame) - javax.xml.parsers.FactoryFinder.findJarServiceProvider(java.lang.String) @bci=38, line=250 (Interpreted frame) - javax.xml.parsers.FactoryFinder.find(java.lang.String, java.lang.String) @bci=273, line=223 (Interpreted frame) - javax.xml.parsers.DocumentBuilderFactory.newInstance() @bci=4, line=123 (Compiled frame) - org.apache.hadoop.conf.Configuration.loadResource(java.util.Properties, org.apache.hadoop.conf.Configuration$Resource, boolean) @bci=16, line=1890 (Compiled frame) - org.apache.hadoop.conf.Configuration.loadResources(java.util.Properties, java.util.ArrayList, boolean) @bci=49, line=1867 (Compiled frame) - org.apache.hadoop.conf.Configuration.getProps() @bci=43, line=1785 (Compiled frame) - org.apache.hadoop.conf.Configuration.get(java.lang.String) @bci=35, line=712 (Compiled frame) - org.apache.hadoop.conf.Configuration.getTrimmed(java.lang.String) @bci=2, line=731 (Compiled frame) - org.apache.hadoop.conf.Configuration.getBoolean(java.lang.String, boolean) @bci=2, line=1047 (Interpreted frame) - org.apache.hadoop.mapred.IFileInputStream.init(java.io.InputStream, long, org.apache.hadoop.conf.Configuration) @bci=111, line=93 (Interpreted frame) - org.apache.hadoop.mapred.IFile$Reader.init(org.apache.hadoop.conf.Configuration, org.apache.hadoop.fs.FSDataInputStream, long, org.apache.hadoop.io.compress.CompressionCodec, org.apache.hadoop.mapred.Counters$Counter) @bci=60, line=303 (Interpreted frame)
[jira] [Updated] (MAPREDUCE-5450) Unnecessary Configuration instantiation in IFileInputStream slows down merge - Port to branch-1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-5450: - Fix Version/s: (was: 1.2.1) 1.2.2 Unnecessary Configuration instantiation in IFileInputStream slows down merge - Port to branch-1 --- Key: MAPREDUCE-5450 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5450 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Affects Versions: 1.1.0 Reporter: Stanislav Barton Assignee: Stanislav Barton Priority: Blocker Fix For: 1.2.2 Attachments: MAPREDUCE-5450-1.1.0.txt, mapreduce-5450.txt We are using hadoop-2.0.0+1357-1.cdh4.3.0.p0.21 with MRv1. After upgrade from 4.1.2 to 4.3.0, I have noticed some performance deterioration in our MR job in the Reduce phase. The MR job has usually 10 000 map tasks (10 000 files on input each about 100MB) and 6 000 reducers (one reducer per table region). I was trying to figure out what at which phase the slow down appears (firstly I suspected that the slow gathering of the 1 map output files is the culprit) and found out that the problem is not reading the map output (the shuffle) but the sort/merge phase that follows - the last and actual reduce phase is fast. I have tried to up the io.sort.factor because I thought the lots of small files are being merged on disk, but again upping that to 1000 didnt do any difference. I have then printed the stack trace and found out that the problem is initialization of the org.apache.hadoop.mapred.IFileInputStream namely the creation of the Configuration object which is not propagated along from earlier context, see the stack trace: Thread 13332: (state = IN_NATIVE) - java.io.UnixFileSystem.getBooleanAttributes0(java.io.File) @bci=0 (Compiled frame; information may be imprecise) - java.io.UnixFileSystem.getBooleanAttributes(java.io.File) @bci=2, line=228 (Compiled frame) - java.io.File.exists() @bci=20, line=733 (Compiled frame) - sun.misc.URLClassPath$FileLoader.getResource(java.lang.String, boolean) @bci=136, line=999 (Compiled frame) - sun.misc.URLClassPath$FileLoader.findResource(java.lang.String, boolean) @bci=3, line=966 (Compiled frame) - sun.misc.URLClassPath.findResource(java.lang.String, boolean) @bci=17, line=146 (Compiled frame) - java.net.URLClassLoader$2.run() @bci=12, line=385 (Compiled frame) - java.security.AccessController.doPrivileged(java.security.PrivilegedAction, java.security.AccessControlContext) @bci=0 (Compiled frame) - java.net.URLClassLoader.findResource(java.lang.String) @bci=13, line=382 (Compiled frame) - java.lang.ClassLoader.getResource(java.lang.String) @bci=30, line=1002 (Compiled frame) - java.lang.ClassLoader.getResourceAsStream(java.lang.String) @bci=2, line=1192 (Compiled frame) - javax.xml.parsers.SecuritySupport$4.run() @bci=26, line=96 (Compiled frame) - java.security.AccessController.doPrivileged(java.security.PrivilegedAction) @bci=0 (Compiled frame) - javax.xml.parsers.SecuritySupport.getResourceAsStream(java.lang.ClassLoader, java.lang.String) @bci=10, line=89 (Compiled frame) - javax.xml.parsers.FactoryFinder.findJarServiceProvider(java.lang.String) @bci=38, line=250 (Interpreted frame) - javax.xml.parsers.FactoryFinder.find(java.lang.String, java.lang.String) @bci=273, line=223 (Interpreted frame) - javax.xml.parsers.DocumentBuilderFactory.newInstance() @bci=4, line=123 (Compiled frame) - org.apache.hadoop.conf.Configuration.loadResource(java.util.Properties, org.apache.hadoop.conf.Configuration$Resource, boolean) @bci=16, line=1890 (Compiled frame) - org.apache.hadoop.conf.Configuration.loadResources(java.util.Properties, java.util.ArrayList, boolean) @bci=49, line=1867 (Compiled frame) - org.apache.hadoop.conf.Configuration.getProps() @bci=43, line=1785 (Compiled frame) - org.apache.hadoop.conf.Configuration.get(java.lang.String) @bci=35, line=712 (Compiled frame) - org.apache.hadoop.conf.Configuration.getTrimmed(java.lang.String) @bci=2, line=731 (Compiled frame) - org.apache.hadoop.conf.Configuration.getBoolean(java.lang.String, boolean) @bci=2, line=1047 (Interpreted frame) - org.apache.hadoop.mapred.IFileInputStream.init(java.io.InputStream, long, org.apache.hadoop.conf.Configuration) @bci=111, line=93 (Interpreted frame) - org.apache.hadoop.mapred.IFile$Reader.init(org.apache.hadoop.conf.Configuration, org.apache.hadoop.fs.FSDataInputStream, long, org.apache.hadoop.io.compress.CompressionCodec, org.apache.hadoop.mapred.Counters$Counter) @bci=60, line=303 (Interpreted frame) -
[jira] [Commented] (MAPREDUCE-5311) Replace SLOTS_MILLIS counters with MEM_MILLIS
[ https://issues.apache.org/jira/browse/MAPREDUCE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13731444#comment-13731444 ] Arun C Murthy commented on MAPREDUCE-5311: -- This will break every single MR app. I'm -1 on this. Replace SLOTS_MILLIS counters with MEM_MILLIS - Key: MAPREDUCE-5311 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5311 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.0.4-alpha Reporter: Alejandro Abdelnur Assignee: Sandy Ryza Priority: Blocker Fix For: 2.1.0-beta Attachments: MAPREDUCE-5311-1.patch, MAPREDUCE-5311.patch, MAPREDUCE-5311.patch Per discussion in MAPREDUCE-5310 and comments in the code we should remove all the related logic and just leave the counter constant for backwards compatibility and deprecate the counter constants. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5311) Replace SLOTS_MILLIS counters with MEM_MILLIS
[ https://issues.apache.org/jira/browse/MAPREDUCE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13731445#comment-13731445 ] Arun C Murthy commented on MAPREDUCE-5311: -- To Sandy's point - several folks do use mapred.cluster.(map,reduce).memory.mb, so it will be a big BC break for them. Also, what does this mean for BC after we spent all the time fixing MAPREDUCE-5108. Replace SLOTS_MILLIS counters with MEM_MILLIS - Key: MAPREDUCE-5311 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5311 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.0.4-alpha Reporter: Alejandro Abdelnur Assignee: Sandy Ryza Priority: Blocker Fix For: 2.1.0-beta Attachments: MAPREDUCE-5311-1.patch, MAPREDUCE-5311.patch, MAPREDUCE-5311.patch Per discussion in MAPREDUCE-5310 and comments in the code we should remove all the related logic and just leave the counter constant for backwards compatibility and deprecate the counter constants. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5399) Large number of map tasks cause slow sort at reduce phase, invariant to amount of data to sort
[ https://issues.apache.org/jira/browse/MAPREDUCE-5399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730102#comment-13730102 ] Arun C Murthy commented on MAPREDUCE-5399: -- [~sandyr] Can you pls commit this to trunk, branch-2, branch-2.1-beta and branch-2.1.0-beta? Thanks! Large number of map tasks cause slow sort at reduce phase, invariant to amount of data to sort -- Key: MAPREDUCE-5399 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5399 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1, mrv2 Affects Versions: 1.1.0, 2.0.2-alpha Reporter: Stanislav Barton Assignee: Stanislav Barton Priority: Blocker Attachments: MAPREDUCE-5399.patch We are using hadoop-2.0.0+1357-1.cdh4.3.0.p0.21 with MRv1. After upgrade from 4.1.2 to 4.3.0, I have noticed some performance deterioration in our MR job in the Reduce phase. The MR job has usually 10 000 map tasks (10 000 files on input each about 100MB) and 6 000 reducers (one reducer per table region). I was trying to figure out what at which phase the slow down appears (firstly I suspected that the slow gathering of the 1 map output files is the culprit) and found out that the problem is not reading the map output (the shuffle) but the sort/merge phase that follows - the last and actual reduce phase is fast. I have tried to up the io.sort.factor because I thought the lots of small files are being merged on disk, but again upping that to 1000 didnt do any difference. I have then printed the stack trace and found out that the problem is initialization of the org.apache.hadoop.mapred.IFileInputStream namely the creation of the Configuration object which is not propagated along from earlier context, see the stack trace: Thread 13332: (state = IN_NATIVE) - java.io.UnixFileSystem.getBooleanAttributes0(java.io.File) @bci=0 (Compiled frame; information may be imprecise) - java.io.UnixFileSystem.getBooleanAttributes(java.io.File) @bci=2, line=228 (Compiled frame) - java.io.File.exists() @bci=20, line=733 (Compiled frame) - sun.misc.URLClassPath$FileLoader.getResource(java.lang.String, boolean) @bci=136, line=999 (Compiled frame) - sun.misc.URLClassPath$FileLoader.findResource(java.lang.String, boolean) @bci=3, line=966 (Compiled frame) - sun.misc.URLClassPath.findResource(java.lang.String, boolean) @bci=17, line=146 (Compiled frame) - java.net.URLClassLoader$2.run() @bci=12, line=385 (Compiled frame) - java.security.AccessController.doPrivileged(java.security.PrivilegedAction, java.security.AccessControlContext) @bci=0 (Compiled frame) - java.net.URLClassLoader.findResource(java.lang.String) @bci=13, line=382 (Compiled frame) - java.lang.ClassLoader.getResource(java.lang.String) @bci=30, line=1002 (Compiled frame) - java.lang.ClassLoader.getResourceAsStream(java.lang.String) @bci=2, line=1192 (Compiled frame) - javax.xml.parsers.SecuritySupport$4.run() @bci=26, line=96 (Compiled frame) - java.security.AccessController.doPrivileged(java.security.PrivilegedAction) @bci=0 (Compiled frame) - javax.xml.parsers.SecuritySupport.getResourceAsStream(java.lang.ClassLoader, java.lang.String) @bci=10, line=89 (Compiled frame) - javax.xml.parsers.FactoryFinder.findJarServiceProvider(java.lang.String) @bci=38, line=250 (Interpreted frame) - javax.xml.parsers.FactoryFinder.find(java.lang.String, java.lang.String) @bci=273, line=223 (Interpreted frame) - javax.xml.parsers.DocumentBuilderFactory.newInstance() @bci=4, line=123 (Compiled frame) - org.apache.hadoop.conf.Configuration.loadResource(java.util.Properties, org.apache.hadoop.conf.Configuration$Resource, boolean) @bci=16, line=1890 (Compiled frame) - org.apache.hadoop.conf.Configuration.loadResources(java.util.Properties, java.util.ArrayList, boolean) @bci=49, line=1867 (Compiled frame) - org.apache.hadoop.conf.Configuration.getProps() @bci=43, line=1785 (Compiled frame) - org.apache.hadoop.conf.Configuration.get(java.lang.String) @bci=35, line=712 (Compiled frame) - org.apache.hadoop.conf.Configuration.getTrimmed(java.lang.String) @bci=2, line=731 (Compiled frame) - org.apache.hadoop.conf.Configuration.getBoolean(java.lang.String, boolean) @bci=2, line=1047 (Interpreted frame) - org.apache.hadoop.mapred.IFileInputStream.init(java.io.InputStream, long, org.apache.hadoop.conf.Configuration) @bci=111, line=93 (Interpreted frame) - org.apache.hadoop.mapred.IFile$Reader.init(org.apache.hadoop.conf.Configuration, org.apache.hadoop.fs.FSDataInputStream, long, org.apache.hadoop.io.compress.CompressionCodec, org.apache.hadoop.mapred.Counters$Counter) @bci=60, line=303
[jira] [Updated] (MAPREDUCE-5399) Large number of map tasks cause slow sort at reduce phase, invariant to amount of data to sort
[ https://issues.apache.org/jira/browse/MAPREDUCE-5399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-5399: - Target Version/s: 2.1.0-beta Large number of map tasks cause slow sort at reduce phase, invariant to amount of data to sort -- Key: MAPREDUCE-5399 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5399 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1, mrv2 Affects Versions: 1.1.0, 2.0.2-alpha Reporter: Stanislav Barton Assignee: Stanislav Barton Priority: Blocker Attachments: MAPREDUCE-5399.patch We are using hadoop-2.0.0+1357-1.cdh4.3.0.p0.21 with MRv1. After upgrade from 4.1.2 to 4.3.0, I have noticed some performance deterioration in our MR job in the Reduce phase. The MR job has usually 10 000 map tasks (10 000 files on input each about 100MB) and 6 000 reducers (one reducer per table region). I was trying to figure out what at which phase the slow down appears (firstly I suspected that the slow gathering of the 1 map output files is the culprit) and found out that the problem is not reading the map output (the shuffle) but the sort/merge phase that follows - the last and actual reduce phase is fast. I have tried to up the io.sort.factor because I thought the lots of small files are being merged on disk, but again upping that to 1000 didnt do any difference. I have then printed the stack trace and found out that the problem is initialization of the org.apache.hadoop.mapred.IFileInputStream namely the creation of the Configuration object which is not propagated along from earlier context, see the stack trace: Thread 13332: (state = IN_NATIVE) - java.io.UnixFileSystem.getBooleanAttributes0(java.io.File) @bci=0 (Compiled frame; information may be imprecise) - java.io.UnixFileSystem.getBooleanAttributes(java.io.File) @bci=2, line=228 (Compiled frame) - java.io.File.exists() @bci=20, line=733 (Compiled frame) - sun.misc.URLClassPath$FileLoader.getResource(java.lang.String, boolean) @bci=136, line=999 (Compiled frame) - sun.misc.URLClassPath$FileLoader.findResource(java.lang.String, boolean) @bci=3, line=966 (Compiled frame) - sun.misc.URLClassPath.findResource(java.lang.String, boolean) @bci=17, line=146 (Compiled frame) - java.net.URLClassLoader$2.run() @bci=12, line=385 (Compiled frame) - java.security.AccessController.doPrivileged(java.security.PrivilegedAction, java.security.AccessControlContext) @bci=0 (Compiled frame) - java.net.URLClassLoader.findResource(java.lang.String) @bci=13, line=382 (Compiled frame) - java.lang.ClassLoader.getResource(java.lang.String) @bci=30, line=1002 (Compiled frame) - java.lang.ClassLoader.getResourceAsStream(java.lang.String) @bci=2, line=1192 (Compiled frame) - javax.xml.parsers.SecuritySupport$4.run() @bci=26, line=96 (Compiled frame) - java.security.AccessController.doPrivileged(java.security.PrivilegedAction) @bci=0 (Compiled frame) - javax.xml.parsers.SecuritySupport.getResourceAsStream(java.lang.ClassLoader, java.lang.String) @bci=10, line=89 (Compiled frame) - javax.xml.parsers.FactoryFinder.findJarServiceProvider(java.lang.String) @bci=38, line=250 (Interpreted frame) - javax.xml.parsers.FactoryFinder.find(java.lang.String, java.lang.String) @bci=273, line=223 (Interpreted frame) - javax.xml.parsers.DocumentBuilderFactory.newInstance() @bci=4, line=123 (Compiled frame) - org.apache.hadoop.conf.Configuration.loadResource(java.util.Properties, org.apache.hadoop.conf.Configuration$Resource, boolean) @bci=16, line=1890 (Compiled frame) - org.apache.hadoop.conf.Configuration.loadResources(java.util.Properties, java.util.ArrayList, boolean) @bci=49, line=1867 (Compiled frame) - org.apache.hadoop.conf.Configuration.getProps() @bci=43, line=1785 (Compiled frame) - org.apache.hadoop.conf.Configuration.get(java.lang.String) @bci=35, line=712 (Compiled frame) - org.apache.hadoop.conf.Configuration.getTrimmed(java.lang.String) @bci=2, line=731 (Compiled frame) - org.apache.hadoop.conf.Configuration.getBoolean(java.lang.String, boolean) @bci=2, line=1047 (Interpreted frame) - org.apache.hadoop.mapred.IFileInputStream.init(java.io.InputStream, long, org.apache.hadoop.conf.Configuration) @bci=111, line=93 (Interpreted frame) - org.apache.hadoop.mapred.IFile$Reader.init(org.apache.hadoop.conf.Configuration, org.apache.hadoop.fs.FSDataInputStream, long, org.apache.hadoop.io.compress.CompressionCodec, org.apache.hadoop.mapred.Counters$Counter) @bci=60, line=303 (Interpreted frame) - org.apache.hadoop.mapred.IFile$InMemoryReader.init(org.apache.hadoop.mapred.RamManager,
[jira] [Updated] (MAPREDUCE-5450) Unnecessary Configuration instantiation in IFileInputStream slows down merge - Port to branch-1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-5450: - Component/s: (was: mrv2) Unnecessary Configuration instantiation in IFileInputStream slows down merge - Port to branch-1 --- Key: MAPREDUCE-5450 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5450 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Affects Versions: 1.1.0, 2.0.2-alpha Reporter: Stanislav Barton Assignee: Stanislav Barton Priority: Blocker Fix For: 2.1.0-beta We are using hadoop-2.0.0+1357-1.cdh4.3.0.p0.21 with MRv1. After upgrade from 4.1.2 to 4.3.0, I have noticed some performance deterioration in our MR job in the Reduce phase. The MR job has usually 10 000 map tasks (10 000 files on input each about 100MB) and 6 000 reducers (one reducer per table region). I was trying to figure out what at which phase the slow down appears (firstly I suspected that the slow gathering of the 1 map output files is the culprit) and found out that the problem is not reading the map output (the shuffle) but the sort/merge phase that follows - the last and actual reduce phase is fast. I have tried to up the io.sort.factor because I thought the lots of small files are being merged on disk, but again upping that to 1000 didnt do any difference. I have then printed the stack trace and found out that the problem is initialization of the org.apache.hadoop.mapred.IFileInputStream namely the creation of the Configuration object which is not propagated along from earlier context, see the stack trace: Thread 13332: (state = IN_NATIVE) - java.io.UnixFileSystem.getBooleanAttributes0(java.io.File) @bci=0 (Compiled frame; information may be imprecise) - java.io.UnixFileSystem.getBooleanAttributes(java.io.File) @bci=2, line=228 (Compiled frame) - java.io.File.exists() @bci=20, line=733 (Compiled frame) - sun.misc.URLClassPath$FileLoader.getResource(java.lang.String, boolean) @bci=136, line=999 (Compiled frame) - sun.misc.URLClassPath$FileLoader.findResource(java.lang.String, boolean) @bci=3, line=966 (Compiled frame) - sun.misc.URLClassPath.findResource(java.lang.String, boolean) @bci=17, line=146 (Compiled frame) - java.net.URLClassLoader$2.run() @bci=12, line=385 (Compiled frame) - java.security.AccessController.doPrivileged(java.security.PrivilegedAction, java.security.AccessControlContext) @bci=0 (Compiled frame) - java.net.URLClassLoader.findResource(java.lang.String) @bci=13, line=382 (Compiled frame) - java.lang.ClassLoader.getResource(java.lang.String) @bci=30, line=1002 (Compiled frame) - java.lang.ClassLoader.getResourceAsStream(java.lang.String) @bci=2, line=1192 (Compiled frame) - javax.xml.parsers.SecuritySupport$4.run() @bci=26, line=96 (Compiled frame) - java.security.AccessController.doPrivileged(java.security.PrivilegedAction) @bci=0 (Compiled frame) - javax.xml.parsers.SecuritySupport.getResourceAsStream(java.lang.ClassLoader, java.lang.String) @bci=10, line=89 (Compiled frame) - javax.xml.parsers.FactoryFinder.findJarServiceProvider(java.lang.String) @bci=38, line=250 (Interpreted frame) - javax.xml.parsers.FactoryFinder.find(java.lang.String, java.lang.String) @bci=273, line=223 (Interpreted frame) - javax.xml.parsers.DocumentBuilderFactory.newInstance() @bci=4, line=123 (Compiled frame) - org.apache.hadoop.conf.Configuration.loadResource(java.util.Properties, org.apache.hadoop.conf.Configuration$Resource, boolean) @bci=16, line=1890 (Compiled frame) - org.apache.hadoop.conf.Configuration.loadResources(java.util.Properties, java.util.ArrayList, boolean) @bci=49, line=1867 (Compiled frame) - org.apache.hadoop.conf.Configuration.getProps() @bci=43, line=1785 (Compiled frame) - org.apache.hadoop.conf.Configuration.get(java.lang.String) @bci=35, line=712 (Compiled frame) - org.apache.hadoop.conf.Configuration.getTrimmed(java.lang.String) @bci=2, line=731 (Compiled frame) - org.apache.hadoop.conf.Configuration.getBoolean(java.lang.String, boolean) @bci=2, line=1047 (Interpreted frame) - org.apache.hadoop.mapred.IFileInputStream.init(java.io.InputStream, long, org.apache.hadoop.conf.Configuration) @bci=111, line=93 (Interpreted frame) - org.apache.hadoop.mapred.IFile$Reader.init(org.apache.hadoop.conf.Configuration, org.apache.hadoop.fs.FSDataInputStream, long, org.apache.hadoop.io.compress.CompressionCodec, org.apache.hadoop.mapred.Counters$Counter) @bci=60, line=303 (Interpreted frame) - org.apache.hadoop.mapred.IFile$InMemoryReader.init(org.apache.hadoop.mapred.RamManager,
[jira] [Updated] (MAPREDUCE-5450) Unnecessary Configuration instantiation in IFileInputStream slows down merge - Port to branch-1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-5450: - Affects Version/s: (was: 2.0.2-alpha) Unnecessary Configuration instantiation in IFileInputStream slows down merge - Port to branch-1 --- Key: MAPREDUCE-5450 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5450 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Affects Versions: 1.1.0 Reporter: Stanislav Barton Assignee: Stanislav Barton Priority: Blocker Fix For: 2.1.0-beta We are using hadoop-2.0.0+1357-1.cdh4.3.0.p0.21 with MRv1. After upgrade from 4.1.2 to 4.3.0, I have noticed some performance deterioration in our MR job in the Reduce phase. The MR job has usually 10 000 map tasks (10 000 files on input each about 100MB) and 6 000 reducers (one reducer per table region). I was trying to figure out what at which phase the slow down appears (firstly I suspected that the slow gathering of the 1 map output files is the culprit) and found out that the problem is not reading the map output (the shuffle) but the sort/merge phase that follows - the last and actual reduce phase is fast. I have tried to up the io.sort.factor because I thought the lots of small files are being merged on disk, but again upping that to 1000 didnt do any difference. I have then printed the stack trace and found out that the problem is initialization of the org.apache.hadoop.mapred.IFileInputStream namely the creation of the Configuration object which is not propagated along from earlier context, see the stack trace: Thread 13332: (state = IN_NATIVE) - java.io.UnixFileSystem.getBooleanAttributes0(java.io.File) @bci=0 (Compiled frame; information may be imprecise) - java.io.UnixFileSystem.getBooleanAttributes(java.io.File) @bci=2, line=228 (Compiled frame) - java.io.File.exists() @bci=20, line=733 (Compiled frame) - sun.misc.URLClassPath$FileLoader.getResource(java.lang.String, boolean) @bci=136, line=999 (Compiled frame) - sun.misc.URLClassPath$FileLoader.findResource(java.lang.String, boolean) @bci=3, line=966 (Compiled frame) - sun.misc.URLClassPath.findResource(java.lang.String, boolean) @bci=17, line=146 (Compiled frame) - java.net.URLClassLoader$2.run() @bci=12, line=385 (Compiled frame) - java.security.AccessController.doPrivileged(java.security.PrivilegedAction, java.security.AccessControlContext) @bci=0 (Compiled frame) - java.net.URLClassLoader.findResource(java.lang.String) @bci=13, line=382 (Compiled frame) - java.lang.ClassLoader.getResource(java.lang.String) @bci=30, line=1002 (Compiled frame) - java.lang.ClassLoader.getResourceAsStream(java.lang.String) @bci=2, line=1192 (Compiled frame) - javax.xml.parsers.SecuritySupport$4.run() @bci=26, line=96 (Compiled frame) - java.security.AccessController.doPrivileged(java.security.PrivilegedAction) @bci=0 (Compiled frame) - javax.xml.parsers.SecuritySupport.getResourceAsStream(java.lang.ClassLoader, java.lang.String) @bci=10, line=89 (Compiled frame) - javax.xml.parsers.FactoryFinder.findJarServiceProvider(java.lang.String) @bci=38, line=250 (Interpreted frame) - javax.xml.parsers.FactoryFinder.find(java.lang.String, java.lang.String) @bci=273, line=223 (Interpreted frame) - javax.xml.parsers.DocumentBuilderFactory.newInstance() @bci=4, line=123 (Compiled frame) - org.apache.hadoop.conf.Configuration.loadResource(java.util.Properties, org.apache.hadoop.conf.Configuration$Resource, boolean) @bci=16, line=1890 (Compiled frame) - org.apache.hadoop.conf.Configuration.loadResources(java.util.Properties, java.util.ArrayList, boolean) @bci=49, line=1867 (Compiled frame) - org.apache.hadoop.conf.Configuration.getProps() @bci=43, line=1785 (Compiled frame) - org.apache.hadoop.conf.Configuration.get(java.lang.String) @bci=35, line=712 (Compiled frame) - org.apache.hadoop.conf.Configuration.getTrimmed(java.lang.String) @bci=2, line=731 (Compiled frame) - org.apache.hadoop.conf.Configuration.getBoolean(java.lang.String, boolean) @bci=2, line=1047 (Interpreted frame) - org.apache.hadoop.mapred.IFileInputStream.init(java.io.InputStream, long, org.apache.hadoop.conf.Configuration) @bci=111, line=93 (Interpreted frame) - org.apache.hadoop.mapred.IFile$Reader.init(org.apache.hadoop.conf.Configuration, org.apache.hadoop.fs.FSDataInputStream, long, org.apache.hadoop.io.compress.CompressionCodec, org.apache.hadoop.mapred.Counters$Counter) @bci=60, line=303 (Interpreted frame) - org.apache.hadoop.mapred.IFile$InMemoryReader.init(org.apache.hadoop.mapred.RamManager,
