[jira] [Created] (MAPREDUCE-4991) coverage for grid mix
Aleksey Gorshkov created MAPREDUCE-4991: --- Summary: coverage for grid mix Key: MAPREDUCE-4991 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4991 Project: Hadoop Map/Reduce Issue Type: Test Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.7 Reporter: Aleksey Gorshkov Fix For: 3.0.0, 2.0.3-alpha, 0.23.7 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Hadoop-Mapreduce-trunk - Build # 1338 - Failure
See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1338/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 28871 lines...] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.914 sec Running org.apache.hadoop.mapreduce.jobhistory.TestJobHistoryEventHandler Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.144 sec Results : Failed tests: testMultipleCrashes(org.apache.hadoop.mapreduce.v2.app.TestRecovery): Reduce Task state not correct expected:RUNNING but was:SCHEDULED testOutputRecovery(org.apache.hadoop.mapreduce.v2.app.TestRecovery): Task state is not correct (timedout) expected:SUCCEEDED but was:RUNNING testOutputRecoveryMapsOnly(org.apache.hadoop.mapreduce.v2.app.TestRecovery): Task state is not correct (timedout) expected:SUCCEEDED but was:RUNNING testRecoveryWithOldCommiter(org.apache.hadoop.mapreduce.v2.app.TestRecovery): Task state is not correct (timedout) expected:SUCCEEDED but was:RUNNING Tests run: 205, Failures: 4, Errors: 0, Skipped: 0 [INFO] [INFO] Reactor Summary: [INFO] [INFO] hadoop-mapreduce-client ... SUCCESS [1.651s] [INFO] hadoop-mapreduce-client-core .. SUCCESS [22.722s] [INFO] hadoop-mapreduce-client-common SUCCESS [23.300s] [INFO] hadoop-mapreduce-client-shuffle ... SUCCESS [1.651s] [INFO] hadoop-mapreduce-client-app ... FAILURE [5:11.002s] [INFO] hadoop-mapreduce-client-hs SKIPPED [INFO] hadoop-mapreduce-client-jobclient . SKIPPED [INFO] hadoop-mapreduce-client-hs-plugins SKIPPED [INFO] Apache Hadoop MapReduce Examples .. SKIPPED [INFO] hadoop-mapreduce .. SKIPPED [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 6:00.974s [INFO] Finished at: Fri Feb 08 13:20:58 UTC 2013 [INFO] Final Memory: 20M/174M [INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test (default-test) on project hadoop-mapreduce-client-app: There are test failures. [ERROR] [ERROR] Please refer to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/target/surefire-reports for the individual test results. [ERROR] - [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn goals -rf :hadoop-mapreduce-client-app Build step 'Execute shell' marked build as failure [FINDBUGS] Skipping publisher since build result is FAILURE Archiving artifacts Updating HDFS-4470 Updating YARN-377 Updating HADOOP-9252 Updating HDFS-4471 Updating HADOOP-9253 Updating YARN-383 Updating YARN-385 Updating HADOOP-9277 Email was triggered for: Failure Sending email for trigger: Failure ### ## FAILED TESTS (if any) ## No tests ran.
[jira] [Created] (MAPREDUCE-4992) AM hung in RecoveryService
Robert Parker created MAPREDUCE-4992: Summary: AM hung in RecoveryService Key: MAPREDUCE-4992 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4992 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am Affects Versions: 0.23.6 Reporter: Robert Parker A job hung in the Recovery Service on an AM restart. There were four map tasks events that were not processed and that prevented the complete task count from reaching zero which exits the recovery service. All four tasks were speculative -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Doubt about map reduce version 2
Suresh, The 1.0 line is still the stable line and improvements there can have a large impact on existing users. That being said I think there will be a lot of movement to Yarn/MRv2 starting in the second half of this year and all of next year. Also YARN scheduling is a larger area for study because it doesn't just run Map/Reduce. It allows you to explore how to effectively schedule other workloads in a multi-tennant environment. There has already been a lot of discussion about the scheduler and its protocol recently because it is still a very new area to explore and no one really knows how well the current solutions work for other work loads. As for speculative execution in MRv2 it is completely pluggable by the user. This should make it very easy for you to explore and compare different speculation schemes. --Bobby On 2/7/13 11:39 PM, Suresh S suresh...@gmail.com wrote: Hello Friends, I am working to propose some improved hadoop scheduling algorithm or speculative execution algorithim as part of my Phd research work. Now, the new version of hadoop, YARN/MR v2, is available. I have the following doubts: So, the algorithms (particularly scheduling and speculation algorithms) proposed for old hadoop version are applicable for new version of hadoop (YARN) or not. Is it worth and usefull to propose an algorithm for old hadoop version now? Is the user community can support and discuss the issues releted to old version? Thanks in Advance. *Regards* *S.Suresh,* *Research Scholar,* *Department of Computer Applications,* *National Institute of Technology,* *Tiruchirappalli - 620015.* *+91-9941506562*
[jira] [Created] (MAPREDUCE-4993) AM thinks it was killed when an error occurs setting up a task container launch context
Jason Lowe created MAPREDUCE-4993: - Summary: AM thinks it was killed when an error occurs setting up a task container launch context Key: MAPREDUCE-4993 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4993 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am Affects Versions: 0.23.5, 2.0.3-alpha Reporter: Jason Lowe If an IOException occurs while setting up a container launch context for a task then the AM exits with a KILLED status and no diagnostics. The job should be marked as FAILED (or maybe ERROR) with a useful diagnostics message indicating the nature of the error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: API incompatibility between Hadoop 1.0 and Hadoop 2.0
It isn't complete yet, but Tom did some initial work comparing with 23 which (preceeds 2.0): https://issues.apache.org/jira/browse/HADOOP-7738 We need to take that to the finish line. Thanks, +Vinod On Feb 8, 2013, at 10:21 AM, lohit wrote: Hi Devs, Is there a documented list of API incompatibility between version 1.0 and version 2.0 of APIs. -- Have a Nice Day! Lohit
Re: API incompatibility between Hadoop 1.0 and Hadoop 2.0
It seems that there're no documented API changes since 2.0. I'm studying the backward compatibility, if there's anything I can help, please let me know. Thanks, Zhijie On Fri, Feb 8, 2013 at 10:37 AM, Vinod Kumar Vavilapalli vino...@hortonworks.com wrote: It isn't complete yet, but Tom did some initial work comparing with 23 which (preceeds 2.0): https://issues.apache.org/jira/browse/HADOOP-7738 We need to take that to the finish line. Thanks, +Vinod On Feb 8, 2013, at 10:21 AM, lohit wrote: Hi Devs, Is there a documented list of API incompatibility between version 1.0 and version 2.0 of APIs. -- Have a Nice Day! Lohit -- Zhijie Shen Hortonworks Inc. http://hortonworks.com/
Re: API incompatibility between Hadoop 1.0 and Hadoop 2.0
Thanks Vinod. 2013/2/8 Vinod Kumar Vavilapalli vino...@hortonworks.com It isn't complete yet, but Tom did some initial work comparing with 23 which (preceeds 2.0): https://issues.apache.org/jira/browse/HADOOP-7738 We need to take that to the finish line. Thanks, +Vinod On Feb 8, 2013, at 10:21 AM, lohit wrote: Hi Devs, Is there a documented list of API incompatibility between version 1.0 and version 2.0 of APIs. -- Have a Nice Day! Lohit -- Have a Nice Day! Lohit
[jira] [Created] (MAPREDUCE-4994) Can't submit local job with hadoop jar -jt local
Sandy Ryza created MAPREDUCE-4994: - Summary: Can't submit local job with hadoop jar -jt local Key: MAPREDUCE-4994 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4994 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Affects Versions: 2.0.2-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza hadoop jar myjar.jar MyDriver -fs file:/// -jt local input.txt output/ should run a job using the local file system and the local job runner. Instead it tries to connect to a jobtracker. This appears to be because Cluster#initialize, which loads the ClientProtocol, contains no special handling for mapred.job.tracker. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4995) task-controller fails compilation on Mac
Chris Nauroth created MAPREDUCE-4995: Summary: task-controller fails compilation on Mac Key: MAPREDUCE-4995 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4995 Project: Hadoop Map/Reduce Issue Type: Bug Components: task-controller Affects Versions: 1.2.0 Reporter: Chris Nauroth The branch-1 task-controller codebase will not compile on Mac. This also means that Mac users generally can't run ant package unless they hack the build.xml to skip task-controller. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
mapreduce.jobtracker.address vs. yarn.resourcemanager.address
Is there a reason why mapreduce.jobtracker.address is not deprecated to yarn.resourcemanager.address? thanks, Sandy
Re: mapreduce.jobtracker.address vs. yarn.resourcemanager.address
There was a time when there were thoughts of being able to run MR1 in parallel with YARN while the later took off. So they were supposed to coexist, hence the non-deprecation. I am not sure we formally did, but once it is cast in stone that MR1 cannot be run with Hadoop 2.0 *(by which time we should remove JT/TT etc.), then we can deprecate these configs. Thanks, +Vinod On Feb 8, 2013, at 2:15 PM, Sandy Ryza wrote: Is there a reason why mapreduce.jobtracker.address is not deprecated to yarn.resourcemanager.address? thanks, Sandy
Re: mapreduce.jobtracker.address vs. yarn.resourcemanager.address
MR1 code is already gone from trunk and branch-2 Thx On Fri, Feb 8, 2013 at 2:27 PM, Vinod Kumar Vavilapalli vino...@hortonworks.com wrote: There was a time when there were thoughts of being able to run MR1 in parallel with YARN while the later took off. So they were supposed to coexist, hence the non-deprecation. I am not sure we formally did, but once it is cast in stone that MR1 cannot be run with Hadoop 2.0 *(by which time we should remove JT/TT etc.), then we can deprecate these configs. Thanks, +Vinod On Feb 8, 2013, at 2:15 PM, Sandy Ryza wrote: Is there a reason why mapreduce.jobtracker.address is not deprecated to yarn.resourcemanager.address? thanks, Sandy -- Alejandro
[jira] [Resolved] (MAPREDUCE-4922) Request with multiple data local nodes can cause NPE in AppSchedulingInfo
[ https://issues.apache.org/jira/browse/MAPREDUCE-4922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved MAPREDUCE-4922. --- Resolution: Won't Fix This is the expected behavior Request with multiple data local nodes can cause NPE in AppSchedulingInfo - Key: MAPREDUCE-4922 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4922 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mr-am, mrv2, scheduler Affects Versions: 2.0.2-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza With the way that the schedulers work, each request for a container on a node must consist of 3 ResourceRequests - one on the node, one on the rack, and one with *. AppSchedulingInfo tracks the outstanding requests. When a node is assigned a node-local container, allocateNodeLocal decrements the outstanding requests at each level - node, rack, and *. If the rack requests reach 0, it removes the mapping. A mapreduce task with multiple data local nodes submits multiple container requests, one for each node. It also submits one for each unique rack, and one for *. If there are fewer unique racks than data local nodes, this means that fewer rack-local ResourceRequests will be submitted than node-local ResourceRequests, so the rack-local mapping will be deleted before all the node-local requests are allocated and an NPE will come up the next time a node-local request from that rack is allocated. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4982) AM hung with one pending map task
[ https://issues.apache.org/jira/browse/MAPREDUCE-4982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe resolved MAPREDUCE-4982. --- Resolution: Duplicate This is fixed by the changes in MAPREDUCE-4893. That has since been pulled into branch-0.23, so closing this as a duplicate. AM hung with one pending map task - Key: MAPREDUCE-4982 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4982 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am Affects Versions: 0.23.6 Reporter: Jason Lowe Saw a job that hung with one pending map task that never ran. The task was in the SCHEDULED state with a single attempt that was in the UNASSIGNED state. The AM looked like it was waiting for a container from the RM, but the RM was never granting it the one container it needed. I suspect the AM botched the container request bookkeeping somehow. More details to follow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [VOTE] Release hadoop-2.0.3-alpha
+1 (binding) I downloaded the src tar ball, built it with the native bits enabled, started up a little cluster, and ran some sample jobs. Things worked as expected. I also verified the signatures on the source artifact. I did bump into one little issue, but I don't think it should be considered a blocker. When I first tried to start up the RM, it failed to start with this error: 13/02/08 16:00:31 FATAL resourcemanager.ResourceManager: Error starting ResourceManager java.lang.IllegalStateException: Queue configuration missing child queue names for root at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.parseQueue(CapacityScheduler.java:328) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:255) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:220) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.init(ResourceManager.java:226) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:710) And then this on shutdown: 13/02/08 16:00:31 INFO service.CompositeService: Error stopping ResourceManager java.lang.NullPointerException at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.stop(ResourceManager.java:590) at org.apache.hadoop.yarn.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:122) at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54) Presumably this is because I don't have the CapacityScheduler queues configured at all, and the default scheduler is now the CapacityScheduler. To work around this for my testing, I switched to the FairScheduler and the RM came up just fine. -- Aaron T. Myers Software Engineer, Cloudera On Wed, Feb 6, 2013 at 7:59 PM, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created a release candidate (rc0) for hadoop-2.0.3-alpha that I would like to release. This release contains several major enhancements such as QJM for HDFS HA, multi-resource scheduling for YARN, YARN ResourceManager restart etc. Also YARN has achieved significant stability at scale (more details from Y! folks here: http://s.apache.org/VYO). The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.0.3-alpha-rc0/ The RC tag in svn is here: http://svn.apache.org/viewvc/hadoop/common/tags/release-2.0.3-alpha-rc0/ The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/
Re: [VOTE] Release hadoop-2.0.3-alpha
The issue with the configuration is raised (and adressed) in https://issues.apache.org/jira/browse/BIGTOP-841 Cos On Fri, Feb 08, 2013 at 04:25PM, Aaron T. Myers wrote: +1 (binding) I downloaded the src tar ball, built it with the native bits enabled, started up a little cluster, and ran some sample jobs. Things worked as expected. I also verified the signatures on the source artifact. I did bump into one little issue, but I don't think it should be considered a blocker. When I first tried to start up the RM, it failed to start with this error: 13/02/08 16:00:31 FATAL resourcemanager.ResourceManager: Error starting ResourceManager java.lang.IllegalStateException: Queue configuration missing child queue names for root at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.parseQueue(CapacityScheduler.java:328) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:255) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:220) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.init(ResourceManager.java:226) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:710) And then this on shutdown: 13/02/08 16:00:31 INFO service.CompositeService: Error stopping ResourceManager java.lang.NullPointerException at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.stop(ResourceManager.java:590) at org.apache.hadoop.yarn.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:122) at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54) Presumably this is because I don't have the CapacityScheduler queues configured at all, and the default scheduler is now the CapacityScheduler. To work around this for my testing, I switched to the FairScheduler and the RM came up just fine. -- Aaron T. Myers Software Engineer, Cloudera On Wed, Feb 6, 2013 at 7:59 PM, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created a release candidate (rc0) for hadoop-2.0.3-alpha that I would like to release. This release contains several major enhancements such as QJM for HDFS HA, multi-resource scheduling for YARN, YARN ResourceManager restart etc. Also YARN has achieved significant stability at scale (more details from Y! folks here: http://s.apache.org/VYO). The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.0.3-alpha-rc0/ The RC tag in svn is here: http://svn.apache.org/viewvc/hadoop/common/tags/release-2.0.3-alpha-rc0/ The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ signature.asc Description: Digital signature
Re: [VOTE] Release hadoop-2.0.3-alpha
+1 Deployed on more than 100 nodes. Ran 30TB teragen/terasort. Will run few more over the weekend to test scheduler. Things looks stable. I do see few failures, but those I believe are hardware problems. Thanks, @lohitvijayarenu 2013/2/8 Konstantin Boudnik c...@apache.org The issue with the configuration is raised (and adressed) in https://issues.apache.org/jira/browse/BIGTOP-841 Cos On Fri, Feb 08, 2013 at 04:25PM, Aaron T. Myers wrote: +1 (binding) I downloaded the src tar ball, built it with the native bits enabled, started up a little cluster, and ran some sample jobs. Things worked as expected. I also verified the signatures on the source artifact. I did bump into one little issue, but I don't think it should be considered a blocker. When I first tried to start up the RM, it failed to start with this error: 13/02/08 16:00:31 FATAL resourcemanager.ResourceManager: Error starting ResourceManager java.lang.IllegalStateException: Queue configuration missing child queue names for root at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.parseQueue(CapacityScheduler.java:328) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:255) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:220) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.init(ResourceManager.java:226) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:710) And then this on shutdown: 13/02/08 16:00:31 INFO service.CompositeService: Error stopping ResourceManager java.lang.NullPointerException at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.stop(ResourceManager.java:590) at org.apache.hadoop.yarn.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:122) at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54) Presumably this is because I don't have the CapacityScheduler queues configured at all, and the default scheduler is now the CapacityScheduler. To work around this for my testing, I switched to the FairScheduler and the RM came up just fine. -- Aaron T. Myers Software Engineer, Cloudera On Wed, Feb 6, 2013 at 7:59 PM, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created a release candidate (rc0) for hadoop-2.0.3-alpha that I would like to release. This release contains several major enhancements such as QJM for HDFS HA, multi-resource scheduling for YARN, YARN ResourceManager restart etc. Also YARN has achieved significant stability at scale (more details from Y! folks here: http://s.apache.org/VYO). The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.0.3-alpha-rc0/ The RC tag in svn is here: http://svn.apache.org/viewvc/hadoop/common/tags/release-2.0.3-alpha-rc0/ The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- Have a Nice Day! Lohit
pre-historic record IO stuff, is this used anywhere?
This seems to be used only in tests in common and in a standalone class in streaming tests. What is the purpose of these classes as they don't seem to be used in the any of the source that ends up in Hadoop? hadoop-common-project/hadoop-common/src/test/ddl/buffer.jr hadoop-common-project/hadoop-common/src/test/ddl/int.jr hadoop-common-project/hadoop-common/src/test/ddl/string.jr hadoop-common-project/hadoop-common/src/test/ddl/test.jr hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/record/FromCpp.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/record/RecordBench.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/record/TestBuffer.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/record/TestRecordIO.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/record/TestRecordVersioning.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/record/ToCpp.java hadoop-tools/hadoop-streaming/src/test/java/org/apache/hadoop/typedbytes/TestIO.java I've deleted the above classes, cleaned up the common POM (not to compile the JR files) and everything compiles fine. To me all this is dead code, if so, can we nuke them? Thx -- Alejandro
Re: pre-historic record IO stuff, is this used anywhere?
Hadoop streaming is also tied to recordio as it is today: https://issues.apache.org/jira/browse/MAPREDUCE-3303, but it can be removed per Klaas. On Sat, Feb 9, 2013 at 6:48 AM, Alejandro Abdelnur t...@cloudera.com wrote: This seems to be used only in tests in common and in a standalone class in streaming tests. What is the purpose of these classes as they don't seem to be used in the any of the source that ends up in Hadoop? hadoop-common-project/hadoop-common/src/test/ddl/buffer.jr hadoop-common-project/hadoop-common/src/test/ddl/int.jr hadoop-common-project/hadoop-common/src/test/ddl/string.jr hadoop-common-project/hadoop-common/src/test/ddl/test.jr hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/record/FromCpp.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/record/RecordBench.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/record/TestBuffer.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/record/TestRecordIO.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/record/TestRecordVersioning.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/record/ToCpp.java hadoop-tools/hadoop-streaming/src/test/java/org/apache/hadoop/typedbytes/TestIO.java I've deleted the above classes, cleaned up the common POM (not to compile the JR files) and everything compiles fine. To me all this is dead code, if so, can we nuke them? Thx -- Alejandro -- Harsh J
Re: API incompatibility between Hadoop 1.0 and Hadoop 2.0
Hey Lohit, It may not be exactly in the form you're looking for, but take a look at Dave's API Evolution page http://dbeech.github.com/hadoop-api-evolution.html that presents a matrix. Its psychedelic to look at, I know, but you asked for it ;) We also generate jdiff reports as part of the regular maven build (AFAIR), which may also be what you're looking for. From my experience so far, its the behavior changes from 0.21+ thats biting users more today than any API changes (there are far fewer API changes). On Fri, Feb 8, 2013 at 11:51 PM, lohit lohit.vijayar...@gmail.com wrote: Hi Devs, Is there a documented list of API incompatibility between version 1.0 and version 2.0 of APIs. -- Have a Nice Day! Lohit -- Harsh J
Re: mapreduce.jobtracker.address vs. yarn.resourcemanager.address
Yes, there isn't a classic version of mapreduce.framework.name now either. Its mostly being used as a pseudonym. MR1 no longer exists in 2.x releases/trunk. Note though that Oozie still relies on that specific property (JT address) and 0.22 clients may also perhaps be using it. We can deprecate it and remove on 3.x or later. On Sat, Feb 9, 2013 at 4:00 AM, Alejandro Abdelnur t...@cloudera.com wrote: MR1 code is already gone from trunk and branch-2 Thx On Fri, Feb 8, 2013 at 2:27 PM, Vinod Kumar Vavilapalli vino...@hortonworks.com wrote: There was a time when there were thoughts of being able to run MR1 in parallel with YARN while the later took off. So they were supposed to coexist, hence the non-deprecation. I am not sure we formally did, but once it is cast in stone that MR1 cannot be run with Hadoop 2.0 *(by which time we should remove JT/TT etc.), then we can deprecate these configs. Thanks, +Vinod On Feb 8, 2013, at 2:15 PM, Sandy Ryza wrote: Is there a reason why mapreduce.jobtracker.address is not deprecated to yarn.resourcemanager.address? thanks, Sandy -- Alejandro -- Harsh J
Re: API incompatibility between Hadoop 1.0 and Hadoop 2.0
Thanks. What I am more interested is list of APIs where users have to change code. I know that on YARN I can run jobs which still use org.apache.hadoop.mapred code. But recently we hit a case where we had to change code for one case to start with. For example JobContext was class in 1.0 while Interface in 2.0. So, user has to change code. I was more interested in such cases (if it was documented) 2013/2/8 Harsh J ha...@cloudera.com Hey Lohit, It may not be exactly in the form you're looking for, but take a look at Dave's API Evolution page http://dbeech.github.com/hadoop-api-evolution.html that presents a matrix. Its psychedelic to look at, I know, but you asked for it ;) We also generate jdiff reports as part of the regular maven build (AFAIR), which may also be what you're looking for. From my experience so far, its the behavior changes from 0.21+ thats biting users more today than any API changes (there are far fewer API changes). On Fri, Feb 8, 2013 at 11:51 PM, lohit lohit.vijayar...@gmail.com wrote: Hi Devs, Is there a documented list of API incompatibility between version 1.0 and version 2.0 of APIs. -- Have a Nice Day! Lohit -- Harsh J -- Have a Nice Day! Lohit
Re: API incompatibility between Hadoop 1.0 and Hadoop 2.0
Ah ok, then that is probably the only base incompatibility you'll encounter (it was captured by jdiff IIRC) and is almost the only reason we've demanded jobs from 1.0 be recompiled before running on 2.0 (its a binary incompatible change made around ~0.21 time). On Sat, Feb 9, 2013 at 11:23 AM, lohit lohit.vijayar...@gmail.com wrote: Thanks. What I am more interested is list of APIs where users have to change code. I know that on YARN I can run jobs which still use org.apache.hadoop.mapred code. But recently we hit a case where we had to change code for one case to start with. For example JobContext was class in 1.0 while Interface in 2.0. So, user has to change code. I was more interested in such cases (if it was documented) 2013/2/8 Harsh J ha...@cloudera.com Hey Lohit, It may not be exactly in the form you're looking for, but take a look at Dave's API Evolution page http://dbeech.github.com/hadoop-api-evolution.html that presents a matrix. Its psychedelic to look at, I know, but you asked for it ;) We also generate jdiff reports as part of the regular maven build (AFAIR), which may also be what you're looking for. From my experience so far, its the behavior changes from 0.21+ thats biting users more today than any API changes (there are far fewer API changes). On Fri, Feb 8, 2013 at 11:51 PM, lohit lohit.vijayar...@gmail.com wrote: Hi Devs, Is there a documented list of API incompatibility between version 1.0 and version 2.0 of APIs. -- Have a Nice Day! Lohit -- Harsh J -- Have a Nice Day! Lohit -- Harsh J
