[jira] [Commented] (MAPREDUCE-6440) Duplicate Key in Json Output for Job details
[ https://issues.apache.org/jira/browse/MAPREDUCE-6440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14699221#comment-14699221 ] Ajith S commented on MAPREDUCE-6440: Hi [~bibinchundatt] Thanks for reporting issue and analysis. You are right, the two *type* is ofcourse confusing unless someone knows the implementation details because *type* usually refers to datatype in json (*type:reduceTaskAttemptInfo* is right) . I suggest we can change the name of *org.apache.hadoop.mapreduce.v2.app.webapp.dao.TaskAttemptInfo.type*. Ofcourse a incompatible change Duplicate Key in Json Output for Job details Key: MAPREDUCE-6440 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6440 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Reporter: Anushri Assignee: Bibin A Chundatt Priority: Minor Duplicate key in Json Output for Job details for the url : http://jhs_ip:jhs_port/ws/v1/history/mapreduce/jobs/job_id/tasks/task_id/attempts If the task type is REDUCE the json output for this url contains duplicate key for type. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6440) Duplicate Key in Json Output for Job details
[ https://issues.apache.org/jira/browse/MAPREDUCE-6440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14699850#comment-14699850 ] zhihai xu commented on MAPREDUCE-6440: -- Yes, it looks like a good suggestion to avoid name conflict with reserved keywords. Duplicate Key in Json Output for Job details Key: MAPREDUCE-6440 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6440 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Reporter: Anushri Assignee: Bibin A Chundatt Priority: Minor Duplicate key in Json Output for Job details for the url : http://jhs_ip:jhs_port/ws/v1/history/mapreduce/jobs/job_id/tasks/task_id/attempts If the task type is REDUCE the json output for this url contains duplicate key for type. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6453) Repeatable Input File Format
AbdulRahman AlHamali created MAPREDUCE-6453: --- Summary: Repeatable Input File Format Key: MAPREDUCE-6453 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6453 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: AbdulRahman AlHamali Assignee: AbdulRahman AlHamali Priority: Minor We are interested in running the training process of deep learning architectures on Hadoop clusters. We developed an algorithm that can carry out this training process in a MapReduce fashion. However, there is still a problem that we can improve. In deep learning, training data is usually repeated multiple times (10 or even more). However, we were not able to find a way to go through the input training file multiple times without having to reduce first and then go back and then map and reduce and so on so forth. So, to carry on the experiments, we were forced to phyiscally repeat the files 10 or 20 times. This is not the best solution, obviously, because first the file size is becoming much larger, and second, it is not a neat way to carry out the job. Thus, what we aim to do is to create an interface that input file formats can implement that would provide them with the ability to repeat a file n times before eventually reducing, which will solve the problem and make Hadoop more suitable for the training of deep learning algorithms, or for such problems that require going over the data multiple times before reducing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6440) Duplicate Key in Json Output for Job details
[
https://issues.apache.org/jira/browse/MAPREDUCE-6440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14699739#comment-14699739
]
zhihai xu commented on MAPREDUCE-6440:
--
It looks like app master web service url
http://am_ip:am_port/ws/v1/mapreduce/jobs/{jobid}/tasks/{taskid}/attempts
also has this issue. Yes, it will be an incompatible change to fix this issue.
it will be better to fix it at 3.0.0.
Duplicate Key in Json Output for Job details
Key: MAPREDUCE-6440
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6440
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver
Reporter: Anushri
Assignee: Bibin A Chundatt
Priority: Minor
Duplicate key in Json Output for Job details for the url :
http://jhs_ip:jhs_port/ws/v1/history/mapreduce/jobs/job_id/tasks/task_id/attempts
If the task type is REDUCE the json output for this url contains duplicate
key for type.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6440) Duplicate Key in Json Output for Job details
[ https://issues.apache.org/jira/browse/MAPREDUCE-6440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhihai xu updated MAPREDUCE-6440: - Target Version/s: 3.0.0 Duplicate Key in Json Output for Job details Key: MAPREDUCE-6440 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6440 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Reporter: Anushri Assignee: Bibin A Chundatt Priority: Minor Duplicate key in Json Output for Job details for the url : http://jhs_ip:jhs_port/ws/v1/history/mapreduce/jobs/job_id/tasks/task_id/attempts If the task type is REDUCE the json output for this url contains duplicate key for type. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6452) NPE when intermediate encrypt enabled for LocalRunner
[
https://issues.apache.org/jira/browse/MAPREDUCE-6452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14699771#comment-14699771
]
zhihai xu commented on MAPREDUCE-6452:
--
Hi [~ajithshetty], Can I take this issue if you didn't start working on it yet?
I have a patch ready for this issue. thanks
NPE when intermediate encrypt enabled for LocalRunner
-
Key: MAPREDUCE-6452
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6452
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Bibin A Chundatt
Assignee: Ajith S
Priority: Minor
Enable the below properties try running mapreduce job
mapreduce.framework.name=local
mapreduce.job.encrypted-intermediate-data=true
{code}
2015-08-14 16:27:25,248 WARN [Thread-21] mapred.LocalJobRunner
(LocalJobRunner.java:run(561)) - job_local473843898_0001
java.lang.Exception: java.lang.NullPointerException
at
org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:463)
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:523)
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.crypto.CryptoOutputStream.init(CryptoOutputStream.java:92)
at
org.apache.hadoop.fs.crypto.CryptoFSDataOutputStream.init(CryptoFSDataOutputStream.java:31)
at
org.apache.hadoop.mapreduce.CryptoUtils.wrapIfNecessary(CryptoUtils.java:112)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1611)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1492)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at
org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:244)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
Jobs are failing always
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6452) NPE when intermediate encrypt enabled for LocalRunner
[
https://issues.apache.org/jira/browse/MAPREDUCE-6452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14699620#comment-14699620
]
Bibin A Chundatt commented on MAPREDUCE-6452:
-
[~aw] Any thoughts on this issue?
NPE when intermediate encrypt enabled for LocalRunner
-
Key: MAPREDUCE-6452
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6452
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Bibin A Chundatt
Assignee: Ajith S
Priority: Minor
Enable the below properties try running mapreduce job
mapreduce.framework.name=local
mapreduce.job.encrypted-intermediate-data=true
{code}
2015-08-14 16:27:25,248 WARN [Thread-21] mapred.LocalJobRunner
(LocalJobRunner.java:run(561)) - job_local473843898_0001
java.lang.Exception: java.lang.NullPointerException
at
org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:463)
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:523)
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.crypto.CryptoOutputStream.init(CryptoOutputStream.java:92)
at
org.apache.hadoop.fs.crypto.CryptoFSDataOutputStream.init(CryptoFSDataOutputStream.java:31)
at
org.apache.hadoop.mapreduce.CryptoUtils.wrapIfNecessary(CryptoUtils.java:112)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1611)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1492)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at
org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:244)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6452) NPE when intermediate encrypt enabled for LocalRunner
[
https://issues.apache.org/jira/browse/MAPREDUCE-6452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated MAPREDUCE-6452:
Description:
Enable the below properties try running mapreduce job
mapreduce.framework.name=local
mapreduce.job.encrypted-intermediate-data=true
{code}
2015-08-14 16:27:25,248 WARN [Thread-21] mapred.LocalJobRunner
(LocalJobRunner.java:run(561)) - job_local473843898_0001
java.lang.Exception: java.lang.NullPointerException
at
org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:463)
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:523)
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.crypto.CryptoOutputStream.init(CryptoOutputStream.java:92)
at
org.apache.hadoop.fs.crypto.CryptoFSDataOutputStream.init(CryptoFSDataOutputStream.java:31)
at
org.apache.hadoop.mapreduce.CryptoUtils.wrapIfNecessary(CryptoUtils.java:112)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1611)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1492)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at
org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:244)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
Jobs are failing always
was:
Enable the below properties try running mapreduce job
mapreduce.framework.name=local
mapreduce.job.encrypted-intermediate-data=true
{code}
2015-08-14 16:27:25,248 WARN [Thread-21] mapred.LocalJobRunner
(LocalJobRunner.java:run(561)) - job_local473843898_0001
java.lang.Exception: java.lang.NullPointerException
at
org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:463)
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:523)
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.crypto.CryptoOutputStream.init(CryptoOutputStream.java:92)
at
org.apache.hadoop.fs.crypto.CryptoFSDataOutputStream.init(CryptoFSDataOutputStream.java:31)
at
org.apache.hadoop.mapreduce.CryptoUtils.wrapIfNecessary(CryptoUtils.java:112)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1611)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1492)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at
org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:244)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
NPE when intermediate encrypt enabled for LocalRunner
-
Key: MAPREDUCE-6452
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6452
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Bibin A Chundatt
Assignee: Ajith S
Priority: Minor
Enable the below properties try running mapreduce job
mapreduce.framework.name=local
mapreduce.job.encrypted-intermediate-data=true
{code}
2015-08-14 16:27:25,248 WARN [Thread-21] mapred.LocalJobRunner
(LocalJobRunner.java:run(561)) - job_local473843898_0001
java.lang.Exception: java.lang.NullPointerException
at
org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:463)
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:523)
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.crypto.CryptoOutputStream.init(CryptoOutputStream.java:92)
at
org.apache.hadoop.fs.crypto.CryptoFSDataOutputStream.init(CryptoFSDataOutputStream.java:31)
at
org.apache.hadoop.mapreduce.CryptoUtils.wrapIfNecessary(CryptoUtils.java:112)
at
[jira] [Assigned] (MAPREDUCE-6452) NPE when intermediate encrypt enabled for LocalRunner
[
https://issues.apache.org/jira/browse/MAPREDUCE-6452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ajith S reassigned MAPREDUCE-6452:
--
Assignee: Ajith S
NPE when intermediate encrypt enabled for LocalRunner
-
Key: MAPREDUCE-6452
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6452
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Bibin A Chundatt
Assignee: Ajith S
Priority: Minor
Enable the below properties try running mapreduce job
mapreduce.framework.name=local
mapreduce.job.encrypted-intermediate-data=true
{code}
2015-08-14 16:27:25,248 WARN [Thread-21] mapred.LocalJobRunner
(LocalJobRunner.java:run(561)) - job_local473843898_0001
java.lang.Exception: java.lang.NullPointerException
at
org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:463)
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:523)
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.crypto.CryptoOutputStream.init(CryptoOutputStream.java:92)
at
org.apache.hadoop.fs.crypto.CryptoFSDataOutputStream.init(CryptoFSDataOutputStream.java:31)
at
org.apache.hadoop.mapreduce.CryptoUtils.wrapIfNecessary(CryptoUtils.java:112)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1611)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1492)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at
org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:244)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-2454) Allow external sorter plugin for MR
[ https://issues.apache.org/jira/browse/MAPREDUCE-2454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated MAPREDUCE-2454: - Assignee: Mariappan Asokan (was: Bharat Jha) Allow external sorter plugin for MR --- Key: MAPREDUCE-2454 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2454 Project: Hadoop Map/Reduce Issue Type: New Feature Affects Versions: 2.0.0-alpha, 3.0.0, 2.0.2-alpha Reporter: Mariappan Asokan Assignee: Mariappan Asokan Priority: Minor Labels: features, performance, plugin, sort Fix For: 2.0.3-alpha Attachments: HadoopSortPlugin.pdf, HadoopSortPlugin.pdf, KeyValueIterator.java, MR-2454-trunkPatchPreview.gz, MapOutputSorter.java, MapOutputSorterAbstract.java, ReduceInputSorter.java, mapreduce-2454-modified-code.patch, mapreduce-2454-modified-test.patch, mapreduce-2454-new-test.patch, mapreduce-2454-protection-change.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mr-2454-on-mr-279-build82.patch.gz Define interfaces and some abstract classes in the Hadoop framework to facilitate external sorter plugins both on the Map and Reduce sides. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
