[
https://issues.apache.org/jira/browse/IGNITE-909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14556187#comment-14556187
]
Evans Ye commented on IGNITE-909:
---------------------------------
Thanks for your review.
> Support HadoopExternalSplit in HadoopV2Context to get pig jobs to function
> --------------------------------------------------------------------------
>
> Key: IGNITE-909
> URL: https://issues.apache.org/jira/browse/IGNITE-909
> Project: Ignite
> Issue Type: Task
> Components: hadoop
> Affects Versions: sprint-4
> Reporter: Evans Ye
> Assignee: Evans Ye
> Fix For: sprint-5
>
> Attachments: IGNITE-909.patch
>
>
> Currently ignite does not support to run pig jobs on top of it. A test
> script like the following:
> {code}
> grunt> A = load '/passwd';
> grunt> B = foreach A generate $0;
> grunt> dump B;
> {code}
> can lead to a failure stack trace showing below:
> {code}
> [13:33:49,455][ERROR][Hadoop-task-16820899-f25d-40ee-acc7-7ec0b86f4c27_10-MAP-0-0-#165%null%][HadoopRunnableTask]
> Task execution failed.
> class org.apache.ignite.IgniteCheckedException: class
> org.apache.ignite.IgniteCheckedException: null
> at
> org.apache.ignite.internal.processors.hadoop.v2.HadoopV2MapTask.run0(HadoopV2MapTask.java:102)
> at
> org.apache.ignite.internal.processors.hadoop.v2.HadoopV2Task.run(HadoopV2Task.java:50)
> at
> org.apache.ignite.internal.processors.hadoop.v2.HadoopV2TaskContext.run(HadoopV2TaskContext.java:193)
> at
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.runTask(HadoopRunnableTask.java:176)
> at
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call(HadoopRunnableTask.java:120)
> at
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call(HadoopRunnableTask.java:36)
> at
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopExecutorService$2.body(HadoopExecutorService.java:183)
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:107)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.UnsupportedOperationException
> at
> org.apache.ignite.internal.processors.hadoop.v2.HadoopV2Context.getInputSplit(HadoopV2Context.java:93)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.getInputSplit(WrappedMapper.java:76)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.setup(PigGenericMapBase.java:202)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
> at
> org.apache.ignite.internal.processors.hadoop.v2.HadoopV2MapTask.run0(HadoopV2MapTask.java:84)
> ... 8 more
> {code}
> Implementing the TODO section in HadoopV2Context should get pig jobs to
> function.
> {code}
> @Override public InputSplit getInputSplit() {
> if (inputSplit == null) {
> HadoopInputSplit split = ctx.taskInfo().inputSplit();
> if (split == null)
> return null;
> if (split instanceof HadoopFileBlock) {
> HadoopFileBlock fileBlock = (HadoopFileBlock)split;
> inputSplit = new FileSplit(new Path(fileBlock.file()),
> fileBlock.start(), fileBlock.length(), null);
> }
> else if (split instanceof HadoopExternalSplit)
> throw new UnsupportedOperationException(); // TODO
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
