[
https://issues.apache.org/jira/browse/HIVE-15178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15655335#comment-15655335
]
Sergey Shelukhin edited comment on HIVE-15178 at 11/10/16 10:20 PM:
--------------------------------------------------------------------
[~prasanth_j] fyi
Actually I suspect that ORC stripe merge needs proper split generation.
I can probably fix the settings propagation, but the target size (merge
resolver uses target size for split sizes for the MR job) is taken from some
config setting, so it can still produce invalid splits that are not on stripe
boundaries.
was (Author: sershe):
[~prasanth_j] fyi
> ORC stripe merge may produce many MR jobs and no merge if split size is small
> -----------------------------------------------------------------------------
>
> Key: HIVE-15178
> URL: https://issues.apache.org/jira/browse/HIVE-15178
> Project: Hive
> Issue Type: Bug
> Reporter: Sergey Shelukhin
> Assignee: Sergey Shelukhin
>
> orc_createas1
> logs the following:
> {noformat}
> 2016-11-10T13:38:54,366 INFO [LocalJobRunner Map Task Executor #0]
> mapred.MapTask: Processing split:
> Paths:/Users/sergey/git/hivegit2/itests/qtest/target/warehouse/.hive-staging_hive_2016-11-10_13-38-52_334_1323113125332102866-1/-ext-10004/000001_0:2400+100InputFormatClass:
> org.apache.hadoop.hive.ql.io.orc.OrcFileStripeMergeInputFormat
> 2016-11-10T13:38:54,373 INFO [LocalJobRunner Map Task Executor #0]
> mapred.MapTask: Processing split:
> Paths:/Users/sergey/git/hivegit2/itests/qtest/target/warehouse/.hive-staging_hive_2016-11-10_13-38-52_334_1323113125332102866-1/-ext-10004/000001_0:2500+100InputFormatClass:
> org.apache.hadoop.hive.ql.io.orc.OrcFileStripeMergeInputFormat
> 2016-11-10T13:38:54,380 INFO [LocalJobRunner Map Task Executor #0]
> mapred.MapTask: Processing split:
> Paths:/Users/sergey/git/hivegit2/itests/qtest/target/warehouse/.hive-staging_hive_2016-11-10_13-38-52_334_1323113125332102866-1/-ext-10004/000001_0:2600+100InputFormatClass:
> org.apache.hadoop.hive.ql.io.orc.OrcFileStripeMergeInputFormat
> 2016-11-10T13:38:54,387 INFO [LocalJobRunner Map Task Executor #0]
> mapred.MapTask: Processing split:
> Paths:/Users/sergey/git/hivegit2/itests/qtest/target/warehouse/.hive-staging_hive_2016-11-10_13-38-52_334_1323113125332102866-1/-ext-10004/000001_0:2700+100InputFormatClass:
> org.apache.hadoop.hive.ql.io.orc.OrcFileStripeMergeInputFormat
> ...
> {noformat}
> It tries to merge 2 files, but instead ends up running tons of MR tasks for
> every 100 bytes and produces 2 files again (I assume most tasks don't produce
> the files because the split at a random 100-byte offset is invalid).
> {noformat}
> 2016-11-10T13:38:53,985 INFO [LocalJobRunner Map Task Executor #0]
> OrcFileMergeOperator: Merged stripe from file
> pfile:/Users/sergey/git/hivegit2/itests/qtest/target/warehouse/.hive-staging_hive_2016-11-10_13-38-52_334_1323113125332102866-1/-ext-10004/000000_0
> [ offset : 3 length: 2770 row: 500 ]
> 2016-11-10T13:38:53,995 INFO [LocalJobRunner Map Task Executor #0]
> exec.AbstractFileMergeOperator: renamed path
> pfile:/Users/sergey/git/hivegit2/itests/qtest/target/warehouse/.hive-staging_hive_2016-11-10_13-38-52_334_1323113125332102866-1/_task_tmp.-ext-10002/_tmp.000002_0
> to
> pfile:/Users/sergey/git/hivegit2/itests/qtest/target/warehouse/.hive-staging_hive_2016-11-10_13-38-52_334_1323113125332102866-1/_tmp.-ext-10002/000002_0
> . File size is 2986
> 2016-11-10T13:38:54,206 INFO [LocalJobRunner Map Task Executor #0]
> OrcFileMergeOperator: Merged stripe from file
> pfile:/Users/sergey/git/hivegit2/itests/qtest/target/warehouse/.hive-staging_hive_2016-11-10_13-38-52_334_1323113125332102866-1/-ext-10004/000001_0
> [ offset : 3 length: 2770 row: 500 ]
> 2016-11-10T13:38:54,215 INFO [LocalJobRunner Map Task Executor #0]
> exec.AbstractFileMergeOperator: renamed path
> pfile:/Users/sergey/git/hivegit2/itests/qtest/target/warehouse/.hive-staging_hive_2016-11-10_13-38-52_334_1323113125332102866-1/_task_tmp.-ext-10002/_tmp.000030_0
> to
> pfile:/Users/sergey/git/hivegit2/itests/qtest/target/warehouse/.hive-staging_hive_2016-11-10_13-38-52_334_1323113125332102866-1/_tmp.-ext-10002/000030_0
> . File size is 2986
> {noformat}
> This is because the test sets the max split size to 100. Merge jobs is
> supposed to override that, but that doesn't happen somehow.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
