[
https://issues.apache.org/jira/browse/HIVE-13223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15210878#comment-15210878
]
Ashutosh Chauhan commented on HIVE-13223:
-----------------------------------------
This patch is not ready. I think bug is in Spark itself, which is if you submit
spark job with 0 splits, spark executors just hang. This got exposed by
HIVE-13040 after which we were generating such jobs which was in turn effect of
not generating splits for 0-length files. Note, MR & Tez dont have this issue.
In this patch, I tried to generate splits even for 0-length files (by not
skipping them) but that breaks later at job execution time because ORC reader
is not resilient to 0-length file.
To fix this issue we need to either figure out and fix Spark hang issue or
extend Orc reader to handle 0-length file more gracefully (those failures were
exposed in last Hive QA run)
> HoS may hang for queries that run on 0 splits
> -----------------------------------------------
>
> Key: HIVE-13223
> URL: https://issues.apache.org/jira/browse/HIVE-13223
> Project: Hive
> Issue Type: Bug
> Components: Spark
> Affects Versions: 2.1.0
> Reporter: Ashutosh Chauhan
> Assignee: Ashutosh Chauhan
> Attachments: HIVE-13223.1.patch, HIVE-13223.2.patch, HIVE-13223.patch
>
>
> Can be seen on all timed out tests after HIVE-13040 went in
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
