[
https://issues.apache.org/jira/browse/SPARK-11374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15420503#comment-15420503
]
Dongjoon Hyun commented on SPARK-11374:
---------------------------------------
Hi [[email protected]],
Thank you for comments. Yep. I noticed that option too, but that seems more
tricky.
The current approach of Spark Scala API and my PR is checking if the
partition's file start position is zero. So, it's not straight-forward to apply
to footer option.
For this issue, I think it could be acceptable since Spark Scala API already
supports `header` option.
However, for the `footer` option, I think we need a new JIRA issue to get some
attention and to build consensus for that option.
Thanks,
Dongjoon.
> skip.header.line.count is ignored in HiveContext
> ------------------------------------------------
>
> Key: SPARK-11374
> URL: https://issues.apache.org/jira/browse/SPARK-11374
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.5.1
> Reporter: Daniel Haviv
>
> csv table in Hive which is configured to skip the header row using
> TBLPROPERTIES("skip.header.line.count"="1").
> When querying from Hive the header row is not included in the data, but when
> running the same query via HiveContext I get the header row.
> "show create table " via the HiveContext confirms that it is aware of the
> setting.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
