[
https://issues.apache.org/jira/browse/SPARK-5302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14567061#comment-14567061
]
Reynold Xin commented on SPARK-5302:
------------------------------------
[~btiernay] is this resolved now SPARK-5182 is resolved?
> Add support for SQLContext "partition" columns
> ----------------------------------------------
>
> Key: SPARK-5302
> URL: https://issues.apache.org/jira/browse/SPARK-5302
> Project: Spark
> Issue Type: New Feature
> Components: SQL
> Reporter: Bob Tiernay
>
> For {{SQLContext}} (not {{HiveContext}}) it would be very convenient to
> support a virtual column that maps to part of the the file path, similar to
> what is done in Hive for partitions (e.g. {{/data/clicks/dt=2015-01-01/}}
> where {{dt}} is a column of type {{TEXT}}).
> The API could allow the user to type the column using an appropriate
> {{DataType}} instance. This new field could be addressed in SQL statements
> much the same as is done in Hive.
> As a consequence, pruning of partitions could be possible when executing a
> query and also remove the need to materialize a column in each logical
> partition that is already encoded in the path name. Furthermore, this would
> provide an nice interop and migration strategy for Hive users who may one day
> use {{SQLContext}} directly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
