[
https://issues.apache.org/jira/browse/SPARK-3559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yin Huai updated SPARK-3559:
----------------------------
Description:
Because we are using the same hiveConf and we are currently using
ColumnProjectionUtils.appendReadColumnIDs
ColumnProjectionUtils.appendReadColumnNames to append needed column ids and
names for a table, lists of needed column ids and names can have unnecessary
columns.
Also, for a join operation, TableScanOperators for both tables are sharing the
same hiveConf and they may need to set table-specific properties.
was:Because we are using the same hiveConf and we are currently using
ColumnProjectionUtils.appendReadColumnIDs
ColumnProjectionUtils.appendReadColumnNames to append needed column ids and
names for a table, lists of needed column ids and names can have unnecessary
columns.
> appendReadColumnIDs and appendReadColumnNames introduce unnecessary columns
> in the lists of needed column ids and column names stored in hiveConf
> -------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-3559
> URL: https://issues.apache.org/jira/browse/SPARK-3559
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Reporter: Yin Huai
> Priority: Critical
>
> Because we are using the same hiveConf and we are currently using
> ColumnProjectionUtils.appendReadColumnIDs
> ColumnProjectionUtils.appendReadColumnNames to append needed column ids and
> names for a table, lists of needed column ids and names can have unnecessary
> columns.
> Also, for a join operation, TableScanOperators for both tables are sharing
> the same hiveConf and they may need to set table-specific properties.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
