[
https://issues.apache.org/jira/browse/TEZ-4248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marton Bod updated TEZ-4248:
----------------------------
Description:
Based on a new config, MRReaderMapred should pass properites into its
incremental conf object.
The config name: tez.mrreader.config.update.properties
Example use case: in Hive, we would like to push down the projection properties
(i.e. which columns are selected) set during MRInput initialization, in order
to have this information available when constructing the object inspector (i.e.
to create an object inspector which reads only the selected columns) during the
map operation. To achieve that, in Hive we would set the following property:
conf.set("tez.mrreader.config.update.properties",
"hive.io.file.readcolumn.names,hive.io.file.readcolumn.ids"). This would ensure
we get these two property values propagated into the main conf object.
was:Based on a new config, MRReaderMapred should pass properites into its
incremental conf object.
> MRReaderMapred should propagate properties based on config
> ----------------------------------------------------------
>
> Key: TEZ-4248
> URL: https://issues.apache.org/jira/browse/TEZ-4248
> Project: Apache Tez
> Issue Type: New Feature
> Reporter: Marton Bod
> Assignee: Marton Bod
> Priority: Major
> Attachments: TEZ-4248.1.patch, TEZ-4248.2.patch, TEZ-4248.4.patch
>
>
> Based on a new config, MRReaderMapred should pass properites into its
> incremental conf object.
> The config name: tez.mrreader.config.update.properties
> Example use case: in Hive, we would like to push down the projection
> properties (i.e. which columns are selected) set during MRInput
> initialization, in order to have this information available when constructing
> the object inspector (i.e. to create an object inspector which reads only the
> selected columns) during the map operation. To achieve that, in Hive we would
> set the following property: conf.set("tez.mrreader.config.update.properties",
> "hive.io.file.readcolumn.names,hive.io.file.readcolumn.ids"). This would
> ensure we get these two property values propagated into the main conf object.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
