[
https://issues.apache.org/jira/browse/IMPALA-10332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong updated IMPALA-10332:
-----------------------------------
Fix Version/s: Impala 4.0
> Add file formats to HdfsScanNode's thrift representation and codegen for those
> ------------------------------------------------------------------------------
>
> Key: IMPALA-10332
> URL: https://issues.apache.org/jira/browse/IMPALA-10332
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend, Frontend
> Reporter: Daniel Becker
> Assignee: Daniel Becker
> Priority: Major
> Fix For: Impala 4.0
>
>
> List all file formats that a HdfsScanNode needs to process in any fragment
> instance. It is possible that some file formats will not be needed in all
> fragment instances.
> This is a step towards sharing codegen between different impala backends.
> Using the file formats provided in the thrift file, a backend can codegen
> code for file formats that are not needed in its own process but are needed
> in other fragment instances running on other backends, and the resulting
> binary can be shared between multiple backends.
> Codegenning for file formats will be done based on the thrift message and not
> on what is needed for the actual backend. This leads to some extra work in
> case a file format is not needed for the current backend and codegen sharing
> is not available (at this point it is not implemented). However, the overall
> number of such cases is low.
> Also adding the file formats to the node's explain string.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
