Daniel Becker created IMPALA-10332:
--------------------------------------
Summary: Add file formats to HdfsScanNode's thrift representation
and codegen for those
Key: IMPALA-10332
URL: https://issues.apache.org/jira/browse/IMPALA-10332
Project: IMPALA
Issue Type: Improvement
Components: Backend, Frontend
Reporter: Daniel Becker
Assignee: Daniel Becker
List all file formats that a HdfsScanNode needs to process in any fragment
instance. It is possible that some file formats will not be needed in all
fragment instances.
This is a step towards sharing codegen between different impala backends. Using
the file formats provided in the thrift file, a backend can codegen code for
file formats that are not needed in its own process but are needed in other
fragment instances running on other backends, and the resulting binary can be
shared between multiple backends.
Codegenning for file formats will be done based on the thrift message and not
on what is needed for the actual backend. This leads to some extra work in case
a file format is not needed for the current backend and codegen sharing is not
available (at this point it is not implemented). However, the overall number of
such cases is low.
Also adding the file formats to the node's explain string.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
