[Impala-ASF-CR] IMPALA-10332: Add file formats to HdfsScanNode's thrift representation.

2020-11-20 Thread Csaba Ringhofer (Code Review)
Csaba Ringhofer has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/16728 )

Change subject: IMPALA-10332: Add file formats to HdfsScanNode's thrift 
representation.
..

IMPALA-10332: Add file formats to HdfsScanNode's thrift representation.

List all file formats that a HdfsScanNode needs to process in any
fragment instance. It is possible that some file formats will not be
needed in all fragment instances.

This is a step towards sharing codegen between different impala
backends. Using the file formats provided in the thrift file, a backend
can codegen code for file formats that are not needed in its own process
but are needed in other fragment instances running on other backends,
and the resulting binary can be shared between multiple backends.

Codegenning for file formats will be done based on the thrift message
and not on what is needed for the actual backend. This leads to some
extra work in case a file format is not needed for the current backend
and codegen sharing is not available (at this point it is not
implemented). However, the overall number of such cases is low.

Also adding the file formats to the node's explain string at level 3.

Testing:
 - Added tests to verify that the file formats are present in the
   explain string at level 3.

Change-Id: Iad6b8271bd248983f327c07883a3bedf50f25b5d
Reviewed-on: http://gerrit.cloudera.org:8080/16728
Tested-by: Impala Public Jenkins 
Reviewed-by: Csaba Ringhofer 
---
M be/src/exec/hdfs-scan-node-base.cc
M common/thrift/PlanNodes.thrift
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M testdata/workloads/functional-planner/queries/PlannerTest/acid-scans.test
M testdata/workloads/functional-query/queries/QueryTest/explain-level3.test
5 files changed, 52 insertions(+), 8 deletions(-)

Approvals:
  Impala Public Jenkins: Verified
  Csaba Ringhofer: Looks good to me, approved

--
To view, visit http://gerrit.cloudera.org:8080/16728
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Iad6b8271bd248983f327c07883a3bedf50f25b5d
Gerrit-Change-Number: 16728
Gerrit-PatchSet: 11
Gerrit-Owner: Daniel Becker 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Daniel Becker 
Gerrit-Reviewer: Impala Public Jenkins 


[Impala-ASF-CR] IMPALA-10332: Add file formats to HdfsScanNode's thrift representation.

2020-11-20 Thread Csaba Ringhofer (Code Review)
Csaba Ringhofer has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16728 )

Change subject: IMPALA-10332: Add file formats to HdfsScanNode's thrift 
representation.
..


Patch Set 10: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/16728
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iad6b8271bd248983f327c07883a3bedf50f25b5d
Gerrit-Change-Number: 16728
Gerrit-PatchSet: 10
Gerrit-Owner: Daniel Becker 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Daniel Becker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Fri, 20 Nov 2020 17:54:05 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-10332: Add file formats to HdfsScanNode's thrift representation.

2020-11-20 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16728 )

Change subject: IMPALA-10332: Add file formats to HdfsScanNode's thrift 
representation.
..


Patch Set 10: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/16728
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iad6b8271bd248983f327c07883a3bedf50f25b5d
Gerrit-Change-Number: 16728
Gerrit-PatchSet: 10
Gerrit-Owner: Daniel Becker 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Daniel Becker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Fri, 20 Nov 2020 17:25:26 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-10332: Add file formats to HdfsScanNode's thrift representation.

2020-11-20 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16728 )

Change subject: IMPALA-10332: Add file formats to HdfsScanNode's thrift 
representation.
..


Patch Set 9:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/7697/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/16728
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iad6b8271bd248983f327c07883a3bedf50f25b5d
Gerrit-Change-Number: 16728
Gerrit-PatchSet: 9
Gerrit-Owner: Daniel Becker 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Daniel Becker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Fri, 20 Nov 2020 12:16:15 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-10332: Add file formats to HdfsScanNode's thrift representation.

2020-11-20 Thread Csaba Ringhofer (Code Review)
Csaba Ringhofer has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16728 )

Change subject: IMPALA-10332: Add file formats to HdfsScanNode's thrift 
representation.
..


Patch Set 9: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/16728
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iad6b8271bd248983f327c07883a3bedf50f25b5d
Gerrit-Change-Number: 16728
Gerrit-PatchSet: 9
Gerrit-Owner: Daniel Becker 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Daniel Becker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Fri, 20 Nov 2020 11:57:33 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-10332: Add file formats to HdfsScanNode's thrift representation.

2020-11-20 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16728 )

Change subject: IMPALA-10332: Add file formats to HdfsScanNode's thrift 
representation.
..


Patch Set 10:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6685/ 
DRY_RUN=false


--
To view, visit http://gerrit.cloudera.org:8080/16728
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iad6b8271bd248983f327c07883a3bedf50f25b5d
Gerrit-Change-Number: 16728
Gerrit-PatchSet: 10
Gerrit-Owner: Daniel Becker 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Daniel Becker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Fri, 20 Nov 2020 11:55:36 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-10332: Add file formats to HdfsScanNode's thrift representation.

2020-11-20 Thread Daniel Becker (Code Review)
Daniel Becker has uploaded a new patch set (#9). ( 
http://gerrit.cloudera.org:8080/16728 )

Change subject: IMPALA-10332: Add file formats to HdfsScanNode's thrift 
representation.
..

IMPALA-10332: Add file formats to HdfsScanNode's thrift representation.

List all file formats that a HdfsScanNode needs to process in any
fragment instance. It is possible that some file formats will not be
needed in all fragment instances.

This is a step towards sharing codegen between different impala
backends. Using the file formats provided in the thrift file, a backend
can codegen code for file formats that are not needed in its own process
but are needed in other fragment instances running on other backends,
and the resulting binary can be shared between multiple backends.

Codegenning for file formats will be done based on the thrift message
and not on what is needed for the actual backend. This leads to some
extra work in case a file format is not needed for the current backend
and codegen sharing is not available (at this point it is not
implemented). However, the overall number of such cases is low.

Also adding the file formats to the node's explain string at level 3.

Testing:
 - Added tests to verify that the file formats are present in the
   explain string at level 3.

Change-Id: Iad6b8271bd248983f327c07883a3bedf50f25b5d
---
M be/src/exec/hdfs-scan-node-base.cc
M common/thrift/PlanNodes.thrift
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M testdata/workloads/functional-planner/queries/PlannerTest/acid-scans.test
M testdata/workloads/functional-query/queries/QueryTest/explain-level3.test
5 files changed, 52 insertions(+), 8 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/28/16728/9
--
To view, visit http://gerrit.cloudera.org:8080/16728
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iad6b8271bd248983f327c07883a3bedf50f25b5d
Gerrit-Change-Number: 16728
Gerrit-PatchSet: 9
Gerrit-Owner: Daniel Becker 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Daniel Becker 
Gerrit-Reviewer: Impala Public Jenkins 


[Impala-ASF-CR] IMPALA-10332: Add file formats to HdfsScanNode's thrift representation.

2020-11-19 Thread Csaba Ringhofer (Code Review)
Csaba Ringhofer has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16728 )

Change subject: IMPALA-10332: Add file formats to HdfsScanNode's thrift 
representation.
..


Patch Set 8:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/16728/8/testdata/workloads/functional-planner/queries/PlannerTest/acid-scans.test
File testdata/workloads/functional-planner/queries/PlannerTest/acid-scans.test:

http://gerrit.cloudera.org:8080/#/c/16728/8/testdata/workloads/functional-planner/queries/PlannerTest/acid-scans.test@298
PS8, Line 298: |  | file formats: [ORC]
We shouldn't see this, as explain_level is only 2, right?



--
To view, visit http://gerrit.cloudera.org:8080/16728
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iad6b8271bd248983f327c07883a3bedf50f25b5d
Gerrit-Change-Number: 16728
Gerrit-PatchSet: 8
Gerrit-Owner: Daniel Becker 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Daniel Becker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Thu, 19 Nov 2020 18:44:08 +
Gerrit-HasComments: Yes


[Impala-ASF-CR] IMPALA-10332: Add file formats to HdfsScanNode's thrift representation.

2020-11-19 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16728 )

Change subject: IMPALA-10332: Add file formats to HdfsScanNode's thrift 
representation.
..


Patch Set 8: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6680/


--
To view, visit http://gerrit.cloudera.org:8080/16728
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iad6b8271bd248983f327c07883a3bedf50f25b5d
Gerrit-Change-Number: 16728
Gerrit-PatchSet: 8
Gerrit-Owner: Daniel Becker 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Daniel Becker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Thu, 19 Nov 2020 18:35:11 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-10332: Add file formats to HdfsScanNode's thrift representation.

2020-11-19 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16728 )

Change subject: IMPALA-10332: Add file formats to HdfsScanNode's thrift 
representation.
..


Patch Set 8:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6680/ 
DRY_RUN=false


--
To view, visit http://gerrit.cloudera.org:8080/16728
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iad6b8271bd248983f327c07883a3bedf50f25b5d
Gerrit-Change-Number: 16728
Gerrit-PatchSet: 8
Gerrit-Owner: Daniel Becker 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Daniel Becker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Thu, 19 Nov 2020 13:08:01 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-10332: Add file formats to HdfsScanNode's thrift representation.

2020-11-19 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16728 )

Change subject: IMPALA-10332: Add file formats to HdfsScanNode's thrift 
representation.
..


Patch Set 7:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/7689/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/16728
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iad6b8271bd248983f327c07883a3bedf50f25b5d
Gerrit-Change-Number: 16728
Gerrit-PatchSet: 7
Gerrit-Owner: Daniel Becker 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Daniel Becker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Thu, 19 Nov 2020 12:38:06 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-10332: Add file formats to HdfsScanNode's thrift representation.

2020-11-19 Thread Daniel Becker (Code Review)
Daniel Becker has uploaded a new patch set (#7). ( 
http://gerrit.cloudera.org:8080/16728 )

Change subject: IMPALA-10332: Add file formats to HdfsScanNode's thrift 
representation.
..

IMPALA-10332: Add file formats to HdfsScanNode's thrift representation.

List all file formats that a HdfsScanNode needs to process in any
fragment instance. It is possible that some file formats will not be
needed in all fragment instances.

This is a step towards sharing codegen between different impala
backends. Using the file formats provided in the thrift file, a backend
can codegen code for file formats that are not needed in its own process
but are needed in other fragment instances running on other backends,
and the resulting binary can be shared between multiple backends.

Codegenning for file formats will be done based on the thrift message
and not on what is needed for the actual backend. This leads to some
extra work in case a file format is not needed for the current backend
and codegen sharing is not available (at this point it is not
implemented). However, the overall number of such cases is low.

Also adding the file formats to the node's explain string at level 3.

Testing:
 - Added tests to verify that the file formats are present in the
   explain string at level 3.

Change-Id: Iad6b8271bd248983f327c07883a3bedf50f25b5d
---
M be/src/exec/hdfs-scan-node-base.cc
M common/thrift/PlanNodes.thrift
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M testdata/workloads/functional-planner/queries/PlannerTest/acid-scans.test
M testdata/workloads/functional-query/queries/QueryTest/explain-level3.test
5 files changed, 60 insertions(+), 8 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/28/16728/7
--
To view, visit http://gerrit.cloudera.org:8080/16728
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iad6b8271bd248983f327c07883a3bedf50f25b5d
Gerrit-Change-Number: 16728
Gerrit-PatchSet: 7
Gerrit-Owner: Daniel Becker 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Daniel Becker 
Gerrit-Reviewer: Impala Public Jenkins 


[Impala-ASF-CR] IMPALA-10332: Add file formats to HdfsScanNode's thrift representation.

2020-11-18 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16728 )

Change subject: IMPALA-10332: Add file formats to HdfsScanNode's thrift 
representation.
..


Patch Set 6: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6669/


--
To view, visit http://gerrit.cloudera.org:8080/16728
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iad6b8271bd248983f327c07883a3bedf50f25b5d
Gerrit-Change-Number: 16728
Gerrit-PatchSet: 6
Gerrit-Owner: Daniel Becker 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Daniel Becker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Wed, 18 Nov 2020 23:21:39 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-10332: Add file formats to HdfsScanNode's thrift representation.

2020-11-18 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16728 )

Change subject: IMPALA-10332: Add file formats to HdfsScanNode's thrift 
representation.
..


Patch Set 6:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6669/ 
DRY_RUN=false


--
To view, visit http://gerrit.cloudera.org:8080/16728
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iad6b8271bd248983f327c07883a3bedf50f25b5d
Gerrit-Change-Number: 16728
Gerrit-PatchSet: 6
Gerrit-Owner: Daniel Becker 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Daniel Becker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Wed, 18 Nov 2020 17:54:23 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-10332: Add file formats to HdfsScanNode's thrift representation.

2020-11-18 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16728 )

Change subject: IMPALA-10332: Add file formats to HdfsScanNode's thrift 
representation.
..


Patch Set 6: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/16728
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iad6b8271bd248983f327c07883a3bedf50f25b5d
Gerrit-Change-Number: 16728
Gerrit-PatchSet: 6
Gerrit-Owner: Daniel Becker 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Daniel Becker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Wed, 18 Nov 2020 17:54:22 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-10332: Add file formats to HdfsScanNode's thrift representation.

2020-11-18 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16728 )

Change subject: IMPALA-10332: Add file formats to HdfsScanNode's thrift 
representation.
..


Patch Set 5:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/7675/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/16728
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iad6b8271bd248983f327c07883a3bedf50f25b5d
Gerrit-Change-Number: 16728
Gerrit-PatchSet: 5
Gerrit-Owner: Daniel Becker 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Daniel Becker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Wed, 18 Nov 2020 17:36:28 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-10332: Add file formats to HdfsScanNode's thrift representation.

2020-11-18 Thread Csaba Ringhofer (Code Review)
Csaba Ringhofer has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16728 )

Change subject: IMPALA-10332: Add file formats to HdfsScanNode's thrift 
representation.
..


Patch Set 5: Code-Review+2

(1 comment)

http://gerrit.cloudera.org:8080/#/c/16728/4/testdata/workloads/functional-query/queries/QueryTest/explain-level3.test
File testdata/workloads/functional-query/queries/QueryTest/explain-level3.test:

http://gerrit.cloudera.org:8080/#/c/16728/4/testdata/workloads/functional-query/queries/QueryTest/explain-level3.test@86
PS4, Line 86: Iceberg
> You mean capital I as Iceberg?
oops, yes



--
To view, visit http://gerrit.cloudera.org:8080/16728
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iad6b8271bd248983f327c07883a3bedf50f25b5d
Gerrit-Change-Number: 16728
Gerrit-PatchSet: 5
Gerrit-Owner: Daniel Becker 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Daniel Becker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Wed, 18 Nov 2020 17:25:47 +
Gerrit-HasComments: Yes


[Impala-ASF-CR] IMPALA-10332: Add file formats to HdfsScanNode's thrift representation.

2020-11-18 Thread Daniel Becker (Code Review)
Daniel Becker has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/16728


Change subject: IMPALA-10332: Add file formats to HdfsScanNode's thrift 
representation.
..

IMPALA-10332: Add file formats to HdfsScanNode's thrift representation.

List all file formats that a HdfsScanNode needs to process in any
fragment instance. It is possible that some file formats will not be
needed in all fragment instances.

This is a step towards sharing codegen between different impala
backends. Using the file formats provided in the thrift file, a backend
can codegen code for file formats that are not needed in its own process
but are needed in other fragment instances running on other backends,
and the resulting binary can be shared between multiple backends.

Codegenning for file formats will be done based on the thrift message
and not on what is needed for the actual backend. This leads to some
extra work in case a file format is not needed for the current backend
and codegen sharing is not available (at this point it is not
implemented). However, the overall number of such cases is low.

Also adding the file formats to the node's explain string at level 3.

Testing:
 - Added tests to verify that the file formats are present in the
   explain string at level 3.

Change-Id: Iad6b8271bd248983f327c07883a3bedf50f25b5d
---
M be/src/exec/hdfs-scan-node-base.cc
M common/thrift/PlanNodes.thrift
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M testdata/workloads/functional-query/queries/QueryTest/explain-level3.test
4 files changed, 44 insertions(+), 8 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/28/16728/5
--
To view, visit http://gerrit.cloudera.org:8080/16728
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Iad6b8271bd248983f327c07883a3bedf50f25b5d
Gerrit-Change-Number: 16728
Gerrit-PatchSet: 5
Gerrit-Owner: Daniel Becker 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Daniel Becker