[Impala-ASF-CR] IMPALA-7869: break up parquet-column-readers.cc
Tim Armstrong has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/11949 ) Change subject: IMPALA-7869: break up parquet-column-readers.cc .. IMPALA-7869: break up parquet-column-readers.cc Move CollectionColumnReader and ParquetLevelDecoder into separate files. Switch BOOLEAN decoding to use composition instead of inheritance. This lets the boolean decoding use the faster batched implementations in ScalarColumnReader and avoids some confusing aspects of the class hierarchy, like the ReadValueBatch() implementation on the base class that was shared between BoolColumnReader and CollectionColumnReader. Improve compile times by instantiating BitPacking templates in a separate file (this looks to give a 30s+ speedup for compiling parquet-column-readers.cc). Testing: Ran exhaustive tests. Change-Id: I0efd5c50b781fe9e3c022b33c66c06cfb529c0b8 --- M be/src/benchmarks/bit-packing-benchmark.cc M be/src/exec/CMakeLists.txt M be/src/exec/hdfs-parquet-scanner.cc M be/src/exec/hdfs-parquet-scanner.h A be/src/exec/parquet-bool-decoder.cc A be/src/exec/parquet-bool-decoder.h A be/src/exec/parquet-collection-column-reader.cc A be/src/exec/parquet-collection-column-reader.h M be/src/exec/parquet-column-readers.cc M be/src/exec/parquet-column-readers.h M be/src/exec/parquet-common.h A be/src/exec/parquet-level-decoder.cc A be/src/exec/parquet-level-decoder.h M be/src/util/CMakeLists.txt M be/src/util/bit-packing-test.cc A be/src/util/bit-packing.cc M be/src/util/bit-packing.h M be/src/util/bit-packing.inline.h M be/src/util/bit-stream-utils.inline.h M be/src/util/dict-test.cc M be/src/util/rle-test.cc M common/thrift/generate_error_codes.py M testdata/workloads/functional-query/queries/QueryTest/parquet-num-values-def-levels-mismatch.test 23 files changed, 983 insertions(+), 683 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/11949/7 -- To view, visit http://gerrit.cloudera.org:8080/11949 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0efd5c50b781fe9e3c022b33c66c06cfb529c0b8 Gerrit-Change-Number: 11949 Gerrit-PatchSet: 7 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] IMPALA-7310: Use NDV=1 for a Column with all nulls
Paul Rogers has abandoned this change. ( http://gerrit.cloudera.org:8080/11528 ) Change subject: IMPALA-7310: Use NDV=1 for a Column with all nulls .. Abandoned Split test code into a separate patch. Will reopen this with just NDV changes after the test code is committed. -- To view, visit http://gerrit.cloudera.org:8080/11528 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: Ife657a43c9cafc451bd12ddf857dcb7169e97459 Gerrit-Change-Number: 11528 Gerrit-PatchSet: 15 Gerrit-Owner: Paul Rogers Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7655: Rewrite if, isnull to use CASE
Paul Rogers has abandoned this change. ( http://gerrit.cloudera.org:8080/11760 ) Change subject: IMPALA-7655: Rewrite if, isnull to use CASE .. Abandoned Will revisit after cleaning up blocking issues. -- To view, visit http://gerrit.cloudera.org:8080/11760 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: I526654d8546e17b2545c42cc59dab66d9fe1b163 Gerrit-Change-Number: 11760 Gerrit-PatchSet: 11 Gerrit-Owner: Paul Rogers Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] MPALA-7867, part 1: Expose List in TreeNode, parser
Paul Rogers has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11954 Change subject: MPALA-7867, part 1: Expose List in TreeNode, parser .. MPALA-7867, part 1: Expose List in TreeNode, parser When using Java collections, a common Java best practice is to expose the collection interface, but hide the implementation choice. This pattern allows us to start with a generic implementation (an ArrayList, say), but evolve to a more specific implementation to achieve certain goals (a LinkedList or ImmutableList, say.) For whatever reason, the Impala FE code exposes ArrayList, HashMap and other implementation choices as variable types and in method signatures. Also, since Java 7, the preferred way to create an array is new ArrayList<>() Replaced older forms: new ArrayList() // Pre-Java 7 Lists.newArrayList() // Guava form, pre-Java 7 This ticket cleans up two files, and their dependencies: * TreeNode (the root of all parser nodes) * sql-parser.cup (the code which creates the parser nodes) Many other uses exist, and will be submitted as separate patches to keep patches small. Tests: This is purely a refactoring, no functionality changed. Ran the FE unit tests to verify no regressions. Change-Id: Iebab7dccdb4b2fa0b5ca812beab0e8bdba39f539 --- M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java M fe/src/main/java/org/apache/impala/analysis/Expr.java M fe/src/main/java/org/apache/impala/analysis/FunctionName.java M fe/src/main/java/org/apache/impala/analysis/QueryStmt.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/SlotRef.java M fe/src/main/java/org/apache/impala/analysis/UnionStmt.java M fe/src/main/java/org/apache/impala/analysis/ValuesStmt.java M fe/src/main/java/org/apache/impala/analysis/WithClause.java M fe/src/main/java/org/apache/impala/catalog/StructType.java M fe/src/main/java/org/apache/impala/common/TreeNode.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/Planner.java M fe/src/main/java/org/apache/impala/service/Frontend.java 15 files changed, 99 insertions(+), 89 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/54/11954/1 -- To view, visit http://gerrit.cloudera.org:8080/11954 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Iebab7dccdb4b2fa0b5ca812beab0e8bdba39f539 Gerrit-Change-Number: 11954 Gerrit-PatchSet: 1 Gerrit-Owner: Paul Rogers
[Impala-ASF-CR] IMPALA-7804: Mitigate s3 consistency issues for test scanners
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11959 ) Change subject: IMPALA-7804: Mitigate s3 consistency issues for test_scanners .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/11959 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id042496beabe0d0226b347e0653b820fee369f4e Gerrit-Change-Number: 11959 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 20 Nov 2018 06:12:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7233: [DOCS] Support for IANA timezone database
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/11946 ) Change subject: IMPALA-7233: [DOCS] Support for IANA timezone database .. Patch Set 1: (6 comments) http://gerrit.cloudera.org:8080/#/c/11946/1/docs/topics/impala_timestamp.xml File docs/topics/impala_timestamp.xml: http://gerrit.cloudera.org:8080/#/c/11946/1/docs/topics/impala_timestamp.xml@180 PS1, Line 180: + INTERVAL n-hours can be affected Daylight Saving Time, which > nit: missing 'by'? Done http://gerrit.cloudera.org:8080/#/c/11946/1/docs/topics/impala_timestamp.xml@193 PS1, Line 193: Configuring custom time zones: > I agree. Done http://gerrit.cloudera.org:8080/#/c/11946/1/docs/topics/impala_timestamp.xml@193 PS1, Line 193: Configuring custom time zones: > I think that it would be better to move custom timezone databases/aliases t Done http://gerrit.cloudera.org:8080/#/c/11946/1/docs/topics/impala_timestamp.xml@272 PS1, Line 272: 2017c > Instead of '2017c' probably we should use a more generic directory name her Done http://gerrit.cloudera.org:8080/#/c/11946/1/docs/topics/impala_timestamp.xml@333 PS1, Line 333: values the same way it stores : without any adjustment. > This part of the sentence is not clear for me. Hive reads and writes TIMESTAMP values without converting with respect to time zones. Better? http://gerrit.cloudera.org:8080/#/c/11946/1/docs/topics/impala_timestamp.xml@367 PS1, Line 367: turned off by : default to avoid performance overhead > Turning on '-convert_legacy_hive_parquet_utc_timestamps' still has a cost, Done -- To view, visit http://gerrit.cloudera.org:8080/11946 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id400cda5a1be321063d17e0ee6337e92a5da732a Gerrit-Change-Number: 11946 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 20 Nov 2018 03:15:57 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7233: [DOCS] Support for IANA timezone database
Hello Attila Jeges, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11946 to look at the new patch set (#2). Change subject: IMPALA-7233: [DOCS] Support for IANA timezone database .. IMPALA-7233: [DOCS] Support for IANA timezone database - Updated the timezone section - Added the sections on customizing timezone db and aliases Change-Id: Id400cda5a1be321063d17e0ee6337e92a5da732a --- M docs/impala.ditamap A docs/topics/impala_custom_timezones.xml M docs/topics/impala_timestamp.xml 3 files changed, 337 insertions(+), 290 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/46/11946/2 -- To view, visit http://gerrit.cloudera.org:8080/11946 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id400cda5a1be321063d17e0ee6337e92a5da732a Gerrit-Change-Number: 11946 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-7233: [DOCS] Support for IANA timezone database
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11946 ) Change subject: IMPALA-7233: [DOCS] Support for IANA timezone database .. Patch Set 2: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/155/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/11946 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id400cda5a1be321063d17e0ee6337e92a5da732a Gerrit-Change-Number: 11946 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 20 Nov 2018 03:18:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7233: [DOCS] Support for IANA timezone database
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11946 ) Change subject: IMPALA-7233: [DOCS] Support for IANA timezone database .. Patch Set 2: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/155/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/11946 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id400cda5a1be321063d17e0ee6337e92a5da732a Gerrit-Change-Number: 11946 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 20 Nov 2018 03:15:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7804: Mitigate s3 consistency issues for test scanners
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11959 ) Change subject: IMPALA-7804: Mitigate s3 consistency issues for test_scanners .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1403/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11959 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id042496beabe0d0226b347e0653b820fee369f4e Gerrit-Change-Number: 11959 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 20 Nov 2018 02:55:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7804: Mitigate s3 consistency issues for test scanners
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11959 ) Change subject: IMPALA-7804: Mitigate s3 consistency issues for test_scanners .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1402/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11959 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id042496beabe0d0226b347e0653b820fee369f4e Gerrit-Change-Number: 11959 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 20 Nov 2018 02:40:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7804: Mitigate s3 consistency issues for test scanners
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/11959 ) Change subject: IMPALA-7804: Mitigate s3 consistency issues for test_scanners .. Patch Set 2: I'm running on s3 to see if it helps. This is somewhat experimental. -- To view, visit http://gerrit.cloudera.org:8080/11959 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id042496beabe0d0226b347e0653b820fee369f4e Gerrit-Change-Number: 11959 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 20 Nov 2018 02:08:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5031: Fix use of uninitialised var in SendTopicUpdate()
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11957 ) Change subject: IMPALA-5031: Fix use of uninitialised var in SendTopicUpdate() .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/11957 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I373c6f65854244fe90889412b38b0260fe1d1f13 Gerrit-Change-Number: 11957 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 20 Nov 2018 01:50:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7804: Mitigate s3 consistency issues for test scanners
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11959 ) Change subject: IMPALA-7804: Mitigate s3 consistency issues for test_scanners .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3479/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/11959 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id042496beabe0d0226b347e0653b820fee369f4e Gerrit-Change-Number: 11959 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 20 Nov 2018 02:08:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7804: Mitigate s3 consistency issues for test scanners
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11959 to look at the new patch set (#2). Change subject: IMPALA-7804: Mitigate s3 consistency issues for test_scanners .. IMPALA-7804: Mitigate s3 consistency issues for test_scanners test_scanners.py has seen several flaky failures on s3 due to eventual consistency. The symptom is Impala being unable to read a file that it just loaded to s3. A large number of tables used in test_scanners.py use the file_utils helper functions for creating the tables. These follow the pattern: 1. Copy files to temporary directory in HDFS/S3/etc 2. Create table 3. Run LOAD DATA to move the files to the table In step #3, LOAD DATA gets the metadata for the table before it runs the move statement on the files. Subsequent queries on the table will not need to reload metadata and can access the file quickly after the move. This changes the ordering to put the files in place before loading metadata. This may improve the likelihood that the filesystem is consistent by the time we read it. Specifically, we now do: 1. Put the files in directory that the table will use when it is created. 2. Create table Neither of these steps load metadata, so the next query that runs will load metadata. Change-Id: Id042496beabe0d0226b347e0653b820fee369f4e --- M tests/common/file_utils.py 1 file changed, 25 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/11959/2 -- To view, visit http://gerrit.cloudera.org:8080/11959 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id042496beabe0d0226b347e0653b820fee369f4e Gerrit-Change-Number: 11959 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-7804: Mitigate s3 consistency issues for test scanners
Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11959 Change subject: IMPALA-7804: Mitigate s3 consistency issues for test_scanners .. IMPALA-7804: Mitigate s3 consistency issues for test_scanners test_scanners.py has seen several flaky failures on s3 due to eventual consistency. The symptom is Impala being unable to read a file that it just loaded to s3. A large number of tables used in test_scanners.py use the file_utils helper functions for creating the tables. These follow the pattern: 1. Copy files to temporary directory in HDFS/S3/etc 2. Create table 3. Run LOAD DATA to move the files to the table In step #3, LOAD DATA gets the metadata for the table before it runs the move statement on the files. Subsequent queries on the table will not need to reload metadata and can access the file quickly after the move. This changes the ordering to put the files in place before loading metadata. This may improve the likelihood that the filesystem is consistent by the time we read it. Specifically, we now do: 1. Put the files in directory that the table will use when it is created. 2. Create table Neither of these steps load metadata, so the next query that runs will load metadata. Change-Id: Id042496beabe0d0226b347e0653b820fee369f4e --- M tests/common/file_utils.py 1 file changed, 24 insertions(+), 14 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/11959/1 -- To view, visit http://gerrit.cloudera.org:8080/11959 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Id042496beabe0d0226b347e0653b820fee369f4e Gerrit-Change-Number: 11959 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell
[Impala-ASF-CR] IMPALA-5031: Fix use of uninitialised var in SendTopicUpdate()
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11957 ) Change subject: IMPALA-5031: Fix use of uninitialised var in SendTopicUpdate() .. IMPALA-5031: Fix use of uninitialised var in SendTopicUpdate() When SendTopicUpdate() returns an error, it does not set update_skipped, so the variable is uninitialised. This means that the duration between topic updates will depend on the uninitialised value. Change-Id: I373c6f65854244fe90889412b38b0260fe1d1f13 Reviewed-on: http://gerrit.cloudera.org:8080/11957 Reviewed-by: Tim Armstrong Tested-by: Impala Public Jenkins --- M be/src/statestore/statestore.cc 1 file changed, 3 insertions(+), 1 deletion(-) Approvals: Tim Armstrong: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/11957 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I373c6f65854244fe90889412b38b0260fe1d1f13 Gerrit-Change-Number: 11957 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7829: Mark a fragment instance as done only after Close() is called
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11939 ) Change subject: IMPALA-7829: Mark a fragment instance as done only after Close() is called .. Patch Set 1: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/11939/1/be/src/runtime/fragment-instance-state.cc File be/src/runtime/fragment-instance-state.cc: http://gerrit.cloudera.org:8080/#/c/11939/1/be/src/runtime/fragment-instance-state.cc@100 PS1, Line 100: ReleaseThreadToken(); Consider moving to Close()? It's acquired in Prepare() so that feels symmetrical to me. -- To view, visit http://gerrit.cloudera.org:8080/11939 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I61618854ae3f4e7ef20028dcb0ff5cbcfa8adb01 Gerrit-Change-Number: 11939 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 20 Nov 2018 01:40:39 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7861: [DOCS] TLS enabled by default regardless of URI scheme
Alex Rodoni has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11948 ) Change subject: IMPALA-7861: [DOCS] TLS enabled by default regardless of URI scheme .. IMPALA-7861: [DOCS] TLS enabled by default regardless of URI scheme Change-Id: I88f615cf23f406035e544e68adacdd0393f69ab3 Reviewed-on: http://gerrit.cloudera.org:8080/11948 Reviewed-by: Joe McDonnell Tested-by: Impala Public Jenkins --- M docs/topics/impala_adls.xml 1 file changed, 43 insertions(+), 23 deletions(-) Approvals: Joe McDonnell: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/11948 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I88f615cf23f406035e544e68adacdd0393f69ab3 Gerrit-Change-Number: 11948 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell
[Impala-ASF-CR] IMPALA-7659: Simplify expression to collect NULLs count
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/11565 ) Change subject: IMPALA-7659: Simplify expression to collect NULLs count .. Patch Set 5: Thanks, Piotr for getting back. I'll take this one forward and submit a new PS that addresses the comments. -- To view, visit http://gerrit.cloudera.org:8080/11565 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic68f8b4c3756eb1980ce299a602a7d56db1e507a Gerrit-Change-Number: 11565 Gerrit-PatchSet: 5 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Tue, 20 Nov 2018 00:26:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7801: Remove toSql() from ParseNode interface.
Andrew Sherman has abandoned this change. ( http://gerrit.cloudera.org:8080/11942 ) Change subject: IMPALA-7801: Remove toSql() from ParseNode interface. .. Abandoned Will let Paul Rogers fix (or at least think about) in a bigger change -- To view, visit http://gerrit.cloudera.org:8080/11942 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: I17025901838e9ffd753894a8087170123f9d8b33 Gerrit-Change-Number: 11942 Gerrit-PatchSet: 4 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Thomas Marshall
[Impala-ASF-CR] IMPALA-7801: Remove toSql() from ParseNode interface.
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/11942 ) Change subject: IMPALA-7801: Remove toSql() from ParseNode interface. .. Patch Set 4: Thanks Csaba, hope you didn't waste too much time on this -- To view, visit http://gerrit.cloudera.org:8080/11942 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I17025901838e9ffd753894a8087170123f9d8b33 Gerrit-Change-Number: 11942 Gerrit-PatchSet: 4 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 20 Nov 2018 00:04:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5031: Zero-length arrays are undefined behavior
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11811 ) Change subject: IMPALA-5031: Zero-length arrays are undefined behavior .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/11811 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93f7d0b0506e4b6a2ff3303477d887a428431f96 Gerrit-Change-Number: 11811 Gerrit-PatchSet: 3 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Jinchul Kim Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 19 Nov 2018 23:41:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5031: Zero-length arrays are undefined behavior
Impala Public Jenkins has submitted this change and it was merged. (
http://gerrit.cloudera.org:8080/11811 )
Change subject: IMPALA-5031: Zero-length arrays are undefined behavior
..
IMPALA-5031: Zero-length arrays are undefined behavior
This patch prevents the creation of a zero-length array. This is
illegal under paragraph 5 of §6.7.5.2 ("Array declarators") of the C99
standard, which reads:
If the size is an expression that is not an integer constant
expression: if it occurs in a declaration at function prototype
scope, it is treated as if it were replaced by *; otherwise, each
time it is evaluated it shall have a value greater than zero.
Variable-length arrays are not part of C++14, but they are a common
compiler extension and are available in both clang++ and g++. That the
semantics of missing features are the same is implied by [intro.scope]
in the C++14 standard, which reads:
C++ is a general purpose programming language based on the C
programming language as described in ISO/IEC 9899:1999 Programming
languages — C (hereinafter referred to as the C standard).
Change-Id: I93f7d0b0506e4b6a2ff3303477d887a428431f96
Reviewed-on: http://gerrit.cloudera.org:8080/11811
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
---
M be/src/runtime/row-batch-serialize-test.cc
1 file changed, 1 insertion(+), 1 deletion(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/11811
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I93f7d0b0506e4b6a2ff3303477d887a428431f96
Gerrit-Change-Number: 11811
Gerrit-PatchSet: 4
Gerrit-Owner: Jim Apple
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Jim Apple
Gerrit-Reviewer: Jinchul Kim
Gerrit-Reviewer: Lars Volker
Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7871: Don't load Hive builtins
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11944 ) Change subject: IMPALA-7871: Don't load Hive builtins .. IMPALA-7871: Don't load Hive builtins Dataload has a step of "Loading Hive builtins" that loads a bunch of jars into HDFS/S3/etc. Despite its name, nothing seems to be using these. Dataload and core tests succeed without this step. This removes the Hive builtins step and associated scripts. Change-Id: Iaca5ffdaca4b5506e9401b17a7806d37fd7b1844 Reviewed-on: http://gerrit.cloudera.org:8080/11944 Reviewed-by: Tim Armstrong Tested-by: Impala Public Jenkins --- M testdata/bin/create-load-data.sh D testdata/bin/load-hive-builtins.sh M testdata/bin/load-test-warehouse-snapshot.sh 3 files changed, 0 insertions(+), 80 deletions(-) Approvals: Tim Armstrong: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/11944 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Iaca5ffdaca4b5506e9401b17a7806d37fd7b1844 Gerrit-Change-Number: 11944 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7871: Don't load Hive builtins
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11944 ) Change subject: IMPALA-7871: Don't load Hive builtins .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/11944 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaca5ffdaca4b5506e9401b17a7806d37fd7b1844 Gerrit-Change-Number: 11944 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 19 Nov 2018 23:27:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5031: method calls on NULL are not UBSAN-clean
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11950 ) Change subject: IMPALA-5031: method calls on NULL are not UBSAN-clean .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/11950 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8b38cb1ebba02fc163534ffcc95e4ebe41cbb115 Gerrit-Change-Number: 11950 Gerrit-PatchSet: 2 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Mon, 19 Nov 2018 23:11:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5031: method calls on NULL are not UBSAN-clean
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11950 ) Change subject: IMPALA-5031: method calls on NULL are not UBSAN-clean .. IMPALA-5031: method calls on NULL are not UBSAN-clean According to [expr.post] in the C++14 standard, a call to a member function like a->b() is interpreted as (a->b)(). In other words, the dereferencing is done separately from the call. This makes calling member functions on nullptr undefined behavior, since the dereference invokes undefined behavior. Change-Id: I8b38cb1ebba02fc163534ffcc95e4ebe41cbb115 Reviewed-on: http://gerrit.cloudera.org:8080/11950 Reviewed-by: Jim Apple Tested-by: Impala Public Jenkins --- M be/src/udf/udf.cc 1 file changed, 1 insertion(+), 3 deletions(-) Approvals: Jim Apple: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/11950 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I8b38cb1ebba02fc163534ffcc95e4ebe41cbb115 Gerrit-Change-Number: 11950 Gerrit-PatchSet: 3 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Lars Volker
[Impala-ASF-CR] IMPALA-7857: log more information about statestore failure detection
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11937 ) Change subject: IMPALA-7857: log more information about statestore failure detection .. IMPALA-7857: log more information about statestore failure detection This adds a couple of log messages for state transitions in the statestore's failure detector. Testing: Ran test_statestore.py and checked for presence of new log messages. Added a new tests to test_statestore that exercises handling of intermittent heartbeat failures (required to produce one of the new log messages). Change-Id: Ie6ff85bee117000e4434dcffd3d1680a79905f14 Reviewed-on: http://gerrit.cloudera.org:8080/11937 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/statestore/failure-detector.cc M be/src/statestore/failure-detector.h M tests/statestore/test_statestore.py 3 files changed, 44 insertions(+), 9 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/11937 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ie6ff85bee117000e4434dcffd3d1680a79905f14 Gerrit-Change-Number: 11937 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Anuj Phadke Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-7857: log more information about statestore failure detection
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11937 ) Change subject: IMPALA-7857: log more information about statestore failure detection .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/11937 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie6ff85bee117000e4434dcffd3d1680a79905f14 Gerrit-Change-Number: 11937 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Anuj Phadke Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 19 Nov 2018 22:55:39 + Gerrit-HasComments: No
[Impala-ASF-CR] Revert "IMPALA-6910/IMPALA-7070: Increase log level for HDFS S3 code"
Tim Armstrong has abandoned this change. ( http://gerrit.cloudera.org:8080/11699 ) Change subject: Revert "IMPALA-6910/IMPALA-7070: Increase log level for HDFS S3 code" .. Abandoned Will abandon for now to reflect that it's not ready for review. -- To view, visit http://gerrit.cloudera.org:8080/11699 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: I5ab7f2f6317f1281928af5ea6cf3fd7d0c6e0a09 Gerrit-Change-Number: 11699 Gerrit-PatchSet: 2 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7670: Avoid getting the latest tables in bulkAlterPartitions()
Tianyi Wang has abandoned this change. ( http://gerrit.cloudera.org:8080/11641 ) Change subject: IMPALA-7670: Avoid getting the latest tables in bulkAlterPartitions() .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/11641 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: I0ec120f9df64d6e7e7d4978b5e190376721a6897 Gerrit-Change-Number: 11641 Gerrit-PatchSet: 1 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-5031: Fix use of uninitialised var in SendTopicUpdate()
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11957 ) Change subject: IMPALA-5031: Fix use of uninitialised var in SendTopicUpdate() .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3478/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/11957 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I373c6f65854244fe90889412b38b0260fe1d1f13 Gerrit-Change-Number: 11957 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 19 Nov 2018 21:51:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5031: Fix use of uninitialised var in SendTopicUpdate()
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11957 ) Change subject: IMPALA-5031: Fix use of uninitialised var in SendTopicUpdate() .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11957 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I373c6f65854244fe90889412b38b0260fe1d1f13 Gerrit-Change-Number: 11957 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 19 Nov 2018 21:51:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5031: Fix use of uninitialised var in SendTopicUpdate()
Hello Jim Apple, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11957 to look at the new patch set (#2). Change subject: IMPALA-5031: Fix use of uninitialised var in SendTopicUpdate() .. IMPALA-5031: Fix use of uninitialised var in SendTopicUpdate() When SendTopicUpdate() returns an error, it does not set update_skipped, so the variable is uninitialised. This means that the duration between topic updates will depend on the uninitialised value. Change-Id: I373c6f65854244fe90889412b38b0260fe1d1f13 --- M be/src/statestore/statestore.cc 1 file changed, 3 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/11957/2 -- To view, visit http://gerrit.cloudera.org:8080/11957 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I373c6f65854244fe90889412b38b0260fe1d1f13 Gerrit-Change-Number: 11957 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] Fix use of uninitialised variale in SendTopicUpdate()
Jim Apple has posted comments on this change. ( http://gerrit.cloudera.org:8080/11957 ) Change subject: Fix use of uninitialised variale in SendTopicUpdate() .. Patch Set 1: Code-Review+2 Thanks for fixing this! Feel free to tag an "IMPALA-5031: " on to the front of the git commit first line if you feel like it. -- To view, visit http://gerrit.cloudera.org:8080/11957 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I373c6f65854244fe90889412b38b0260fe1d1f13 Gerrit-Change-Number: 11957 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 19 Nov 2018 21:47:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6964: Track stats about column and page sizes in Parquet reader
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11575 ) Change subject: IMPALA-6964: Track stats about column and page sizes in Parquet reader .. Patch Set 12: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1401/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11575 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I322f9b324b6828df28e5caf79529085c43d7c817 Gerrit-Change-Number: 11575 Gerrit-PatchSet: 12 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Mon, 19 Nov 2018 21:09:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6656: BufferAllocator observability
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11947 ) Change subject: IMPALA-6656: BufferAllocator observability .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1400/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11947 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I12b740b8ea7773b3215681531dfa62db55cfdf18 Gerrit-Change-Number: 11947 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 19 Nov 2018 20:38:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6964: Track stats about column and page sizes in Parquet reader
Hello Lars Volker, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11575 to look at the new patch set (#12). Change subject: IMPALA-6964: Track stats about column and page sizes in Parquet reader .. IMPALA-6964: Track stats about column and page sizes in Parquet reader Adds the following new stats: * ParquetCompressedPageSize - a summary (average, min, max) counter that tracks the size of compressed pages read, if no compressed pages are read then this counter is empty * ParquetUncompressedPageSize - a summary counter that tracks the size of uncompressed pages read, it is updated in two places: (1) when a compressed page is de-compressed, and (2) when a page that is not compressed is read * ParquetCompressedDataReadPerColumn - a summary counter that tracks the amount of compressed data read per column for a scan node * ParquetUncompressedDataReadPerColumn - a summary counter that tracks the amount of uncompressed data read per column for a scan node The PerColumn counters are calculated by aggregating the number of bytes read for each column across all scan ranges processed by a scan node. Each sample in the counter is the size of a single column. Here is an example of what the updated HDFS scan profile looks like: - ParquetCompressedDataReadPerColumn: (Avg: 227.56 KB (233018) ; Min: 225.14 KB (230540) ; Max: 229.98 KB (235496) ; Number of samples: 2) - ParquetUncompressedDataReadPerColumn: (Avg: 227.96 KB (233426) ; Min: 224.91 KB (230306) ; Max: 231.00 KB (236547) ; Number of samples: 2) - ParquetCompressedPageSize: (Avg: 4.46 KB (4568) ; Min: 3.86 KB (3955) ; Max: 5.19 KB (5315) ; Number of samples: 102) - ParquetDecompressedPageSize: (Avg: 4.47 KB (4576) ; Min: 3.86 KB (3950) ; Max: 5.22 KB (5349) ; Number of samples: 102) Testing: * Added new tests to test_scanners.py that do some basic validation of the new counters above Change-Id: I322f9b324b6828df28e5caf79529085c43d7c817 --- M be/src/exec/hdfs-parquet-scanner.cc M be/src/exec/hdfs-parquet-scanner.h M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/hdfs-scan-node-base.h M be/src/exec/parquet-column-readers.cc M be/src/util/runtime-profile.cc M tests/infra/test_utils.py M tests/query_test/test_scanners.py M tests/util/parse_util.py 9 files changed, 260 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/11575/12 -- To view, visit http://gerrit.cloudera.org:8080/11575 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I322f9b324b6828df28e5caf79529085c43d7c817 Gerrit-Change-Number: 11575 Gerrit-PatchSet: 12 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Sahil Takiar
[Impala-ASF-CR] Fix use of uninitialised variale in SendTopicUpdate()
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11957 ) Change subject: Fix use of uninitialised variale in SendTopicUpdate() .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1399/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11957 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I373c6f65854244fe90889412b38b0260fe1d1f13 Gerrit-Change-Number: 11957 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 19 Nov 2018 20:29:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7865: Repeated type widening of arithmetic expressions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11952 ) Change subject: IMPALA-7865: Repeated type widening of arithmetic expressions .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1398/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11952 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3a8f6d1c508a289083b5e026f33581bc44117ca2 Gerrit-Change-Number: 11952 Gerrit-PatchSet: 2 Gerrit-Owner: Paul Rogers Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Comment-Date: Mon, 19 Nov 2018 20:30:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7866: Predicates, helpers for implicit casts, slot refs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11953 ) Change subject: IMPALA-7866: Predicates, helpers for implicit casts, slot refs .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1397/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11953 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ieaa0aee1b9015e0aed521f2038bf44513d7f8613 Gerrit-Change-Number: 11953 Gerrit-PatchSet: 1 Gerrit-Owner: Paul Rogers Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Comment-Date: Mon, 19 Nov 2018 20:21:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6656: BufferAllocator observability
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11947 to look at the new patch set (#3). Change subject: IMPALA-6656: BufferAllocator observability .. IMPALA-6656: BufferAllocator observability Adds a set of metrics per allocator arena in the buffer pool that help understand how buffers are being allocated and how much time is spent in the system allocator (i.e. TCMalloc). These are low level metrics that require some interpretation but provide visibility into behaviour that was previously totally opaque. Also tracks the total time spent in the system allocator in the query profile, to provide clues if time spent in TCMalloc is a perf issue for a particular query (e.g. if it's hitting a lot of lock contention). Backend tests required tweaks to avoid double-registration of the new metrics. Also switch default sort in /metrics to be by name, so that it's easier to locate metrics. Change-Id: I12b740b8ea7773b3215681531dfa62db55cfdf18 --- M be/src/runtime/bufferpool/buffer-allocator-test.cc M be/src/runtime/bufferpool/buffer-allocator.cc M be/src/runtime/bufferpool/buffer-allocator.h M be/src/runtime/bufferpool/buffer-pool-counters.h M be/src/runtime/bufferpool/buffer-pool-test.cc M be/src/runtime/bufferpool/buffer-pool.cc M be/src/runtime/bufferpool/buffer-pool.h M be/src/runtime/bufferpool/suballocator-test.cc M be/src/runtime/exec-env.cc M be/src/runtime/test-env.cc M be/src/runtime/test-env.h M be/src/util/metrics.h M common/thrift/metrics.json M www/metric_group.tmpl 14 files changed, 271 insertions(+), 72 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/47/11947/3 -- To view, visit http://gerrit.cloudera.org:8080/11947 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I12b740b8ea7773b3215681531dfa62db55cfdf18 Gerrit-Change-Number: 11947 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-6964: Track stats about column and page sizes in Parquet reader
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11575 ) Change subject: IMPALA-6964: Track stats about column and page sizes in Parquet reader .. Patch Set 11: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/1396/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/11575 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I322f9b324b6828df28e5caf79529085c43d7c817 Gerrit-Change-Number: 11575 Gerrit-PatchSet: 11 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Mon, 19 Nov 2018 20:00:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6656: BufferAllocator observability
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11947 ) Change subject: IMPALA-6656: BufferAllocator observability .. Patch Set 2: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/1395/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/11947 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I12b740b8ea7773b3215681531dfa62db55cfdf18 Gerrit-Change-Number: 11947 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 19 Nov 2018 19:54:41 + Gerrit-HasComments: No
[Impala-ASF-CR] Fix use of uninitialised variale in SendTopicUpdate()
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11957 ) Change subject: Fix use of uninitialised variale in SendTopicUpdate() .. Patch Set 1: I just noticed while reading code. I thought I should just fix it. -- To view, visit http://gerrit.cloudera.org:8080/11957 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I373c6f65854244fe90889412b38b0260fe1d1f13 Gerrit-Change-Number: 11957 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 19 Nov 2018 19:54:27 + Gerrit-HasComments: No
[Impala-ASF-CR] Fix use of uninitialised variale in SendTopicUpdate()
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11957 Change subject: Fix use of uninitialised variale in SendTopicUpdate() .. Fix use of uninitialised variale in SendTopicUpdate() When SendTopicUpdate() returns an error, it does not set update_skipped, so the variable is uninitialised. This means that the duration between topic updates will depend on the uninitialised value. Change-Id: I373c6f65854244fe90889412b38b0260fe1d1f13 --- M be/src/statestore/statestore.cc 1 file changed, 3 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/11957/1 -- To view, visit http://gerrit.cloudera.org:8080/11957 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I373c6f65854244fe90889412b38b0260fe1d1f13 Gerrit-Change-Number: 11957 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] IMPALA-7865: Repeated type widening of arithmetic expressions
Paul Rogers has posted comments on this change. ( http://gerrit.cloudera.org:8080/11952 ) Change subject: IMPALA-7865: Repeated type widening of arithmetic expressions .. Patch Set 2: Pre-commit tests passed: https://jenkins.impala.io/job/pre-review-test/229/ -- To view, visit http://gerrit.cloudera.org:8080/11952 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3a8f6d1c508a289083b5e026f33581bc44117ca2 Gerrit-Change-Number: 11952 Gerrit-PatchSet: 2 Gerrit-Owner: Paul Rogers Gerrit-Reviewer: Paul Rogers Gerrit-Comment-Date: Mon, 19 Nov 2018 19:37:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7865: Repeated type widening of arithmetic expressions
Paul Rogers has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11952 Change subject: IMPALA-7865: Repeated type widening of arithmetic expressions .. IMPALA-7865: Repeated type widening of arithmetic expressions In its present form, the analyser will analyze some expressions multiple times: during analysis, after substututing references, again after performing rewrites. The analyzer implements a set of type propagation rules that generates a resulting type for arithmetic expressions that is wider than the types of the arguments. For example for a + 1, a and 1 are TINYINT, but the result of the expression is promoted to SMALLINT. The planner then sets the type of the constant (1 here) to SMALLINT. Now, repeat the process on the next cycle. a is TINYINT, 1 is a SMALLINT. Now the result of the expression is INT and 1 is retyped to be an INT. Repeat again and the expression (and constant) are promoted to BIGINT. Meanwhile, analysis has taken a clone of the expression with the old types. As a result, the types of columns in the result list for a SELECT statement can differ from the same columns recorded in the SELECT list. This patch applies a number of partial fixes: * When analyzing a numeric expression, the type of any constant argument is reset back to the default. (TINYINT in the example above.) * When performing substitution, expressions are re-analyzed only if that expression itself changed, or one of its children changed. A full fix is to enhance the analyzer to do a bottom-up, one pass analysis, which is beyond the scope of this change. Tests: * Reran all FE tests, adjusting one Planner test. * Added a new test case that demonstrates that types are widened only statically (for a + 1 + 1, say), but not dynamically (due to repeated re-analysis.) Change-Id: I3a8f6d1c508a289083b5e026f33581bc44117ca2 --- M fe/src/main/java/org/apache/impala/analysis/ArithmeticExpr.java M fe/src/main/java/org/apache/impala/analysis/Expr.java M fe/src/main/java/org/apache/impala/analysis/NumericLiteral.java M fe/src/test/java/org/apache/impala/analysis/ExprRewriteRulesTest.java M fe/src/test/java/org/apache/impala/analysis/ExprRewriterTest.java M testdata/workloads/functional-planner/queries/PlannerTest/predicate-propagation.test 6 files changed, 112 insertions(+), 32 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/11952/2 -- To view, visit http://gerrit.cloudera.org:8080/11952 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I3a8f6d1c508a289083b5e026f33581bc44117ca2 Gerrit-Change-Number: 11952 Gerrit-PatchSet: 2 Gerrit-Owner: Paul Rogers
[Impala-ASF-CR] IMPALA-7866: Predicates, helpers for implicit casts, slot refs
Paul Rogers has posted comments on this change. ( http://gerrit.cloudera.org:8080/11953 ) Change subject: IMPALA-7866: Predicates, helpers for implicit casts, slot refs .. Patch Set 1: Pre-commit tests passed: https://jenkins.impala.io/job/pre-review-test/228/ -- To view, visit http://gerrit.cloudera.org:8080/11953 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ieaa0aee1b9015e0aed521f2038bf44513d7f8613 Gerrit-Change-Number: 11953 Gerrit-PatchSet: 1 Gerrit-Owner: Paul Rogers Gerrit-Reviewer: Paul Rogers Gerrit-Comment-Date: Mon, 19 Nov 2018 19:33:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7866: Predicates, helpers for implicit casts, slot refs
Paul Rogers has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11953 Change subject: IMPALA-7866: Predicates, helpers for implicit casts, slot refs .. IMPALA-7866: Predicates, helpers for implicit casts, slot refs * Refactors implicit cast functions into a predicate and a static helper rather than the methods on the Expr base class. * Adds an IS_SLOT_REF predicate and replaces explcit instanceof checks with uses of the predicate. * Other minor clean-up. Tests: No functional changes, just refactoring. Reran existing tests. Change-Id: Ieaa0aee1b9015e0aed521f2038bf44513d7f8613 --- M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/CastExpr.java M fe/src/main/java/org/apache/impala/analysis/Expr.java M fe/src/main/java/org/apache/impala/analysis/PartitionSet.java M fe/src/main/java/org/apache/impala/analysis/QueryStmt.java M fe/src/main/java/org/apache/impala/analysis/SelectListItem.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/SlotRef.java M fe/src/main/java/org/apache/impala/analysis/SortInfo.java M fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java M fe/src/main/java/org/apache/impala/analysis/UnionStmt.java M fe/src/main/java/org/apache/impala/catalog/AggregateFunction.java M fe/src/main/java/org/apache/impala/common/TreeNode.java M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java M fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/main/java/org/apache/impala/planner/SortNode.java M fe/src/main/java/org/apache/impala/rewrite/RemoveRedundantStringCast.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java 20 files changed, 100 insertions(+), 95 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/11953/1 -- To view, visit http://gerrit.cloudera.org:8080/11953 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ieaa0aee1b9015e0aed521f2038bf44513d7f8613 Gerrit-Change-Number: 11953 Gerrit-PatchSet: 1 Gerrit-Owner: Paul Rogers
[Impala-ASF-CR] IMPALA-6964: Track stats about column and page sizes in Parquet reader
Hello Lars Volker, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11575 to look at the new patch set (#11). Change subject: IMPALA-6964: Track stats about column and page sizes in Parquet reader .. IMPALA-6964: Track stats about column and page sizes in Parquet reader Adds the following new stats: * ParquetCompressedPageSize - a summary (average, min, max) counter that tracks the size of compressed pages read, if no compressed pages are read then this counter is empty * ParquetUncompressedPageSize - a summary counter that tracks the size of uncompressed pages read, it is updated in two places: (1) when a compressed page is de-compressed, and (2) when a page that is not compressed is read * ParquetCompressedDataReadPerColumn - a summary counter that tracks the amount of compressed data read per column for a scan node * ParquetUncompressedDataReadPerColumn - a summary counter that tracks the amount of uncompressed data read per column for a scan node The PerColumn counters are calculated by aggregating the number of bytes read for each column across all scan ranges processed by a scan node. Each sample in the counter is the size of a single column. Here is an example of what the updated HDFS scan profile looks like: - ParquetCompressedDataReadPerColumn: (Avg: 227.56 KB (233018) ; Min: 225.14 KB (230540) ; Max: 229.98 KB (235496) ; Number of samples: 2) - ParquetUncompressedDataReadPerColumn: (Avg: 227.96 KB (233426) ; Min: 224.91 KB (230306) ; Max: 231.00 KB (236547) ; Number of samples: 2) - ParquetCompressedPageSize: (Avg: 4.46 KB (4568) ; Min: 3.86 KB (3955) ; Max: 5.19 KB (5315) ; Number of samples: 102) - ParquetDecompressedPageSize: (Avg: 4.47 KB (4576) ; Min: 3.86 KB (3950) ; Max: 5.22 KB (5349) ; Number of samples: 102) Testing: * Added new tests to test_scanners.py that do some basic validation of the new counters above Change-Id: I322f9b324b6828df28e5caf79529085c43d7c817 --- M be/src/exec/hdfs-parquet-scanner.cc M be/src/exec/hdfs-parquet-scanner.h M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/hdfs-scan-node-base.h M be/src/exec/parquet-column-readers.cc M be/src/util/runtime-profile.cc M tests/infra/test_utils.py M tests/query_test/test_scanners.py M tests/util/parse_util.py 9 files changed, 259 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/11575/11 -- To view, visit http://gerrit.cloudera.org:8080/11575 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I322f9b324b6828df28e5caf79529085c43d7c817 Gerrit-Change-Number: 11575 Gerrit-PatchSet: 11 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Sahil Takiar
[Impala-ASF-CR] IMPALA-5031: Zero-length arrays are undefined behavior
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11811 ) Change subject: IMPALA-5031: Zero-length arrays are undefined behavior .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3477/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/11811 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93f7d0b0506e4b6a2ff3303477d887a428431f96 Gerrit-Change-Number: 11811 Gerrit-PatchSet: 3 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Jinchul Kim Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 19 Nov 2018 19:23:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5031: Zero-length arrays are undefined behavior
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11811 ) Change subject: IMPALA-5031: Zero-length arrays are undefined behavior .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11811 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93f7d0b0506e4b6a2ff3303477d887a428431f96 Gerrit-Change-Number: 11811 Gerrit-PatchSet: 3 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Jinchul Kim Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 19 Nov 2018 19:23:41 + Gerrit-HasComments: No
[Impala-ASF-CR] Revert "IMPALA-6910/IMPALA-7070: Increase log level for HDFS S3 code"
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/11699 ) Change subject: Revert "IMPALA-6910/IMPALA-7070: Increase log level for HDFS S3 code" .. Patch Set 2: > Any luck? I hit some issues with the S3 tests and couldn't get to a point where I could validate my fix. I'll try to get to this when I have some cycles but I don't have an ETA. -- To view, visit http://gerrit.cloudera.org:8080/11699 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5ab7f2f6317f1281928af5ea6cf3fd7d0c6e0a09 Gerrit-Change-Number: 11699 Gerrit-PatchSet: 2 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 19 Nov 2018 19:19:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7871: Don't load Hive builtins
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11944 ) Change subject: IMPALA-7871: Don't load Hive builtins .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3476/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/11944 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaca5ffdaca4b5506e9401b17a7806d37fd7b1844 Gerrit-Change-Number: 11944 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 19 Nov 2018 19:16:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5031: Zero-length arrays are undefined behavior
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11811 ) Change subject: IMPALA-5031: Zero-length arrays are undefined behavior .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11811 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93f7d0b0506e4b6a2ff3303477d887a428431f96 Gerrit-Change-Number: 11811 Gerrit-PatchSet: 2 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Jinchul Kim Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 19 Nov 2018 19:09:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7871: Don't load Hive builtins
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11944 ) Change subject: IMPALA-7871: Don't load Hive builtins .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1394/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11944 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaca5ffdaca4b5506e9401b17a7806d37fd7b1844 Gerrit-Change-Number: 11944 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 19 Nov 2018 19:06:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6656: BufferAllocator observability
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11947 ) Change subject: IMPALA-6656: BufferAllocator observability .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/11947/2/be/src/runtime/bufferpool/buffer-pool.cc File be/src/runtime/bufferpool/buffer-pool.cc: http://gerrit.cloudera.org:8080/#/c/11947/2/be/src/runtime/bufferpool/buffer-pool.cc@115 PS2, Line 115: // TODO: register metrics Need to remove TODO -- To view, visit http://gerrit.cloudera.org:8080/11947 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I12b740b8ea7773b3215681531dfa62db55cfdf18 Gerrit-Change-Number: 11947 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 19 Nov 2018 18:59:00 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7409. CatalogObjectVersionSet should not allow duplicates
Bharath Vissapragada has abandoned this change. ( http://gerrit.cloudera.org:8080/11151 ) Change subject: IMPALA-7409. CatalogObjectVersionSet should not allow duplicates .. Abandoned Abandoning for now, feel free to reopen. -- To view, visit http://gerrit.cloudera.org:8080/11151 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: I318e59c6a7fe559e86693fce1255fe6d9f829ccf Gerrit-Change-Number: 11151 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-7541. Avoid initializing Metrics for IncompleteTables
Bharath Vissapragada has abandoned this change. ( http://gerrit.cloudera.org:8080/11393 ) Change subject: IMPALA-7541. Avoid initializing Metrics for IncompleteTables .. Abandoned Todd, abandoning for now, doesn't look like the right fix. Feel free to reopen. -- To view, visit http://gerrit.cloudera.org:8080/11393 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: Id0bcaa9c45a8cf75266d4b7c16cc14f0fd669b92 Gerrit-Change-Number: 11393 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-6656: BufferAllocator observability
Tim Armstrong has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/11947 ) Change subject: IMPALA-6656: BufferAllocator observability .. IMPALA-6656: BufferAllocator observability Adds a set of metrics per allocator arena in the buffer pool that help understand how buffers are being allocated and how much time is spent in the system allocator (i.e. TCMalloc). These are low level metrics that require some interpretation but provide visibility into behaviour that was previously totally opaque. Also tracks the total time spent in the system allocator in the query profile, to provide clues if time spent in TCMalloc is a perf issue for a particular query (e.g. if it's hitting a lot of lock contention). Backend tests required tweaks to avoid double-registration of the new metrics. Also switch default sort in /metrics to be by name, so that it's easier to locate metrics. Change-Id: I12b740b8ea7773b3215681531dfa62db55cfdf18 --- M be/src/runtime/bufferpool/buffer-allocator-test.cc M be/src/runtime/bufferpool/buffer-allocator.cc M be/src/runtime/bufferpool/buffer-allocator.h M be/src/runtime/bufferpool/buffer-pool-counters.h M be/src/runtime/bufferpool/buffer-pool-test.cc M be/src/runtime/bufferpool/buffer-pool.cc M be/src/runtime/bufferpool/buffer-pool.h M be/src/runtime/bufferpool/suballocator-test.cc M be/src/runtime/exec-env.cc M be/src/runtime/test-env.cc M be/src/runtime/test-env.h M be/src/util/metrics.h M common/thrift/metrics.json M www/metric_group.tmpl 14 files changed, 272 insertions(+), 72 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/47/11947/2 -- To view, visit http://gerrit.cloudera.org:8080/11947 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I12b740b8ea7773b3215681531dfa62db55cfdf18 Gerrit-Change-Number: 11947 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] IMPALA-7670: Avoid getting the latest tables in bulkAlterPartitions()
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/11641 ) Change subject: IMPALA-7670: Avoid getting the latest tables in bulkAlterPartitions() .. Patch Set 1: Can this be abandoned for now? Don't think this is the correct fix. -- To view, visit http://gerrit.cloudera.org:8080/11641 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0ec120f9df64d6e7e7d4978b5e190376721a6897 Gerrit-Change-Number: 11641 Gerrit-PatchSet: 1 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Mon, 19 Nov 2018 18:57:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5031: method calls on NULL are not UBSAN-clean
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11950 ) Change subject: IMPALA-5031: method calls on NULL are not UBSAN-clean .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3475/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/11950 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8b38cb1ebba02fc163534ffcc95e4ebe41cbb115 Gerrit-Change-Number: 11950 Gerrit-PatchSet: 2 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Mon, 19 Nov 2018 18:55:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7871: Don't load Hive builtins
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11944 ) Change subject: IMPALA-7871: Don't load Hive builtins .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11944 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaca5ffdaca4b5506e9401b17a7806d37fd7b1844 Gerrit-Change-Number: 11944 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 19 Nov 2018 18:55:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5031: method calls on NULL are not UBSAN-clean
Jim Apple has posted comments on this change. ( http://gerrit.cloudera.org:8080/11950 ) Change subject: IMPALA-5031: method calls on NULL are not UBSAN-clean .. Patch Set 2: Code-Review+2 carry Lars's -- To view, visit http://gerrit.cloudera.org:8080/11950 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8b38cb1ebba02fc163534ffcc95e4ebe41cbb115 Gerrit-Change-Number: 11950 Gerrit-PatchSet: 2 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Mon, 19 Nov 2018 18:55:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7857: log more information about statestore failure detection
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11937 ) Change subject: IMPALA-7857: log more information about statestore failure detection .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3474/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/11937 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie6ff85bee117000e4434dcffd3d1680a79905f14 Gerrit-Change-Number: 11937 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Anuj Phadke Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 19 Nov 2018 18:55:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7857: log more information about statestore failure detection
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11937 ) Change subject: IMPALA-7857: log more information about statestore failure detection .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11937 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie6ff85bee117000e4434dcffd3d1680a79905f14 Gerrit-Change-Number: 11937 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Anuj Phadke Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 19 Nov 2018 18:55:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5031: method calls on NULL are not UBSAN-clean
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/11950 ) Change subject: IMPALA-5031: method calls on NULL are not UBSAN-clean .. Patch Set 1: Code-Review+2 Thanks for fixing this! -- To view, visit http://gerrit.cloudera.org:8080/11950 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8b38cb1ebba02fc163534ffcc95e4ebe41cbb115 Gerrit-Change-Number: 11950 Gerrit-PatchSet: 1 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Mon, 19 Nov 2018 18:54:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7585: support LDAP in run-workload.py
Jim Apple has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11938 ) Change subject: IMPALA-7585: support LDAP in run-workload.py .. IMPALA-7585: support LDAP in run-workload.py This patch just threads through the user, password, and ssl settings all the way back to the ImpalaBeeswaxClient. Change-Id: Ibfa987d8a027f50bc1ba3db5aa355331442a74ba Reviewed-on: http://gerrit.cloudera.org:8080/11938 Tested-by: Impala Public Jenkins Reviewed-by: David Knupp --- M bin/run-workload.py M tests/performance/query_exec_functions.py M tests/performance/query_executor.py M tests/performance/workload_runner.py 4 files changed, 21 insertions(+), 2 deletions(-) Approvals: Impala Public Jenkins: Verified David Knupp: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/11938 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ibfa987d8a027f50bc1ba3db5aa355331442a74ba Gerrit-Change-Number: 11938 Gerrit-PatchSet: 6 Gerrit-Owner: Jim Apple Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple
[Impala-ASF-CR] IMPALA-7857: log more information about statestore failure detection
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/11937 ) Change subject: IMPALA-7857: log more information about statestore failure detection .. Patch Set 2: Code-Review+2 Thanks for fixing this. Very helpful in diagnosing flaky heartbeats with SS. -- To view, visit http://gerrit.cloudera.org:8080/11937 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie6ff85bee117000e4434dcffd3d1680a79905f14 Gerrit-Change-Number: 11937 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Anuj Phadke Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 19 Nov 2018 18:45:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7585: support LDAP in run-workload.py
David Knupp has posted comments on this change. ( http://gerrit.cloudera.org:8080/11938 ) Change subject: IMPALA-7585: support LDAP in run-workload.py .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11938 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibfa987d8a027f50bc1ba3db5aa355331442a74ba Gerrit-Change-Number: 11938 Gerrit-PatchSet: 5 Gerrit-Owner: Jim Apple Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Comment-Date: Mon, 19 Nov 2018 18:41:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7801: Remove toSql() from ParseNode interface.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/11942 ) Change subject: IMPALA-7801: Remove toSql() from ParseNode interface. .. Patch Set 4: (6 comments) My comments are not very useful if you plan to abandon the change, but here they are. http://gerrit.cloudera.org:8080/#/c/11942/4//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11942/4//COMMIT_MSG@23 PS4, Line 23: anbd noive typo: something seems wrong here http://gerrit.cloudera.org:8080/#/c/11942/4/fe/src/main/java/org/apache/impala/analysis/Analyzer.java File fe/src/main/java/org/apache/impala/analysis/Analyzer.java: http://gerrit.cloudera.org:8080/#/c/11942/4/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@1542 PS4, Line 1542: + srcConjunct.toSql(ORIGINAL), : e); nit: could be one line http://gerrit.cloudera.org:8080/#/c/11942/4/fe/src/main/java/org/apache/impala/analysis/ColumnDef.java File fe/src/main/java/org/apache/impala/analysis/ColumnDef.java: http://gerrit.cloudera.org:8080/#/c/11942/4/fe/src/main/java/org/apache/impala/analysis/ColumnDef.java@250 PS4, Line 250: defaultValue_.toSql(ORIGINAL)), : e); nit: could be one line http://gerrit.cloudera.org:8080/#/c/11942/4/fe/src/main/java/org/apache/impala/analysis/ColumnDef.java@310 PS4, Line 310: defaultValue_.toSql(ORIGINAL)), : e); nit: could be one line http://gerrit.cloudera.org:8080/#/c/11942/4/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java File fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java: http://gerrit.cloudera.org:8080/#/c/11942/4/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java@712 PS4, Line 712: + conjunct.toSql(ORIGINAL), : e); nit: could be one line http://gerrit.cloudera.org:8080/#/c/11942/4/fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java File fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java: http://gerrit.cloudera.org:8080/#/c/11942/4/fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java@373 PS4, Line 373: e); nit: could be one line -- To view, visit http://gerrit.cloudera.org:8080/11942 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I17025901838e9ffd753894a8087170123f9d8b33 Gerrit-Change-Number: 11942 Gerrit-PatchSet: 4 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Mon, 19 Nov 2018 18:33:21 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7871: Don't load Hive builtins
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11944 to look at the new patch set (#2). Change subject: IMPALA-7871: Don't load Hive builtins .. IMPALA-7871: Don't load Hive builtins Dataload has a step of "Loading Hive builtins" that loads a bunch of jars into HDFS/S3/etc. Despite its name, nothing seems to be using these. Dataload and core tests succeed without this step. This removes the Hive builtins step and associated scripts. Change-Id: Iaca5ffdaca4b5506e9401b17a7806d37fd7b1844 --- M testdata/bin/create-load-data.sh D testdata/bin/load-hive-builtins.sh M testdata/bin/load-test-warehouse-snapshot.sh 3 files changed, 0 insertions(+), 80 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/44/11944/2 -- To view, visit http://gerrit.cloudera.org:8080/11944 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iaca5ffdaca4b5506e9401b17a7806d37fd7b1844 Gerrit-Change-Number: 11944 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-7367: Pack StringValue and CollectionValue slots
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11599 ) Change subject: IMPALA-7367: Pack StringValue and CollectionValue slots .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1393/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32f3b06622c087e4aa288e8db1bf4581b10d386a Gerrit-Change-Number: 11599 Gerrit-PatchSet: 7 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 19 Nov 2018 17:45:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7801: Remove toSql() from ParseNode interface.
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/11942 ) Change subject: IMPALA-7801: Remove toSql() from ParseNode interface. .. Patch Set 4: I'm not happy with using ORIGINAL as the default argument. This doesn't really solve the problems that Paul has identified with the semantics of this parameter. Using ORIGINAL really just makes things worse in this regard. So I am thinking I will abandon this change unless someone wants the change to use an explicit DEFAULT to describe the give-me-whatever-you-have case. -- To view, visit http://gerrit.cloudera.org:8080/11942 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I17025901838e9ffd753894a8087170123f9d8b33 Gerrit-Change-Number: 11942 Gerrit-PatchSet: 4 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Mon, 19 Nov 2018 17:38:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7801: Remove toSql() from ParseNode interface.
Andrew Sherman has removed Vuk Ercegovac from this change. ( http://gerrit.cloudera.org:8080/11942 ) Change subject: IMPALA-7801: Remove toSql() from ParseNode interface. .. Removed reviewer Vuk Ercegovac. -- To view, visit http://gerrit.cloudera.org:8080/11942 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: deleteReviewer Gerrit-Change-Id: I17025901838e9ffd753894a8087170123f9d8b33 Gerrit-Change-Number: 11942 Gerrit-PatchSet: 4 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Thomas Marshall
[Impala-ASF-CR] IMPALA-7367: Pack StringValue and CollectionValue slots
Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11599 ) Change subject: IMPALA-7367: Pack StringValue and CollectionValue slots .. IMPALA-7367: Pack StringValue and CollectionValue slots This change packs StringValue and CollectionValue slots to ensure they now occupy 12 bytes instead of 16 bytes. This reduces the memory requirements and improves the performance. Since Kudu tuples are populated using a memcopy, 4 bytes of padding was added to StringSlots in Kudu tables. Testing: Ran core tests. Added static asserts to ensure the value sizes are as expected. Performance tests on TPCH-40 produced 3.96% improvement. Change-Id: I32f3b06622c087e4aa288e8db1bf4581b10d386a Reviewed-on: http://gerrit.cloudera.org:8080/11599 Tested-by: Impala Public Jenkins Reviewed-by: Tim Armstrong --- M be/src/exec/text-converter.inline.h M be/src/exprs/expr-test.cc M be/src/exprs/scalar-expr.cc M be/src/runtime/collection-value.h M be/src/runtime/descriptors.cc M be/src/runtime/string-value.h M be/src/runtime/types.h M be/src/util/static-asserts.cc M fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java M fe/src/main/java/org/apache/impala/analysis/TupleDescriptor.java M fe/src/main/java/org/apache/impala/catalog/PrimitiveType.java M fe/src/main/java/org/apache/impala/catalog/Type.java M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java M testdata/workloads/functional-planner/queries/PlannerTest/constant-folding.test M testdata/workloads/functional-planner/queries/PlannerTest/fk-pk-join-detection.test M testdata/workloads/functional-planner/queries/PlannerTest/join-order.test M testdata/workloads/functional-planner/queries/PlannerTest/max-row-size.test M testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test M testdata/workloads/functional-planner/queries/PlannerTest/parquet-filtering-disabled.test M testdata/workloads/functional-planner/queries/PlannerTest/parquet-filtering.test M testdata/workloads/functional-planner/queries/PlannerTest/partition-pruning.test M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test M testdata/workloads/functional-planner/queries/PlannerTest/sort-expr-materialization.test M testdata/workloads/functional-planner/queries/PlannerTest/spillable-buffer-sizing.test M testdata/workloads/functional-planner/queries/PlannerTest/subquery-rewrite.test M testdata/workloads/functional-planner/queries/PlannerTest/tablesample.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds-all.test M testdata/workloads/functional-planner/queries/PlannerTest/tpch-all.test M testdata/workloads/functional-planner/queries/PlannerTest/tpch-kudu.test M testdata/workloads/functional-planner/queries/PlannerTest/tpch-nested.test M testdata/workloads/functional-query/queries/QueryTest/explain-level2.test M testdata/workloads/functional-query/queries/QueryTest/explain-level3.test M testdata/workloads/functional-query/queries/QueryTest/spilling-no-debug-action.test 33 files changed, 1,625 insertions(+), 1,634 deletions(-) Approvals: Impala Public Jenkins: Verified Tim Armstrong: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/11599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I32f3b06622c087e4aa288e8db1bf4581b10d386a Gerrit-Change-Number: 11599 Gerrit-PatchSet: 8 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7367: Pack StringValue and CollectionValue slots
Pooja Nilangekar has posted comments on this change. ( http://gerrit.cloudera.org:8080/11599 ) Change subject: IMPALA-7367: Pack StringValue and CollectionValue slots .. Patch Set 7: > Patch Set 7: > > I can submit once you publish the draft. Done. -- To view, visit http://gerrit.cloudera.org:8080/11599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32f3b06622c087e4aa288e8db1bf4581b10d386a Gerrit-Change-Number: 11599 Gerrit-PatchSet: 7 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 19 Nov 2018 17:12:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7367: Pack StringValue and CollectionValue slots
Pooja Nilangekar has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/11599 ) Change subject: IMPALA-7367: Pack StringValue and CollectionValue slots .. IMPALA-7367: Pack StringValue and CollectionValue slots This change packs StringValue and CollectionValue slots to ensure they now occupy 12 bytes instead of 16 bytes. This reduces the memory requirements and improves the performance. Since Kudu tuples are populated using a memcopy, 4 bytes of padding was added to StringSlots in Kudu tables. Testing: Ran core tests. Added static asserts to ensure the value sizes are as expected. Performance tests on TPCH-40 produced 3.96% improvement. Change-Id: I32f3b06622c087e4aa288e8db1bf4581b10d386a --- M be/src/exec/text-converter.inline.h M be/src/exprs/expr-test.cc M be/src/exprs/scalar-expr.cc M be/src/runtime/collection-value.h M be/src/runtime/descriptors.cc M be/src/runtime/string-value.h M be/src/runtime/types.h M be/src/util/static-asserts.cc M fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java M fe/src/main/java/org/apache/impala/analysis/TupleDescriptor.java M fe/src/main/java/org/apache/impala/catalog/PrimitiveType.java M fe/src/main/java/org/apache/impala/catalog/Type.java M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java M testdata/workloads/functional-planner/queries/PlannerTest/constant-folding.test M testdata/workloads/functional-planner/queries/PlannerTest/fk-pk-join-detection.test M testdata/workloads/functional-planner/queries/PlannerTest/join-order.test M testdata/workloads/functional-planner/queries/PlannerTest/max-row-size.test M testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test M testdata/workloads/functional-planner/queries/PlannerTest/parquet-filtering-disabled.test M testdata/workloads/functional-planner/queries/PlannerTest/parquet-filtering.test M testdata/workloads/functional-planner/queries/PlannerTest/partition-pruning.test M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test M testdata/workloads/functional-planner/queries/PlannerTest/sort-expr-materialization.test M testdata/workloads/functional-planner/queries/PlannerTest/spillable-buffer-sizing.test M testdata/workloads/functional-planner/queries/PlannerTest/subquery-rewrite.test M testdata/workloads/functional-planner/queries/PlannerTest/tablesample.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds-all.test M testdata/workloads/functional-planner/queries/PlannerTest/tpch-all.test M testdata/workloads/functional-planner/queries/PlannerTest/tpch-kudu.test M testdata/workloads/functional-planner/queries/PlannerTest/tpch-nested.test M testdata/workloads/functional-query/queries/QueryTest/explain-level2.test M testdata/workloads/functional-query/queries/QueryTest/explain-level3.test M testdata/workloads/functional-query/queries/QueryTest/spilling-no-debug-action.test 33 files changed, 1,625 insertions(+), 1,634 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/99/11599/7 -- To view, visit http://gerrit.cloudera.org:8080/11599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I32f3b06622c087e4aa288e8db1bf4581b10d386a Gerrit-Change-Number: 11599 Gerrit-PatchSet: 7 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7233: [DOCS] Support for IANA timezone database
Attila Jeges has posted comments on this change. (
http://gerrit.cloudera.org:8080/11946 )
Change subject: IMPALA-7233: [DOCS] Support for IANA timezone database
..
Patch Set 1:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/11946/1/docs/topics/impala_timestamp.xml
File docs/topics/impala_timestamp.xml:
http://gerrit.cloudera.org:8080/#/c/11946/1/docs/topics/impala_timestamp.xml@193
PS1, Line 193: Configuring custom time zones:
> I think that it would be better to move custom timezone databases/aliases t
I agree.
http://gerrit.cloudera.org:8080/#/c/11946/1/docs/topics/impala_timestamp.xml@272
PS1, Line 272: 2017c
Instead of '2017c' probably we should use a more generic directory name here
and below, e.g. 'latest'.
('2017c' is the version of tzdb that I tested this script with, but the script
should work with other tzdb versions as well).
http://gerrit.cloudera.org:8080/#/c/11946/1/docs/topics/impala_timestamp.xml@367
PS1, Line 367: turned off by
: default to avoid performance overhead
> I think that this was true before 3.1, but in 3.1 the conversion became muc
Turning on '-convert_legacy_hive_parquet_utc_timestamps' still has a cost, so I
think we should keep this sentence.
--
To view, visit http://gerrit.cloudera.org:8080/11946
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id400cda5a1be321063d17e0ee6337e92a5da732a
Gerrit-Change-Number: 11946
Gerrit-PatchSet: 1
Gerrit-Owner: Alex Rodoni
Gerrit-Reviewer: Attila Jeges
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Mon, 19 Nov 2018 15:47:46 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7233: [DOCS] Support for IANA timezone database
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/11946 ) Change subject: IMPALA-7233: [DOCS] Support for IANA timezone database .. Patch Set 1: (4 comments) http://gerrit.cloudera.org:8080/#/c/11946/1/docs/topics/impala_timestamp.xml File docs/topics/impala_timestamp.xml: http://gerrit.cloudera.org:8080/#/c/11946/1/docs/topics/impala_timestamp.xml@180 PS1, Line 180: + INTERVAL n-hours can be affected Daylight Saving Time, which nit: missing 'by'? http://gerrit.cloudera.org:8080/#/c/11946/1/docs/topics/impala_timestamp.xml@193 PS1, Line 193: Configuring custom time zones: I think that it would be better to move custom timezone databases/aliases to a separate topic/file if possible, as only probably only a small subset of users will be interested in these. http://gerrit.cloudera.org:8080/#/c/11946/1/docs/topics/impala_timestamp.xml@333 PS1, Line 333: values the same way it stores : without any adjustment. This part of the sentence is not clear for me. http://gerrit.cloudera.org:8080/#/c/11946/1/docs/topics/impala_timestamp.xml@367 PS1, Line 367: turned off by : default to avoid performance overhead I think that this was true before 3.1, but in 3.1 the conversion became much faster, so the main reason is compatibility: the result of an existing non-buggy query should not change in a minor version. I think that IMPALA-3307 could be mentioned as the reason for speedup. convert_legacy_hive_parquet_utc_timestamps is known by users to be extremely slow, so the change could be highlighted to make it clear that this a much small issue now. -- To view, visit http://gerrit.cloudera.org:8080/11946 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id400cda5a1be321063d17e0ee6337e92a5da732a Gerrit-Change-Number: 11946 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 19 Nov 2018 14:36:37 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5031: signed overflow in TimestampValue
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/11919 ) Change subject: IMPALA-5031: signed overflow in TimestampValue .. Patch Set 1: (1 comment) I have two concerns with the solution: 1. performance - convert-timestamp-benchmark.cc has a test for this function, can you run it in release mode? I would call this function medium performance critical - it is called in decimal->timestamp conversion, so it is not used in most queries, but has the potential to dominate some specific queries. 2. readability - the change adds some complexity that I would prefer to avoid if possible, see my comment in line 66 for ideas. http://gerrit.cloudera.org:8080/#/c/11919/1/be/src/runtime/timestamp-value.inline.h File be/src/runtime/timestamp-value.inline.h: http://gerrit.cloudera.org:8080/#/c/11919/1/be/src/runtime/timestamp-value.inline.h@66 PS1, Line 66: int64_t nanos Some explanation for the reason why I did not care about overflow when I last touched this code: nanos makes sense in the +-10^9 range, and all (non-test) callers pass int32/uint32, so the range we can get here is not much larger than that (~ -2 to 4 sec). This means that overflow of unix_time is only possible near the min/max value representable on 64 bits, which is far outside the valid range of timestamps, so the overflow can only change an invalid timestamp to another invalid timestamp. I would prefer to avoid the overflow by changing the interface to handle nanos only in the -999'999'999 .. + 999'999'999 range, and treat other values as invalid timestamps. This would mean that unix_time would be affected only in the negative case. The only caller code that would need to be changed are tests and timestamp-test.cc. Another way to do this is to change the valid range to 0 .. 999'999'999, which would make this function even simpler, but would need some changes in DecimalOperators::ConvertToTimestampVal. -- To view, visit http://gerrit.cloudera.org:8080/11919 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaad158e6634314a5690a43a0cc04426c1aba8f41 Gerrit-Change-Number: 11919 Gerrit-PatchSet: 1 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 19 Nov 2018 13:34:26 + Gerrit-HasComments: Yes
