[Impala-ASF-CR] Revert "IMPALA-6389: Make '\0' delimited text files work"

[Zach Amsden (Code Review)]

Zach Amsden has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9851 )

Change subject: Revert "IMPALA-6389: Make '\0' delimited text files work"
..


Patch Set 2:

Strict revert; verified to fix build.


--
To view, visit http://gerrit.cloudera.org:8080/9851
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If581311033de8c26e33316b19192c4579594f261
Gerrit-Change-Number: 9851
Gerrit-PatchSet: 2
Gerrit-Owner: Zach Amsden 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Ho
Gerrit-Reviewer: Zach Amsden 
Gerrit-Comment-Date: Thu, 29 Mar 2018 05:00:27 +
Gerrit-HasComments: No

[Impala-ASF-CR] Revert "IMPALA-6389: Make '\0' delimited text files work"

[Zach Amsden (Code Review)]

Zach Amsden has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/9851 )

Change subject: Revert "IMPALA-6389: Make '\0' delimited text files work"
..

Revert "IMPALA-6389: Make '\0' delimited text files work"

This reverts commit c2bdaf8af4cf35d3462595c2a341ed84dcf5d960.

An ASAN issue and potentially other problem have been found;
reverting to unbreak the build and tests.

Change-Id: If581311033de8c26e33316b19192c4579594f261
Reviewed-on: http://gerrit.cloudera.org:8080/9851
Reviewed-by: Lars Volker 
Tested-by: Zach Amsden 
---
M be/src/exec/delimited-text-parser-test.cc
M be/src/exec/delimited-text-parser.cc
M be/src/exec/delimited-text-parser.h
M be/src/exec/delimited-text-parser.inline.h
M be/src/exec/hdfs-sequence-scanner.cc
M be/src/exec/hdfs-sequence-scanner.h
M be/src/exec/hdfs-text-scanner.cc
M be/src/exec/hdfs-text-scanner.h
8 files changed, 84 insertions(+), 169 deletions(-)

Approvals:
  Lars Volker: Looks good to me, approved
  Zach Amsden: Verified

--
To view, visit http://gerrit.cloudera.org:8080/9851
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: If581311033de8c26e33316b19192c4579594f261
Gerrit-Change-Number: 9851
Gerrit-PatchSet: 2
Gerrit-Owner: Zach Amsden 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Ho
Gerrit-Reviewer: Zach Amsden 

[Impala-ASF-CR] Revert "IMPALA-6389: Make '\0' delimited text files work"

[Zach Amsden (Code Review)]

Zach Amsden has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9851 )

Change subject: Revert "IMPALA-6389: Make '\0' delimited text files work"
..


Patch Set 1: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/9851
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If581311033de8c26e33316b19192c4579594f261
Gerrit-Change-Number: 9851
Gerrit-PatchSet: 1
Gerrit-Owner: Zach Amsden 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Ho
Gerrit-Reviewer: Zach Amsden 
Gerrit-Comment-Date: Thu, 29 Mar 2018 04:59:43 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6731: Move execnet Python dependency to stage 2

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/9850 )

Change subject: IMPALA-6731: Move execnet Python dependency to stage 2
..

IMPALA-6731: Move execnet Python dependency to stage 2

It seems that execnet also cannot be installed together with
setuptools-scm if only a local mirror and index are available
(similar to https://github.com/pywebhdfs/pywebhdfs/issues/52).

Testing: Observed that execnet failed to install during
bootstrap_toolchain.py on a CentOS 6.4 EC2 instanc at 5:02pm (within the
brownout period). With this change, bootstrap_toolchain.py succeeded.

Change-Id: Ic949edcc03f0e068bdd84b6ede487e64dcf2439b
Reviewed-on: http://gerrit.cloudera.org:8080/9850
Reviewed-by: David Knupp 
Tested-by: Impala Public Jenkins
---
M infra/python/deps/requirements.txt
M infra/python/deps/stage2-requirements.txt
2 files changed, 3 insertions(+), 3 deletions(-)

Approvals:
  David Knupp: Looks good to me, approved
  Impala Public Jenkins: Verified

--
To view, visit http://gerrit.cloudera.org:8080/9850
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ic949edcc03f0e068bdd84b6ede487e64dcf2439b
Gerrit-Change-Number: 9850
Gerrit-PatchSet: 2
Gerrit-Owner: Lars Volker 
Gerrit-Reviewer: David Knupp 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Philip Zeyliger 

[Impala-ASF-CR] IMPALA-6731: Move execnet Python dependency to stage 2

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9850 )

Change subject: IMPALA-6731: Move execnet Python dependency to stage 2
..


Patch Set 1: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/9850
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic949edcc03f0e068bdd84b6ede487e64dcf2439b
Gerrit-Change-Number: 9850
Gerrit-PatchSet: 1
Gerrit-Owner: Lars Volker 
Gerrit-Reviewer: David Knupp 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Comment-Date: Thu, 29 Mar 2018 04:38:30 +
Gerrit-HasComments: No

[Impala-ASF-CR] Revert "IMPALA-6389: Make '\0' delimited text files work"

[Lars Volker (Code Review)]

Lars Volker has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9851 )

Change subject: Revert "IMPALA-6389: Make '\0' delimited text files work"
..


Patch Set 1: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/9851
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If581311033de8c26e33316b19192c4579594f261
Gerrit-Change-Number: 9851
Gerrit-PatchSet: 1
Gerrit-Owner: Zach Amsden 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Ho
Gerrit-Comment-Date: Thu, 29 Mar 2018 04:02:36 +
Gerrit-HasComments: No

[Impala-ASF-CR] Revert "IMPALA-6389: Make '\0' delimited text files work"

[Zach Amsden (Code Review)]

Zach Amsden has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/9851


Change subject: Revert "IMPALA-6389: Make '\0' delimited text files work"
..

Revert "IMPALA-6389: Make '\0' delimited text files work"

This reverts commit c2bdaf8af4cf35d3462595c2a341ed84dcf5d960.

An ASAN issue and potentially other problem have been found;
reverting to unbreak the build and tests.

Change-Id: If581311033de8c26e33316b19192c4579594f261
---
M be/src/exec/delimited-text-parser-test.cc
M be/src/exec/delimited-text-parser.cc
M be/src/exec/delimited-text-parser.h
M be/src/exec/delimited-text-parser.inline.h
M be/src/exec/hdfs-sequence-scanner.cc
M be/src/exec/hdfs-sequence-scanner.h
M be/src/exec/hdfs-text-scanner.cc
M be/src/exec/hdfs-text-scanner.h
8 files changed, 84 insertions(+), 169 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/51/9851/1
--
To view, visit http://gerrit.cloudera.org:8080/9851
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: If581311033de8c26e33316b19192c4579594f261
Gerrit-Change-Number: 9851
Gerrit-PatchSet: 1
Gerrit-Owner: Zach Amsden 

[Impala-ASF-CR] IMPALA-5721,IMPALA-6717,IMPALA-6738: improve stress test binary search

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/9770 )

Change subject: IMPALA-5721,IMPALA-6717,IMPALA-6738: improve stress test binary 
search
..

IMPALA-5721,IMPALA-6717,IMPALA-6738: improve stress test binary search

IMPALA-5721:
- Save profiles of queries at the end of both the spilling and
  non-spilling binary search. These were not being saved before. Note
  these profiles won't have ExecSummary until IMPALA-6640 is addressed.

- Save the profile of any query that produces incorrect results during
  binary search. These were not being saved before, either.

- Use descriptive names, like
  tpch_100_parquet_q12_profile_without_spilling.txt, for profiles
  mentioned above. We do this by introducing the concept of a
  "logical_query_id" whose values look like "tpch_100_parquet_q12".

- Use the logical_query_id in critical error paths and include the
  logical_query_id in result hash files.

IMPALA-6717:
- Plumb --common-query-options through to the binary search.

IMPALA-6738:
- Begin a refactoring to reduce the number of parameters used when doing
  the binary search.

- Introduce a notion of "converted args" via class that does the
  conversion (if needed) via property getters.

- Adjust populate_all_queries() to use converted_args

Change-Id: I33d036ec93df3016cd4703205078dbdba0168acb
Reviewed-on: http://gerrit.cloudera.org:8080/9770
Reviewed-by: David Knupp 
Tested-by: Impala Public Jenkins
---
M tests/stress/concurrent_select.py
1 file changed, 213 insertions(+), 76 deletions(-)

Approvals:
  David Knupp: Looks good to me, approved
  Impala Public Jenkins: Verified

--
To view, visit http://gerrit.cloudera.org:8080/9770
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I33d036ec93df3016cd4703205078dbdba0168acb
Gerrit-Change-Number: 9770
Gerrit-PatchSet: 10
Gerrit-Owner: Michael Brown 
Gerrit-Reviewer: David Knupp 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Brown 
Gerrit-Reviewer: Nithya Janarthanan 
Gerrit-Reviewer: Tim Armstrong 

[Impala-ASF-CR] IMPALA-5721,IMPALA-6717,IMPALA-6738: improve stress test binary search

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9770 )

Change subject: IMPALA-5721,IMPALA-6717,IMPALA-6738: improve stress test binary 
search
..


Patch Set 9: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/9770
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I33d036ec93df3016cd4703205078dbdba0168acb
Gerrit-Change-Number: 9770
Gerrit-PatchSet: 9
Gerrit-Owner: Michael Brown 
Gerrit-Reviewer: David Knupp 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Brown 
Gerrit-Reviewer: Nithya Janarthanan 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Thu, 29 Mar 2018 03:38:34 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6759: align stress test memory estimation parse pattern

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/9846 )

Change subject: IMPALA-6759: align stress test memory estimation parse pattern
..

IMPALA-6759: align stress test memory estimation parse pattern

The stress test never expected to see memory estimates on the order of
PB. Apparently it can happen with TPC DS 1, so update the pattern.

It's not clear how to quickly write a test to catch this, because it
involves crossing language boundaries and possibly having a
massively-scaled dataset. I think leaving a comment in both places is
good enough for now.

Change-Id: I317c271888584ed2a817ee52ad70267eae64d341
Reviewed-on: http://gerrit.cloudera.org:8080/9846
Reviewed-by: Lars Volker 
Tested-by: Impala Public Jenkins
---
M fe/src/main/java/org/apache/impala/common/PrintUtils.java
M tests/stress/concurrent_select.py
M tests/util/parse_util.py
3 files changed, 13 insertions(+), 7 deletions(-)

Approvals:
  Lars Volker: Looks good to me, approved
  Impala Public Jenkins: Verified

--
To view, visit http://gerrit.cloudera.org:8080/9846
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I317c271888584ed2a817ee52ad70267eae64d341
Gerrit-Change-Number: 9846
Gerrit-PatchSet: 2
Gerrit-Owner: Michael Brown 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Brown 

[Impala-ASF-CR] IMPALA-6759: align stress test memory estimation parse pattern

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9846 )

Change subject: IMPALA-6759: align stress test memory estimation parse pattern
..


Patch Set 1: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/9846
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I317c271888584ed2a817ee52ad70267eae64d341
Gerrit-Change-Number: 9846
Gerrit-PatchSet: 1
Gerrit-Owner: Michael Brown 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Brown 
Gerrit-Comment-Date: Thu, 29 Mar 2018 03:27:24 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6760: Fix for py2.7-ism in run-tests.py.

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9843 )

Change subject: IMPALA-6760: Fix for py2.7-ism in run-tests.py.
..


Patch Set 1:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2206/


--
To view, visit http://gerrit.cloudera.org:8080/9843
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I81928d1880a493b91abb13b3a8149568c9789f66
Gerrit-Change-Number: 9843
Gerrit-PatchSet: 1
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: David Knupp 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Brown 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Nithya Janarthanan 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Thu, 29 Mar 2018 03:05:46 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6760: Fix for py2.7-ism in run-tests.py.

[Philip Zeyliger (Code Review)]

Philip Zeyliger has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9843 )

Change subject: IMPALA-6760: Fix for py2.7-ism in run-tests.py.
..


Patch Set 1:

GVD failure was as follows, which seems unrelated. The issue matches 
https://issues.apache.org/jira/browse/IMPALA-6092 which was previously flaky 
and I re-opened.

I'm going to try again.

02:18:16 ] === FAILURES 
===
02:18:16 ]  TestUdfExecution.test_java_udfs[exec_option: 
{'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 
'exec_single_node_rows_threshold': 100, 'enable_expr_rewrites': True} | 
table_format: text/none]
02:18:16 ] query_test/test_udfs.py:401: in test_java_udfs
02:18:16 ] self.run_test_case('QueryTest/java-udf', vector, 
use_db=unique_database)
02:18:16 ] common/impala_test_suite.py:398: in run_test_case
02:18:16 ] result = self.__execute_query(target_impalad_client, query, 
user=user)
02:18:16 ] common/impala_test_suite.py:613: in __execute_query
02:18:16 ] return impalad_client.execute(query, user=user)
02:18:16 ] common/impala_connection.py:160: in execute
02:18:16 ] return self.__beeswax_client.execute(sql_stmt, user=user)
02:18:16 ] beeswax/impala_beeswax.py:173: in execute
02:18:16 ] handle = self.__execute_query(query_string.strip(), user=user)
02:18:16 ] beeswax/impala_beeswax.py:341: in __execute_query
02:18:16 ] self.wait_for_completion(handle)
02:18:16 ] beeswax/impala_beeswax.py:361: in wait_for_completion
02:18:16 ] raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
02:18:16 ] E   ImpalaBeeswaxException: ImpalaBeeswaxException:
02:18:16 ] EQuery aborted:ImpalaRuntimeException: Unable to find class.
02:18:16 ] E   CAUSED BY: ClassNotFoundException: org.apache.impala.TestUdf
02:18:16 ] === 2012 tests deselected by "-m 'execute_serially'" 
===
02:18:16 ]  1 failed, 202 passed, 9 skipped, 2012 deselected, 5 xfailed in 
2198.79 seconds
02:18:16 ] MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)


--
To view, visit http://gerrit.cloudera.org:8080/9843
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I81928d1880a493b91abb13b3a8149568c9789f66
Gerrit-Change-Number: 9843
Gerrit-PatchSet: 1
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: David Knupp 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Brown 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Nithya Janarthanan 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Thu, 29 Mar 2018 03:05:01 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5384, part 1: introduce DmlExecState

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9793 )

Change subject: IMPALA-5384, part 1: introduce DmlExecState
..


Patch Set 8:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2205/


--
To view, visit http://gerrit.cloudera.org:8080/9793
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id4c025917620a7bff2acbeb46464f107ab4b7565
Gerrit-Change-Number: 9793
Gerrit-PatchSet: 8
Gerrit-Owner: Dan Hecht 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Thu, 29 Mar 2018 02:35:46 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5384, part 1: introduce DmlExecState

[Dan Hecht (Code Review)]

Dan Hecht has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9793 )

Change subject: IMPALA-5384, part 1: introduce DmlExecState
..


Patch Set 8: Code-Review+2

carry


--
To view, visit http://gerrit.cloudera.org:8080/9793
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id4c025917620a7bff2acbeb46464f107ab4b7565
Gerrit-Change-Number: 9793
Gerrit-PatchSet: 8
Gerrit-Owner: Dan Hecht 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Thu, 29 Mar 2018 02:35:01 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6760: Fix for py2.7-ism in run-tests.py.

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9843 )

Change subject: IMPALA-6760: Fix for py2.7-ism in run-tests.py.
..


Patch Set 1: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/2201/


--
To view, visit http://gerrit.cloudera.org:8080/9843
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I81928d1880a493b91abb13b3a8149568c9789f66
Gerrit-Change-Number: 9843
Gerrit-PatchSet: 1
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: David Knupp 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Brown 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Nithya Janarthanan 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Thu, 29 Mar 2018 02:18:17 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5931: Generates scan ranges in planner for s3/adls

[Vuk Ercegovac (Code Review)]

Vuk Ercegovac has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/8523 )

Change subject: IMPALA-5931: Generates scan ranges in planner for s3/adls
..


Patch Set 11:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/8523/11//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/8523/11//COMMIT_MSG@34
PS11, Line 34: - all core tests pass when configured with s3
> Do the tests validate that an "even" number of bytes is assigned to each ba
That block size is used in the specs here, HdfsScanNode.java L755. The division 
into scan ranges should be the same as before. As before, each of these scan 
ranges is "random", e.g., no process affinity, so as long as assignment of 
remote reads balances bytes per backend, then the result should be balanced.
There's a scheduler unit test that tests remote reads and checks that the 
assignment is balanced.
I did some manual queries and saw roughly even assignment.
That said, I don't have a test that looks at bytes-per-host and checks that 
they're even. Will look at what other tests we have for this and what can be 
added.



--
To view, visit http://gerrit.cloudera.org:8080/8523
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I326065adbb2f7e632814113aae85cb51ca4779a5
Gerrit-Change-Number: 8523
Gerrit-PatchSet: 11
Gerrit-Owner: Vuk Ercegovac 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Mostafa Mokhtar 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Vuk Ercegovac 
Gerrit-Comment-Date: Thu, 29 Mar 2018 01:49:33 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] KUDU-2385: Fix typo in KinitContext::DoRenewal()

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9842 )

Change subject: KUDU-2385: Fix typo in KinitContext::DoRenewal()
..


Patch Set 1: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/9842
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3a22b8d41d15eb1982a3fd5b96575e28edaad31c
Gerrit-Change-Number: 9842
Gerrit-PatchSet: 1
Gerrit-Owner: Sailesh Mukil 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Ho 
Gerrit-Comment-Date: Thu, 29 Mar 2018 01:40:04 +
Gerrit-HasComments: No

[Impala-ASF-CR] KUDU-2385: Fix typo in KinitContext::DoRenewal()

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/9842 )

Change subject: KUDU-2385: Fix typo in KinitContext::DoRenewal()
..

KUDU-2385: Fix typo in KinitContext::DoRenewal()

On platforms without krb5_get_init_creds_opt_set_out_ccache(),
krb5_cc_store_cred() is called to insert the newly acquired
credential into the ccache. However, there was a typo in the code
which resulted in inserting the old credential into ccache.
This change fixes the typo to make sure the new credential is
inserted into ccache.

Testing done: confirmed on SLES11 that the new credential
is being inserted by checking the 'auth time' of the ticket
in ccache. Impala uses a slightly different #ifdef which
explicitly checks if krb5_get_init_creds_opt_set_out_ccache()
is defined on the platform so this code path is actually
used when running Impala on SLES11.

Change-Id: I3a22b8d41d15eb1982a3fd5b96575e28edaad31c
Reviewed-on: http://gerrit.cloudera.org:8080/9840
Reviewed-by: Todd Lipcon 
Tested-by: Todd Lipcon 
Reviewed-on: http://gerrit.cloudera.org:8080/9842
Reviewed-by: Michael Ho 
Tested-by: Impala Public Jenkins
---
M be/src/kudu/security/init.cc
1 file changed, 1 insertion(+), 1 deletion(-)

Approvals:
  Michael Ho: Looks good to me, approved
  Impala Public Jenkins: Verified

--
To view, visit http://gerrit.cloudera.org:8080/9842
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I3a22b8d41d15eb1982a3fd5b96575e28edaad31c
Gerrit-Change-Number: 9842
Gerrit-PatchSet: 2
Gerrit-Owner: Sailesh Mukil 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Ho 

[Impala-ASF-CR] IMPALA-6731: Move execnet Python dependency to stage 2

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9850 )

Change subject: IMPALA-6731: Move execnet Python dependency to stage 2
..


Patch Set 1:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2204/


--
To view, visit http://gerrit.cloudera.org:8080/9850
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic949edcc03f0e068bdd84b6ede487e64dcf2439b
Gerrit-Change-Number: 9850
Gerrit-PatchSet: 1
Gerrit-Owner: Lars Volker 
Gerrit-Reviewer: David Knupp 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Comment-Date: Thu, 29 Mar 2018 00:46:18 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6731: Move execnet Python dependency to stage 2

[David Knupp (Code Review)]

David Knupp has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9850 )

Change subject: IMPALA-6731: Move execnet Python dependency to stage 2
..


Patch Set 1: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/9850
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic949edcc03f0e068bdd84b6ede487e64dcf2439b
Gerrit-Change-Number: 9850
Gerrit-PatchSet: 1
Gerrit-Owner: Lars Volker 
Gerrit-Reviewer: David Knupp 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Comment-Date: Thu, 29 Mar 2018 00:43:06 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6731: Move execnet Python dependency to stage 2

[Lars Volker (Code Review)]

Lars Volker has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/9850


Change subject: IMPALA-6731: Move execnet Python dependency to stage 2
..

IMPALA-6731: Move execnet Python dependency to stage 2

It seems that execnet also cannot be installed together with
setuptools-scm if only a local mirror and index are available
(similar to https://github.com/pywebhdfs/pywebhdfs/issues/52).

Testing: Observed that execnet failed to install during
bootstrap_toolchain.py on a CentOS 6.4 EC2 instanc at 5:02pm (within the
brownout period). With this change, bootstrap_toolchain.py succeeded.

Change-Id: Ic949edcc03f0e068bdd84b6ede487e64dcf2439b
---
M infra/python/deps/requirements.txt
M infra/python/deps/stage2-requirements.txt
2 files changed, 3 insertions(+), 3 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/9850/1
--
To view, visit http://gerrit.cloudera.org:8080/9850
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ic949edcc03f0e068bdd84b6ede487e64dcf2439b
Gerrit-Change-Number: 9850
Gerrit-PatchSet: 1
Gerrit-Owner: Lars Volker 

[Impala-ASF-CR] IMPALA-6747: Automate diagnostics collection.

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/9815 )

Change subject: IMPALA-6747: Automate diagnostics collection.
..

IMPALA-6747: Automate diagnostics collection.

This commit adds the necessary tooling to automate diagnostics
collection for Impala daemons. Following diagnostics are supported.

1. Native core dump (+ shared libs)
2. GDB/Java thread dump (pstack + jstack)
3. Java heap dump (jmap)
4. Minidumps (using breakpad) *
5. Profiles

Given the required inputs, the script outputs a zip compressed
impala diagnostic bundle with all the diagnostics collected.

The script can be run manually with the following command.

python collect_diagnostics.py --help

* minidumps collected here correspond to the state of the Impala
process at the time this script is triggered. This is different
from collect_minidumps.py which archives the entire minidump
directory.

Change-Id: Ib29caec7c3be5b6a31e60461294979c318300f64
Reviewed-on: http://gerrit.cloudera.org:8080/9815
Reviewed-by: Lars Volker 
Reviewed-by: Philip Zeyliger 
Tested-by: Impala Public Jenkins
---
A bin/diagnostics/__init__.py
A bin/diagnostics/collect_diagnostics.py
A bin/diagnostics/collect_shared_libs.sh
M bin/rat_exclude_files.txt
A tests/unittests/test_command.py
5 files changed, 620 insertions(+), 0 deletions(-)

Approvals:
  Lars Volker: Looks good to me, approved
  Philip Zeyliger: Looks good to me, approved
  Impala Public Jenkins: Verified

--
To view, visit http://gerrit.cloudera.org:8080/9815
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ib29caec7c3be5b6a31e60461294979c318300f64
Gerrit-Change-Number: 9815
Gerrit-PatchSet: 4
Gerrit-Owner: Bharath Vissapragada 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Philip Zeyliger 

[Impala-ASF-CR] IMPALA-6747: Automate diagnostics collection.

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9815 )

Change subject: IMPALA-6747: Automate diagnostics collection.
..


Patch Set 3: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/9815
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib29caec7c3be5b6a31e60461294979c318300f64
Gerrit-Change-Number: 9815
Gerrit-PatchSet: 3
Gerrit-Owner: Bharath Vissapragada 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Comment-Date: Thu, 29 Mar 2018 00:12:17 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5721,IMPALA-6717,IMPALA-6738: improve stress test binary search

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9770 )

Change subject: IMPALA-5721,IMPALA-6717,IMPALA-6738: improve stress test binary 
search
..


Patch Set 9:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2203/


--
To view, visit http://gerrit.cloudera.org:8080/9770
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I33d036ec93df3016cd4703205078dbdba0168acb
Gerrit-Change-Number: 9770
Gerrit-PatchSet: 9
Gerrit-Owner: Michael Brown 
Gerrit-Reviewer: David Knupp 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Brown 
Gerrit-Reviewer: Nithya Janarthanan 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Wed, 28 Mar 2018 23:41:07 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6759: align stress test memory estimation parse pattern

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9846 )

Change subject: IMPALA-6759: align stress test memory estimation parse pattern
..


Patch Set 1:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2202/


--
To view, visit http://gerrit.cloudera.org:8080/9846
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I317c271888584ed2a817ee52ad70267eae64d341
Gerrit-Change-Number: 9846
Gerrit-PatchSet: 1
Gerrit-Owner: Michael Brown 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Brown 
Gerrit-Comment-Date: Wed, 28 Mar 2018 23:40:31 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6759: align stress test memory estimation parse pattern

[Lars Volker (Code Review)]

Lars Volker has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9846 )

Change subject: IMPALA-6759: align stress test memory estimation parse pattern
..


Patch Set 1: Code-Review+2

(1 comment)

Looks good to me. The change also had a +2 from David in a prior life and 
nothing substantial has changed.

http://gerrit.cloudera.org:8080/#/c/9846/1/tests/util/parse_util.py
File tests/util/parse_util.py:

http://gerrit.cloudera.org:8080/#/c/9846/1/tests/util/parse_util.py@68
PS1, Line 68:   elif units == "P":
> It was this additional bit I was trying to push when I mis-typed and pushed
Cool, thx.



--
To view, visit http://gerrit.cloudera.org:8080/9846
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I317c271888584ed2a817ee52ad70267eae64d341
Gerrit-Change-Number: 9846
Gerrit-PatchSet: 1
Gerrit-Owner: Michael Brown 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Brown 
Gerrit-Comment-Date: Wed, 28 Mar 2018 23:36:05 +
Gerrit-HasComments: Yes

[Impala-ASF-CR](2.x) Ignore "IMPALA-6759: align stress test memory estimation parse pattern"

[Michael Brown (Code Review)]

Michael Brown has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/9845 )

Change subject: Ignore "IMPALA-6759: align stress test memory estimation parse 
pattern"
..

Ignore "IMPALA-6759: align stress test memory estimation parse pattern"

This was an accidental push to master.

Change-Id: I7c86a03d59abdf31753d9d7bc00710eb21f60e0f
Reviewed-on: http://gerrit.cloudera.org:8080/9845
Reviewed-by: Philip Zeyliger 
Tested-by: Michael Brown 
---
M bin/ignored_commits.json
1 file changed, 3 insertions(+), 1 deletion(-)

Approvals:
  Philip Zeyliger: Looks good to me, approved
  Michael Brown: Verified

--
To view, visit http://gerrit.cloudera.org:8080/9845
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: 2.x
Gerrit-MessageType: merged
Gerrit-Change-Id: I7c86a03d59abdf31753d9d7bc00710eb21f60e0f
Gerrit-Change-Number: 9845
Gerrit-PatchSet: 2
Gerrit-Owner: Michael Brown 
Gerrit-Reviewer: Michael Brown 
Gerrit-Reviewer: Philip Zeyliger 

[Impala-ASF-CR](2.x) Ignore "IMPALA-6759: align stress test memory estimation parse pattern"

[Michael Brown (Code Review)]

Michael Brown has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9845 )

Change subject: Ignore "IMPALA-6759: align stress test memory estimation parse 
pattern"
..


Patch Set 1: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/9845
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: 2.x
Gerrit-MessageType: comment
Gerrit-Change-Id: I7c86a03d59abdf31753d9d7bc00710eb21f60e0f
Gerrit-Change-Number: 9845
Gerrit-PatchSet: 1
Gerrit-Owner: Michael Brown 
Gerrit-Reviewer: Michael Brown 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Comment-Date: Wed, 28 Mar 2018 23:13:56 +
Gerrit-HasComments: No

[Impala-ASF-CR](2.x) Ignore "IMPALA-6759: align stress test memory estimation parse pattern"

[Philip Zeyliger (Code Review)]

Philip Zeyliger has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9845 )

Change subject: Ignore "IMPALA-6759: align stress test memory estimation parse 
pattern"
..


Patch Set 1: Code-Review+2

Feel free to self-verify. I use "python -mjson.tool < ignored_commits.json" to 
make sure the JSON is well-formed.


--
To view, visit http://gerrit.cloudera.org:8080/9845
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: 2.x
Gerrit-MessageType: comment
Gerrit-Change-Id: I7c86a03d59abdf31753d9d7bc00710eb21f60e0f
Gerrit-Change-Number: 9845
Gerrit-PatchSet: 1
Gerrit-Owner: Michael Brown 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Comment-Date: Wed, 28 Mar 2018 23:10:26 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5721,IMPALA-6717,IMPALA-6738: improve stress test binary search

[David Knupp (Code Review)]

David Knupp has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9770 )

Change subject: IMPALA-5721,IMPALA-6717,IMPALA-6738: improve stress test binary 
search
..


Patch Set 9: Code-Review+2

> Patch Set 7:
>
> (1 comment)


--
To view, visit http://gerrit.cloudera.org:8080/9770
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I33d036ec93df3016cd4703205078dbdba0168acb
Gerrit-Change-Number: 9770
Gerrit-PatchSet: 9
Gerrit-Owner: Michael Brown 
Gerrit-Reviewer: David Knupp 
Gerrit-Reviewer: Michael Brown 
Gerrit-Reviewer: Nithya Janarthanan 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Wed, 28 Mar 2018 23:02:44 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6759: align stress test memory estimation parse pattern

[Michael Brown (Code Review)]

Michael Brown has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9846 )

Change subject: IMPALA-6759: align stress test memory estimation parse pattern
..


Patch Set 1:

(1 comment)

History: https://gerrit.cloudera.org/#/c/9844/

http://gerrit.cloudera.org:8080/#/c/9846/1/tests/util/parse_util.py
File tests/util/parse_util.py:

http://gerrit.cloudera.org:8080/#/c/9846/1/tests/util/parse_util.py@68
PS1, Line 68:   elif units == "P":
It was this additional bit I was trying to push when I mis-typed and pushed to 
heads. Lars, I've gone ahead and fixed the base-2 issue. Nothing uses this 
method except the stress test.



--
To view, visit http://gerrit.cloudera.org:8080/9846
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I317c271888584ed2a817ee52ad70267eae64d341
Gerrit-Change-Number: 9846
Gerrit-PatchSet: 1
Gerrit-Owner: Michael Brown 
Gerrit-Reviewer: Michael Brown 
Gerrit-Comment-Date: Wed, 28 Mar 2018 23:02:26 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-6759: align stress test memory estimation parse pattern

[Michael Brown (Code Review)]

Michael Brown has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/9846


Change subject: IMPALA-6759: align stress test memory estimation parse pattern
..

IMPALA-6759: align stress test memory estimation parse pattern

The stress test never expected to see memory estimates on the order of
PB. Apparently it can happen with TPC DS 1, so update the pattern.

It's not clear how to quickly write a test to catch this, because it
involves crossing language boundaries and possibly having a
massively-scaled dataset. I think leaving a comment in both places is
good enough for now.

Change-Id: I317c271888584ed2a817ee52ad70267eae64d341
---
M fe/src/main/java/org/apache/impala/common/PrintUtils.java
M tests/stress/concurrent_select.py
M tests/util/parse_util.py
3 files changed, 13 insertions(+), 7 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/46/9846/1
--
To view, visit http://gerrit.cloudera.org:8080/9846
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I317c271888584ed2a817ee52ad70267eae64d341
Gerrit-Change-Number: 9846
Gerrit-PatchSet: 1
Gerrit-Owner: Michael Brown 

[Impala-ASF-CR] IMPALA-6759: align stress test memory estimation parse pattern

[Michael Brown (Code Review)]

Michael Brown has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9844 )

Change subject: IMPALA-6759: align stress test memory estimation parse pattern
..


Patch Set 4:

See https://gerrit.cloudera.org/#/c/9846/


--
To view, visit http://gerrit.cloudera.org:8080/9844
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I08976f261582b379696fd0e81bc060577e552309
Gerrit-Change-Number: 9844
Gerrit-PatchSet: 4
Gerrit-Owner: Michael Brown 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Brown 
Gerrit-Reviewer: Michael Ho 
Gerrit-Comment-Date: Wed, 28 Mar 2018 23:01:15 +
Gerrit-HasComments: No

[Impala-ASF-CR](2.x) Ignore "IMPALA-6759: align stress test memory estimation parse pattern"

[Michael Brown (Code Review)]

Michael Brown has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/9845


Change subject: Ignore "IMPALA-6759: align stress test memory estimation parse 
pattern"
..

Ignore "IMPALA-6759: align stress test memory estimation parse pattern"

This was an accidental push to master.

Change-Id: I7c86a03d59abdf31753d9d7bc00710eb21f60e0f
---
M bin/ignored_commits.json
1 file changed, 3 insertions(+), 1 deletion(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/9845/1
--
To view, visit http://gerrit.cloudera.org:8080/9845
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: 2.x
Gerrit-MessageType: newchange
Gerrit-Change-Id: I7c86a03d59abdf31753d9d7bc00710eb21f60e0f
Gerrit-Change-Number: 9845
Gerrit-PatchSet: 1
Gerrit-Owner: Michael Brown 

[Impala-ASF-CR] IMPALA-5903: Inconsistent specification of result set and result set metadata

[Zoltan Borok-Nagy (Code Review)]

Hello Dimitris Tsirogiannis, Tim Armstrong,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/9090

to look at the new patch set (#5).

Change subject: IMPALA-5903: Inconsistent specification of result set and 
result set metadata
..

IMPALA-5903: Inconsistent specification of result set and result set metadata

Before this commit it was quite random which DDL oprations
returned a result set and which didn't.

With this commit, every DDL operations return a summary of
its execution. They declare their result set schema in
Frontend.java, and provide the summary in CalatogOpExecutor.java.

Updated the tests according to the new behavior.

Change-Id: Ic542fb8e49e850052416ac663ee329ee3974e3b9
---
M be/src/service/client-request-state.cc
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M shell/impala_client.py
M testdata/workloads/functional-query/queries/QueryTest/alter-table.test
M testdata/workloads/functional-query/queries/QueryTest/chars-tmp-tables.test
M 
testdata/workloads/functional-query/queries/QueryTest/compute-stats-incremental.test
M testdata/workloads/functional-query/queries/QueryTest/compute-stats.test
M testdata/workloads/functional-query/queries/QueryTest/create-database.test
M 
testdata/workloads/functional-query/queries/QueryTest/create-table-like-file.test
M 
testdata/workloads/functional-query/queries/QueryTest/create-table-like-table.test
M testdata/workloads/functional-query/queries/QueryTest/create-table.test
M 
testdata/workloads/functional-query/queries/QueryTest/delimited-latin-text.test
M testdata/workloads/functional-query/queries/QueryTest/delimited-text.test
M testdata/workloads/functional-query/queries/QueryTest/describe-path.test
M testdata/workloads/functional-query/queries/QueryTest/functions-ddl.test
M 
testdata/workloads/functional-query/queries/QueryTest/hbase-compute-stats-incremental.test
M testdata/workloads/functional-query/queries/QueryTest/insert_bad_expr.test
M testdata/workloads/functional-query/queries/QueryTest/kudu_alter.test
M testdata/workloads/functional-query/queries/QueryTest/kudu_create.test
M testdata/workloads/functional-query/queries/QueryTest/kudu_insert.test
M testdata/workloads/functional-query/queries/QueryTest/load.test
M testdata/workloads/functional-query/queries/QueryTest/local-filesystem.test
M testdata/workloads/functional-query/queries/QueryTest/truncate-table.test
M testdata/workloads/functional-query/queries/QueryTest/views-ddl.test
25 files changed, 281 insertions(+), 66 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/9090/5
--
To view, visit http://gerrit.cloudera.org:8080/9090
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic542fb8e49e850052416ac663ee329ee3974e3b9
Gerrit-Change-Number: 9090
Gerrit-PatchSet: 5
Gerrit-Owner: Zoltan Borok-Nagy 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Zoltan Borok-Nagy 

[Impala-ASF-CR] IMPALA-5903: Inconsistent specification of result set and result set metadata

[Zoltan Borok-Nagy (Code Review)]

Zoltan Borok-Nagy has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9090 )

Change subject: IMPALA-5903: Inconsistent specification of result set and 
result set metadata
..


Patch Set 4:

(4 comments)

Thanks for the review.

About INVALIDATE METADATA: its TStmtType is DDL, however its TCatalogOpType is 
RESET_METADATA. And it also doesn't return a TDdlExecResponse, but a 
TResetMetadataResponse, which doesn't have a TResultSet.
So, this commit won't affect INVALIDATE/RESET METADATA.

http://gerrit.cloudera.org:8080/#/c/9090/4//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/9090/4//COMMIT_MSG@14
PS4, Line 14: CalatopOpExecutor
> typo
Done


http://gerrit.cloudera.org:8080/#/c/9090/4/fe/src/main/java/org/apache/impala/service/Frontend.java
File fe/src/main/java/org/apache/impala/service/Frontend.java:

http://gerrit.cloudera.org:8080/#/c/9090/4/fe/src/main/java/org/apache/impala/service/Frontend.java@503
PS4, Line 503: if (ddl.op_type == TCatalogOpType.DDL) {
> This might need a one-line comment, e.g. "All DDL commands return a string
Done


http://gerrit.cloudera.org:8080/#/c/9090/4/fe/src/main/java/org/apache/impala/service/Frontend.java@504
PS4, Line 504:   metadata.setColumns(Arrays.asList(new TColumn("summary", 
Type.STRING.toThrift(;
> Can we add a corresponding assert (probably in impala-server.cc) that all (
I added a DCHECK in ClientRequestState::Wait().

The e2e tests already checked that, however. If an operation declared a result 
set schema, then the test started to fetch the results, and if there was no 
result set, the test failed with an error.


http://gerrit.cloudera.org:8080/#/c/9090/4/testdata/workloads/functional-query/queries/QueryTest/alter-table.test
File testdata/workloads/functional-query/queries/QueryTest/alter-table.test:

http://gerrit.cloudera.org:8080/#/c/9090/4/testdata/workloads/functional-query/queries/QueryTest/alter-table.test@234
PS4, Line 234: jointbl_test
> it's a bit inconsistent that we include the table name in this message but
Yeah, I agree. Done.



--
To view, visit http://gerrit.cloudera.org:8080/9090
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic542fb8e49e850052416ac663ee329ee3974e3b9
Gerrit-Change-Number: 9090
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Comment-Date: Wed, 28 Mar 2018 22:48:27 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-6759: align stress test memory estimation parse pattern

[Michael Brown (Code Review)]

Michael Brown has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9844 )

Change subject: IMPALA-6759: align stress test memory estimation parse pattern
..


Patch Set 4:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/9844/4/tests/util/parse_util.py
File tests/util/parse_util.py:

http://gerrit.cloudera.org:8080/#/c/9844/4/tests/util/parse_util.py@59
PS4, Line 59: mem /= 10 ** 6
> I noticed that PrintUtils.java uses 1024 as the base but here we use 1000.
Yes, thanks, that's something else I noticed. I will look into this. I expect 
to open a different review.



--
To view, visit http://gerrit.cloudera.org:8080/9844
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I08976f261582b379696fd0e81bc060577e552309
Gerrit-Change-Number: 9844
Gerrit-PatchSet: 4
Gerrit-Owner: Michael Brown 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Brown 
Gerrit-Reviewer: Michael Ho 
Gerrit-Comment-Date: Wed, 28 Mar 2018 22:37:16 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-6759: align stress test memory estimation parse pattern

[Lars Volker (Code Review)]

Lars Volker has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9844 )

Change subject: IMPALA-6759: align stress test memory estimation parse pattern
..


Patch Set 4:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/9844/4/tests/util/parse_util.py
File tests/util/parse_util.py:

http://gerrit.cloudera.org:8080/#/c/9844/4/tests/util/parse_util.py@59
PS4, Line 59: mem /= 10 ** 6
I noticed that PrintUtils.java uses 1024 as the base but here we use 1000. 
Should we use 1024 here, too?



--
To view, visit http://gerrit.cloudera.org:8080/9844
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I08976f261582b379696fd0e81bc060577e552309
Gerrit-Change-Number: 9844
Gerrit-PatchSet: 4
Gerrit-Owner: Michael Brown 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Brown 
Gerrit-Reviewer: Michael Ho 
Gerrit-Comment-Date: Wed, 28 Mar 2018 22:33:24 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-6759: align stress test memory estimation parse pattern

[Michael Brown (Code Review)]

Michael Brown has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9844 )

Change subject: IMPALA-6759: align stress test memory estimation parse pattern
..


Patch Set 4:

This was a screwup. It was an honest-mistake push to refs/heads/master. I 
reverted this. Working too quickly. Sorry about that.


--
To view, visit http://gerrit.cloudera.org:8080/9844
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I08976f261582b379696fd0e81bc060577e552309
Gerrit-Change-Number: 9844
Gerrit-PatchSet: 4
Gerrit-Owner: Michael Brown 
Gerrit-Reviewer: Michael Brown 
Gerrit-Reviewer: Michael Ho 
Gerrit-Comment-Date: Wed, 28 Mar 2018 22:29:58 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6759: align stress test memory estimation parse pattern

[Michael Brown (Code Review)]

Hello Michael Ho,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/9844

to look at the new patch set (#4).

Change subject: IMPALA-6759: align stress test memory estimation parse pattern
..

IMPALA-6759: align stress test memory estimation parse pattern

The stress test never expected to see memory estimates on the order of
PB. Apparently it can happen with TPC DS 1, so update the pattern.

It's not clear how to quickly write a test to catch this, because it
involves crossing language boundaries and possibly having a
massively-scaled dataset. I think leaving a comment in both places is
good enough for now.

Change-Id: I08976f261582b379696fd0e81bc060577e552309
---
M fe/src/main/java/org/apache/impala/common/PrintUtils.java
M tests/stress/concurrent_select.py
M tests/util/parse_util.py
3 files changed, 9 insertions(+), 3 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/44/9844/4
--
To view, visit http://gerrit.cloudera.org:8080/9844
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I08976f261582b379696fd0e81bc060577e552309
Gerrit-Change-Number: 9844
Gerrit-PatchSet: 4
Gerrit-Owner: Michael Brown 
Gerrit-Reviewer: Michael Brown 
Gerrit-Reviewer: Michael Ho 

[Impala-ASF-CR] IMPALA-6759: align stress test memory estimation parse pattern

[Michael Brown (Code Review)]

Michael Brown has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/9844 )

Change subject: IMPALA-6759: align stress test memory estimation parse pattern
..

IMPALA-6759: align stress test memory estimation parse pattern

The stress test never expected to see memory estimates on the order of
PB. Apparently it can happen with TPC DS 1, so update the pattern.

It's not clear how to quickly write a test to catch this, because it
involves crossing language boundaries and possibly having a
massively-scaled dataset. I think leaving a comment in both places is
good enough for now.

Change-Id: I08976f261582b379696fd0e81bc060577e552309
---
M fe/src/main/java/org/apache/impala/common/PrintUtils.java
M tests/stress/concurrent_select.py
M tests/util/parse_util.py
3 files changed, 9 insertions(+), 3 deletions(-)

--
To view, visit http://gerrit.cloudera.org:8080/9844
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I08976f261582b379696fd0e81bc060577e552309
Gerrit-Change-Number: 9844
Gerrit-PatchSet: 4
Gerrit-Owner: Michael Brown 
Gerrit-Reviewer: Michael Brown 
Gerrit-Reviewer: Michael Ho 

[Impala-ASF-CR] IMPALA-6759: align stress test memory estimation parse pattern

[Michael Ho (Code Review)]

Michael Ho has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9844 )

Change subject: IMPALA-6759: align stress test memory estimation parse pattern
..


Patch Set 3: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/9844
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I08976f261582b379696fd0e81bc060577e552309
Gerrit-Change-Number: 9844
Gerrit-PatchSet: 3
Gerrit-Owner: Michael Brown 
Gerrit-Reviewer: Michael Ho 
Gerrit-Comment-Date: Wed, 28 Mar 2018 22:22:46 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6760: Fix for py2.7-ism in run-tests.py.

[David Knupp (Code Review)]

David Knupp has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9843 )

Change subject: IMPALA-6760: Fix for py2.7-ism in run-tests.py.
..


Patch Set 1:

> Patch Set 1:
>
> I've not yet meaningfully tested this, but I think this 1-liner is reviewable 
> regardless.

Thanks for fixing this Phil. I was just about to open a bug when your review 
popped up.


--
To view, visit http://gerrit.cloudera.org:8080/9843
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I81928d1880a493b91abb13b3a8149568c9789f66
Gerrit-Change-Number: 9843
Gerrit-PatchSet: 1
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: David Knupp 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Brown 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Nithya Janarthanan 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Wed, 28 Mar 2018 22:23:08 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6760: Fix for py2.7-ism in run-tests.py.

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9843 )

Change subject: IMPALA-6760: Fix for py2.7-ism in run-tests.py.
..


Patch Set 1:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2201/


--
To view, visit http://gerrit.cloudera.org:8080/9843
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I81928d1880a493b91abb13b3a8149568c9789f66
Gerrit-Change-Number: 9843
Gerrit-PatchSet: 1
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: David Knupp 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Brown 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Nithya Janarthanan 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Wed, 28 Mar 2018 22:21:18 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6759: align stress test memory estimation parse pattern

[Michael Brown (Code Review)]

Hello Michael Ho,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/9844

to look at the new patch set (#3).

Change subject: IMPALA-6759: align stress test memory estimation parse pattern
..

IMPALA-6759: align stress test memory estimation parse pattern

The stress test never expected to see memory estimates on the order of
PB. Apparently it can happen with TPC DS 1, so update the pattern.

It's not clear how to quickly write a test to catch this, because it
involves crossing language boundaries and possibly having a
massively-scaled dataset. I think leaving a comment in both places is
good enough for now.

Change-Id: I08976f261582b379696fd0e81bc060577e552309
---
M fe/src/main/java/org/apache/impala/common/PrintUtils.java
M tests/stress/concurrent_select.py
2 files changed, 7 insertions(+), 3 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/44/9844/3
--
To view, visit http://gerrit.cloudera.org:8080/9844
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I08976f261582b379696fd0e81bc060577e552309
Gerrit-Change-Number: 9844
Gerrit-PatchSet: 3
Gerrit-Owner: Michael Brown 
Gerrit-Reviewer: Michael Ho 

[Impala-ASF-CR] IMPALA-6760: Fix for py2.7-ism in run-tests.py.

[Philip Zeyliger (Code Review)]

Philip Zeyliger has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9843 )

Change subject: IMPALA-6760: Fix for py2.7-ism in run-tests.py.
..


Patch Set 1:

I've not yet meaningfully tested this, but I think this 1-liner is reviewable 
regardless.


--
To view, visit http://gerrit.cloudera.org:8080/9843
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I81928d1880a493b91abb13b3a8149568c9789f66
Gerrit-Change-Number: 9843
Gerrit-PatchSet: 1
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: David Knupp 
Gerrit-Reviewer: Michael Brown 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Nithya Janarthanan 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Wed, 28 Mar 2018 22:20:48 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6760: Fix for py2.7-ism in run-tests.py.

[Nithya Janarthanan (Code Review)]

Nithya Janarthanan has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9843 )

Change subject: IMPALA-6760: Fix for py2.7-ism in run-tests.py.
..


Patch Set 1: Code-Review+1


--
To view, visit http://gerrit.cloudera.org:8080/9843
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I81928d1880a493b91abb13b3a8149568c9789f66
Gerrit-Change-Number: 9843
Gerrit-PatchSet: 1
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: David Knupp 
Gerrit-Reviewer: Michael Brown 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Nithya Janarthanan 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Wed, 28 Mar 2018 22:20:24 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6759: align stress test memory estimation parse pattern

[Michael Brown (Code Review)]

Hello Michael Ho,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/9844

to look at the new patch set (#2).

Change subject: IMPALA-6759: align stress test memory estimation parse pattern
..

IMPALA-6759: align stress test memory estimation parse pattern

The stress test never expected to see memory estimates on the order of
PB. Apparently it can happen with TPC DS 1, so update the pattern.

It's not clear how to quickly write a test to catch this, because it
involves crossing language boundaries and possibly having a
massively-scaled dataset. I think leaving a comment in both places is
good enough for now.

Change-Id: I08976f261582b379696fd0e81bc060577e552309
---
M fe/src/main/java/org/apache/impala/common/PrintUtils.java
M tests/stress/concurrent_select.py
2 files changed, 7 insertions(+), 3 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/44/9844/2
--
To view, visit http://gerrit.cloudera.org:8080/9844
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I08976f261582b379696fd0e81bc060577e552309
Gerrit-Change-Number: 9844
Gerrit-PatchSet: 2
Gerrit-Owner: Michael Brown 
Gerrit-Reviewer: Michael Ho 

[Impala-ASF-CR] IMPALA-6760: Fix for py2.7-ism in run-tests.py.

[David Knupp (Code Review)]

David Knupp has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9843 )

Change subject: IMPALA-6760: Fix for py2.7-ism in run-tests.py.
..


Patch Set 1: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/9843
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I81928d1880a493b91abb13b3a8149568c9789f66
Gerrit-Change-Number: 9843
Gerrit-PatchSet: 1
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: David Knupp 
Gerrit-Reviewer: Michael Brown 
Gerrit-Reviewer: Nithya Janarthanan 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Wed, 28 Mar 2018 22:18:00 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6759: align stress test memory estimation parse pattern

[Michael Brown (Code Review)]

Michael Brown has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/9844


Change subject: IMPALA-6759: align stress test memory estimation parse pattern
..

IMPALA-6759: align stress test memory estimation parse pattern

The stress test never expected to see memory estimates on the order of
PB. Apparently it can happen with TPC DS 1, so update the pattern.

It's not clear how to quickly write a test to catch this, because it
involves crossing language boundaries and possibly having a
massively-scaled dataset. I think leaving a comment in both places is
good enough for now.

Change-Id: I08976f261582b379696fd0e81bc060577e552309
---
M fe/src/main/java/org/apache/impala/common/PrintUtils.java
M tests/stress/concurrent_select.py
2 files changed, 7 insertions(+), 3 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/44/9844/1
--
To view, visit http://gerrit.cloudera.org:8080/9844
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I08976f261582b379696fd0e81bc060577e552309
Gerrit-Change-Number: 9844
Gerrit-PatchSet: 1
Gerrit-Owner: Michael Brown 

[Impala-ASF-CR] IMPALA-6760: Fix for py2.7-ism in run-tests.py.

[Tim Armstrong (Code Review)]

Tim Armstrong has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9843 )

Change subject: IMPALA-6760: Fix for py2.7-ism in run-tests.py.
..


Patch Set 1: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/9843
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I81928d1880a493b91abb13b3a8149568c9789f66
Gerrit-Change-Number: 9843
Gerrit-PatchSet: 1
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: David Knupp 
Gerrit-Reviewer: Michael Brown 
Gerrit-Reviewer: Nithya Janarthanan 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Wed, 28 Mar 2018 22:17:20 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6760: Fix for py2.7-ism in run-tests.py.

[Philip Zeyliger (Code Review)]

Philip Zeyliger has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/9843


Change subject: IMPALA-6760: Fix for py2.7-ism in run-tests.py.
..

IMPALA-6760: Fix for py2.7-ism in run-tests.py.

A set-literal snuck into run-tests.py in a recent
change. We wish to avoid these to be able to run on
py2.6.

Change-Id: I81928d1880a493b91abb13b3a8149568c9789f66
---
M tests/run-tests.py
1 file changed, 1 insertion(+), 1 deletion(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/9843/1
--
To view, visit http://gerrit.cloudera.org:8080/9843
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I81928d1880a493b91abb13b3a8149568c9789f66
Gerrit-Change-Number: 9843
Gerrit-PatchSet: 1
Gerrit-Owner: Philip Zeyliger 

[Impala-ASF-CR] IMPALA-5717: Support for reading ORC data files

[Tim Armstrong (Code Review)]

Tim Armstrong has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9134 )

Change subject: IMPALA-5717: Support for reading ORC data files
..


Patch Set 8:

(4 comments)

I think we're close to the point where we'd be best off merging this and 
getting the code into the hands of developers and the community so we can play 
around and see how it works in practice.

As you mentioned, you've run this in production so I'm sure a lot of issues 
were already flushed out.

http://gerrit.cloudera.org:8080/#/c/9134/5//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/9134/5//COMMIT_MSG@19
PS5, Line 19:  - Most of the end-to-end tests can run on ORC format.
> Sorry that I should have mentioned this in the commit message. I've enabled
Haha ok, we can't do too much about that :). The ORC project should thank you 
for the free QA. Some of the existing file formats (Seq, RC) also have bugs 
like this with the fuzz test. I was hoping to avoid that from the start here, 
but we can't do too much about that.

Can you create a follow-on JIRA to enable the fuzz test and link the ORC bugs 
you found?

I think we should also add an --enable-orc-scanner feature flag (it can be 
on-by-default) that fails the query if it tries to scan ORC files, just so 
admins have a safety valve to disable it until we've flushed out all the issues.


http://gerrit.cloudera.org:8080/#/c/9134/5/be/src/exec/hdfs-orc-scanner.cc
File be/src/exec/hdfs-orc-scanner.cc:

http://gerrit.cloudera.org:8080/#/c/9134/5/be/src/exec/hdfs-orc-scanner.cc@72
PS5, Line 72:
> We need this as the above reason.
Ah makes sense :(. Naming member functions the same thing as a standard library 
function is pretty annoying!


http://gerrit.cloudera.org:8080/#/c/9134/5/be/src/exec/hdfs-orc-scanner.cc@259
PS5, Line 259:   VLOG_QUERY << "Unknown compression kind of 
orc::CompressionKind: " << kind;
> The ORC library can read ORC files using ZSTD compression. Here we just mis
I had a look and I don't think anything should be broken if you just add it to 
the end of the enum. Enabling ZSTD support to Parquet, other file formats, etc, 
requires making changes elsewhere, so ZSTD wouldn't get accidentally enable by 
adding it to the enum.

There's a similar situation with LZO where it is only supported for text/lzo, 
not Parquet.


http://gerrit.cloudera.org:8080/#/c/9134/7/be/src/exec/hdfs-orc-scanner.cc
File be/src/exec/hdfs-orc-scanner.cc:

http://gerrit.cloudera.org:8080/#/c/9134/7/be/src/exec/hdfs-orc-scanner.cc@77
PS7, Line 77: MemLimitExceeded
Maybe ThrowMemLimitExceeded()?

Also, there's an extra space here.



--
To view, visit http://gerrit.cloudera.org:8080/9134
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia7b6ae4ce3b9ee8125b21993702faa87537790a4
Gerrit-Change-Number: 9134
Gerrit-PatchSet: 8
Gerrit-Owner: Quanlong Huang 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Wed, 28 Mar 2018 22:02:29 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] KUDU-2385: Fix typo in KinitContext::DoRenewal()

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9842 )

Change subject: KUDU-2385: Fix typo in KinitContext::DoRenewal()
..


Patch Set 1:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2200/


--
To view, visit http://gerrit.cloudera.org:8080/9842
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3a22b8d41d15eb1982a3fd5b96575e28edaad31c
Gerrit-Change-Number: 9842
Gerrit-PatchSet: 1
Gerrit-Owner: Sailesh Mukil 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Ho 
Gerrit-Comment-Date: Wed, 28 Mar 2018 21:44:47 +
Gerrit-HasComments: No

[Impala-ASF-CR] KUDU-2385: Fix typo in KinitContext::DoRenewal()

[Michael Ho (Code Review)]

Michael Ho has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9842 )

Change subject: KUDU-2385: Fix typo in KinitContext::DoRenewal()
..


Patch Set 1: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/9842
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3a22b8d41d15eb1982a3fd5b96575e28edaad31c
Gerrit-Change-Number: 9842
Gerrit-PatchSet: 1
Gerrit-Owner: Sailesh Mukil 
Gerrit-Reviewer: Michael Ho 
Gerrit-Comment-Date: Wed, 28 Mar 2018 21:44:21 +
Gerrit-HasComments: No

[Impala-ASF-CR] KUDU-2385: Fix typo in KinitContext::DoRenewal()

[Sailesh Mukil (Code Review)]

Sailesh Mukil has removed Todd Lipcon from this change.  ( 
http://gerrit.cloudera.org:8080/9842 )

Change subject: KUDU-2385: Fix typo in KinitContext::DoRenewal()
..


Removed reviewer Todd Lipcon.
--
To view, visit http://gerrit.cloudera.org:8080/9842
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: deleteReviewer
Gerrit-Change-Id: I3a22b8d41d15eb1982a3fd5b96575e28edaad31c
Gerrit-Change-Number: 9842
Gerrit-PatchSet: 1
Gerrit-Owner: Sailesh Mukil 
Gerrit-Reviewer: Michael Ho 

[Impala-ASF-CR] KUDU-2385: Fix typo in KinitContext::DoRenewal()

[Sailesh Mukil (Code Review)]

Hello Todd Lipcon,

I'd like you to do a code review. Please visit

http://gerrit.cloudera.org:8080/9842

to review the following change.


Change subject: KUDU-2385: Fix typo in KinitContext::DoRenewal()
..

KUDU-2385: Fix typo in KinitContext::DoRenewal()

On platforms without krb5_get_init_creds_opt_set_out_ccache(),
krb5_cc_store_cred() is called to insert the newly acquired
credential into the ccache. However, there was a typo in the code
which resulted in inserting the old credential into ccache.
This change fixes the typo to make sure the new credential is
inserted into ccache.

Testing done: confirmed on SLES11 that the new credential
is being inserted by checking the 'auth time' of the ticket
in ccache. Impala uses a slightly different #ifdef which
explicitly checks if krb5_get_init_creds_opt_set_out_ccache()
is defined on the platform so this code path is actually
used when running Impala on SLES11.

Change-Id: I3a22b8d41d15eb1982a3fd5b96575e28edaad31c
Reviewed-on: http://gerrit.cloudera.org:8080/9840
Reviewed-by: Todd Lipcon 
Tested-by: Todd Lipcon 
---
M be/src/kudu/security/init.cc
1 file changed, 1 insertion(+), 1 deletion(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/42/9842/1
--
To view, visit http://gerrit.cloudera.org:8080/9842
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I3a22b8d41d15eb1982a3fd5b96575e28edaad31c
Gerrit-Change-Number: 9842
Gerrit-PatchSet: 1
Gerrit-Owner: Sailesh Mukil 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Todd Lipcon 

[Impala-ASF-CR] Do clean as part of "bootstrap development.sh"

[Michael Ho (Code Review)]

Michael Ho has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9841 )

Change subject: Do clean as part of "bootstrap_development.sh"
..


Patch Set 1: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/9841
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I238528ca3938200f0750505a62113630290e8c96
Gerrit-Change-Number: 9841
Gerrit-PatchSet: 1
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: Michael Ho 
Gerrit-Comment-Date: Wed, 28 Mar 2018 21:38:17 +
Gerrit-HasComments: No

[Impala-ASF-CR] Do clean as part of "bootstrap development.sh"

[Philip Zeyliger (Code Review)]

Philip Zeyliger has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/9841


Change subject: Do clean as part of "bootstrap_development.sh"
..

Do clean as part of "bootstrap_development.sh"

This commit instructs bootstrap_development.sh, which is
used in Gerrity-Verify-Dryrun jobs, to clean before
running the build. In the common case, this is essentially
a no-op, because the workspace is new to begin with.

This handles an issue where GVDs would fail if they
happened to run on a machine where a previous build ran
against a workspace where the fe/generated-sources
directories were incompatible. I confirmed that
our clean script does clean out fe/generated-sources
nicely.

Change-Id: I238528ca3938200f0750505a62113630290e8c96
---
M bin/bootstrap_development.sh
1 file changed, 1 insertion(+), 1 deletion(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/41/9841/1
--
To view, visit http://gerrit.cloudera.org:8080/9841
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I238528ca3938200f0750505a62113630290e8c96
Gerrit-Change-Number: 9841
Gerrit-PatchSet: 1
Gerrit-Owner: Philip Zeyliger 

[Impala-ASF-CR] IMPALA-6694: fix "buffer pool" child profile order

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/9749 )

Change subject: IMPALA-6694: fix "buffer pool" child profile order
..

IMPALA-6694: fix "buffer pool" child profile order

The bug is that child profiles can be re-ordered when being sent between
an executor and a coordinator. This occurs if child profile A is present
in one update, then another child profile B is inserted at a position
before A and is sent to the coordinator in a subsequent update. The
algorithm for merging profiles did not preserve the order in that case.

The algorithm is fixed to preserve order when the relative order of
child profiles is consistent between all updates.

Testing:
Added a targeted unit test.

Change-Id: I230f0673edf20a846fdb13191b7a292d329c1bb8
Reviewed-on: http://gerrit.cloudera.org:8080/9749
Reviewed-by: Lars Volker 
Tested-by: Impala Public Jenkins
---
M be/src/util/runtime-profile-test.cc
M be/src/util/runtime-profile.cc
2 files changed, 104 insertions(+), 6 deletions(-)

Approvals:
  Lars Volker: Looks good to me, approved
  Impala Public Jenkins: Verified

--
To view, visit http://gerrit.cloudera.org:8080/9749
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I230f0673edf20a846fdb13191b7a292d329c1bb8
Gerrit-Change-Number: 9749
Gerrit-PatchSet: 5
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Tim Armstrong 

[Impala-ASF-CR] IMPALA-6694: fix "buffer pool" child profile order

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9749 )

Change subject: IMPALA-6694: fix "buffer pool" child profile order
..


Patch Set 4: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/9749
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I230f0673edf20a846fdb13191b7a292d329c1bb8
Gerrit-Change-Number: 9749
Gerrit-PatchSet: 4
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Wed, 28 Mar 2018 21:21:35 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6070: Expose using Docker to run tests faster.

[Philip Zeyliger (Code Review)]

Philip Zeyliger has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9085 )

Change subject: IMPALA-6070: Expose using Docker to run tests faster.
..


Patch Set 1:

(11 comments)

http://gerrit.cloudera.org:8080/#/c/9085/1//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/9085/1//COMMIT_MSG@28
PS1, Line 28: Docker filesystems, and entrypoint.sh works around it.
> I think I understand. Would you file a ticket against me to fix or remove t
IMPALA-6737 - bin/bootstrap_development doesn't capture Docker vagaries with 
Kudu

filed.


http://gerrit.cloudera.org:8080/#/c/9085/3//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/9085/3//COMMIT_MSG@48
PS3, Line 48:
> Now that this can be dealt with by static linking, what are your thoughts?
Good instincts on identifying IMPALA-6494 as a static/dynamic linking thing. I 
should have figured it out, given that I saw a few others of those with boost.


http://gerrit.cloudera.org:8080/#/c/9085/1/docker/entrypoint.sh
File docker/entrypoint.sh:

http://gerrit.cloudera.org:8080/#/c/9085/1/docker/entrypoint.sh@79
PS1, Line 79: # Starts SSH and PostgreSQL; configures container as necessary;
> It looks to me like this comment was missed in the last round.
I tried for a little bit, and it's not worth it. I ended up producing 
considerably more lines than the ~4 lines we could be sharing here.

Even the hostname change is slightly different in the two contexts, with this 
context not needing the 127.0.1.1 deletion.


http://gerrit.cloudera.org:8080/#/c/9085/1/docker/entrypoint.sh@142
PS1, Line 142: build_shared_libs
> That build flag is also implicated in IMPALA-6494, it appears.
Yep. I'm ditching shared libs.


http://gerrit.cloudera.org:8080/#/c/9085/3/docker/monitor.py
File docker/monitor.py:

http://gerrit.cloudera.org:8080/#/c/9085/3/docker/monitor.py@221
PS3, Line 221:
> I think it may be hesitant in reformatting comments, which may need rewriti
I think fixed the rest of these.


http://gerrit.cloudera.org:8080/#/c/9085/4/docker/monitor.py
File docker/monitor.py:

http://gerrit.cloudera.org:8080/#/c/9085/4/docker/monitor.py@149
PS4, Line 149:
> This may be an issue of my mediocre Python literacy, but you refer to conta
I didn't change the comment after I changed this to pass container objects 
around. I fixed the comment.

I also changed container.id_ to container.id where possible. I don't like using 
'id' as a variable name, because it conflicts with built-in id, but I don't see 
a reason to avoid it as an attribute name. The only awkwardness is around the 
keyword-args of the constructor, but I think it's clearer without the 
underscore.


http://gerrit.cloudera.org:8080/#/c/9085/4/docker/test-with-docker.py
File docker/test-with-docker.py:

http://gerrit.cloudera.org:8080/#/c/9085/4/docker/test-with-docker.py@49
PS4, Line 49: # or move to different suite.
> nit: can you move these TODOs next to the other ones in this region of comm
Done


http://gerrit.cloudera.org:8080/#/c/9085/4/docker/timeline.html.template
File docker/timeline.html.template:

http://gerrit.cloudera.org:8080/#/c/9085/4/docker/timeline.html.template@23
PS4, Line 23:
> nit: row
Done


http://gerrit.cloudera.org:8080/#/c/9085/4/docker/timeline.html.template@41
PS4, Line 41:
> Is the unit "percent of CPU used in this container since the previous times
Done


http://gerrit.cloudera.org:8080/#/c/9085/4/docker/timeline.html.template@51
PS4, Line 51:
> What is the string?
Done


http://gerrit.cloudera.org:8080/#/c/9085/4/docker/timeline.html.template@77
PS4, Line 77:
In practice, all of these timestamps are relative, so you'd only get an issue 
with daylight savings time if the build took us over 2 months (into March of 
1970). If Python's time.time() isn't monotonic, we'll get garbage out, but I'm 
not going to worry about that here. (Python 3.3 has a time.monotonic, but I 
didn't do the research about py2.7.)

I added a comment about why I'm using relative times. (It's really all about 
tricking the visualization into displaying 01:02:03 for 1h2m3s rather than just 
3722 (the number of seconds in my example).

Figuring this out made me realize that I need to set yr = 1970 in this string. 
It didn't really matter, but it makes the function more correct.

To answer your question about the function at hand: it ends up being 
discontinuous and the visualization will look odd. I'd prefer not to go into 
the rabbit hole of converting the v

> new Date(1970, 0, 0, 0, 0, 0, (152082 - 1) * 1000)
Sun Mar 11 2018 01:59:59 GMT-0800 (PST)
> new Date(1970, 0, 0, 0, 0, 0, (152082) * 1000)
Sun Mar 11 2018 03:00:00 GMT-0700 (PDT)



--
To view, visit http://gerrit.cloudera.org:8080/9085
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I82052ef31979564968effef13a3c6af0d5c62767

[Impala-ASF-CR] IMPALA-5717: Support for reading ORC data files

[Tim Armstrong (Code Review)]

Tim Armstrong has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9134 )

Change subject: IMPALA-5717: Support for reading ORC data files
..


Patch Set 5:

(2 comments)

Answered your questions (sorry it took so long) and working through the whole 
change.

http://gerrit.cloudera.org:8080/#/c/9134/5/be/src/exec/hdfs-orc-scanner.cc
File be/src/exec/hdfs-orc-scanner.cc:

http://gerrit.cloudera.org:8080/#/c/9134/5/be/src/exec/hdfs-orc-scanner.cc@567
PS5, Line 567: // TODO: combine this with the Parquet implementation
> Not quite familiar with the LLVM Codegen codes. I've simply tried extractin
We need to make sure that any functions that need to be used by codegen are in 
an -ir.cc file and compiled into the correct places. Easiest way to see is 
where a current file is included:

$ git grep hdfs-parquet-scanner-ir.cc
be/src/codegen/impala-ir.cc:#include "exec/hdfs-parquet-scanner-ir.cc"
be/src/exec/CMakeLists.txt:  hdfs-parquet-scanner-ir.cc

If we needed to move this to hdfs-scanner, we could add it to 
hdfs-scanner-ir.cc, which is already cross-compiled to LLVM IR.

Then functions that are loaded by codegen need to have the symbol name added to 
be/src/codegen/gen_ir_descriptions.py so that it can be found. Easiest way to 
figure out the symbol name is dump symbols in a binary and find the appropraite 
mangled name, e.g.:

  $ nm -g be/build/latest/service/impalad  | grep EvalRuntimeFilter
  01d47982 T 
_ZN6impala18HdfsParquetScanner17EvalRuntimeFilterEiPNS_8TupleRowE
  01d2f9de T 
_ZN6impala18HdfsParquetScanner18EvalRuntimeFiltersEPNS_8TupleRowE


http://gerrit.cloudera.org:8080/#/c/9134/5/be/src/exec/hdfs-orc-scanner.cc@590
PS5, Line 590: ColumnVectorBatch
> To change this, we have to change the type of scratch_batch_ to unique_ptr<
Can't the caller do the cast though? My thought was that would make the 
function more self-documenting.



--
To view, visit http://gerrit.cloudera.org:8080/9134
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia7b6ae4ce3b9ee8125b21993702faa87537790a4
Gerrit-Change-Number: 9134
Gerrit-PatchSet: 5
Gerrit-Owner: Quanlong Huang 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Wed, 28 Mar 2018 21:14:34 +
Gerrit-HasComments: Yes

[native-toolchain-CR] Bump LLVM to 5.0.1

[Tim Armstrong (Code Review)]

Tim Armstrong has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/8932 )

Change subject: Bump LLVM to 5.0.1
..

Bump LLVM to 5.0.1

Testing:
Was able to build on all supported OSes. Also got Impala to build and
pass tests.

Change-Id: Ib9b7b97cb135d202eaa9a0bae03e722a2505b712
---
M buildall.sh
1 file changed, 4 insertions(+), 2 deletions(-)

Approvals:
  Tim Armstrong: Looks good to me, approved
  Bikramjeet Vig: Verified

--
To view, visit http://gerrit.cloudera.org:8080/8932
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ib9b7b97cb135d202eaa9a0bae03e722a2505b712
Gerrit-Change-Number: 8932
Gerrit-PatchSet: 2
Gerrit-Owner: Bikramjeet Vig 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Tim Armstrong 

[Impala-ASF-CR] IMPALA-5384, part 1: introduce DmlExecState

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9793 )

Change subject: IMPALA-5384, part 1: introduce DmlExecState
..


Patch Set 7: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/2197/


--
To view, visit http://gerrit.cloudera.org:8080/9793
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id4c025917620a7bff2acbeb46464f107ab4b7565
Gerrit-Change-Number: 9793
Gerrit-PatchSet: 7
Gerrit-Owner: Dan Hecht 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Wed, 28 Mar 2018 20:54:41 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6724: Allow creating/dropping functions with the same name as built-ins

[Bharath Vissapragada (Code Review)]

Bharath Vissapragada has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9800 )

Change subject: IMPALA-6724: Allow creating/dropping functions with the same 
name as built-ins
..


Patch Set 8: Code-Review+1

(1 comment)

http://gerrit.cloudera.org:8080/#/c/9800/8/testdata/workloads/functional-query/queries/QueryTest/functions-ddl.test
File testdata/workloads/functional-query/queries/QueryTest/functions-ddl.test:

http://gerrit.cloudera.org:8080/#/c/9800/8/testdata/workloads/functional-query/queries/QueryTest/functions-ddl.test@326
PS8, Line 326: 
I think you missed a show functions here.. "VERIFY_IS_NOT_IN" case, to check 
the drop is successful.



--
To view, visit http://gerrit.cloudera.org:8080/9800
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic30df56ac276970116715c14454a5a2477b185fa
Gerrit-Change-Number: 9800
Gerrit-PatchSet: 8
Gerrit-Owner: Fredy Wijaya 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Vuk Ercegovac 
Gerrit-Comment-Date: Wed, 28 Mar 2018 20:47:40 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-6747: Automate diagnostics collection.

[Philip Zeyliger (Code Review)]

Philip Zeyliger has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9815 )

Change subject: IMPALA-6747: Automate diagnostics collection.
..


Patch Set 3: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/9815
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib29caec7c3be5b6a31e60461294979c318300f64
Gerrit-Change-Number: 9815
Gerrit-PatchSet: 3
Gerrit-Owner: Bharath Vissapragada 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Comment-Date: Wed, 28 Mar 2018 20:42:16 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6724: Allow creating/dropping functions with the same name as built-ins

[Fredy Wijaya (Code Review)]

Fredy Wijaya has uploaded a new patch set (#8). ( 
http://gerrit.cloudera.org:8080/9800 )

Change subject: IMPALA-6724: Allow creating/dropping functions with the same 
name as built-ins
..

IMPALA-6724: Allow creating/dropping functions with the same name as built-ins

This patch removes restriction on creating a function with the same name
as the built-in function. The reason for lifting the restriction is to
avoid a name clash when introducing new built-in functions. The patch
also fixes some inconsistent behavior when creating or dropping a function
when the name specified is fully-qualified or not.

Refer to the below tables for more information.

Create function:
+-+-+-+---+---+
| FQ Name | Built-in DB | Function Name   | Existing Behavior   
  | New Behavior  |
+-+-+-+---+---+
| Yes | Yes | Same as built-in| Same name exception 
  | Cannot modify system database |
| Yes | Yes | Different than built-in | Cannot modify system 
database | Cannot modify system database |
| Yes | No  | Same as built-in| Function created
  | Function created  |
| Yes | No  | Different than built-in | Function created
  | Function created  |
| No  | Yes | Same as built-in| Same name exception 
  | Cannot modify system database |
| No  | Yes | Different than built-in | Cannot modify system 
database | Cannot modify system database |
| No  | No  | Same as built-in| Same name exception 
  | Function created  |
| No  | No  | Different than built-in | Function created
  | Function created  |
+-+-+-+---+---+

Drop function:
+-+-+-+---+---+
| FQ Name | Built-in DB | Function Name   | Existing Behavior   
  | New Behavior  |
+-+-+-+---+---+
| Yes | Yes | Same as built-in| Cannot modify system 
database | Cannot modify system database |
| Yes | Yes | Different than built-in | Cannot modify system 
database | Cannot modify system database |
| Yes | No  | Same as built-in| Function dropped
  | Function dropped  |
| Yes | No  | Different than built-in | Function dropped
  | Function dropped  |
| No  | Yes | Same as built-in| Cannot modify system 
database | Cannot modify system database |
| No  | Yes | Different than built-in | Cannot modify system 
database | Cannot modify system database |
| No  | No  | Same as built-in| Cannot modify system 
database | Function dropped  |
| No  | No  | Different than built-in | Function dropped
  | Function dropped  |
+-+-+-+---+---+

Select function (no new behavior):
+-+-+-++
| FQ Name | Built-in DB | Function Name   | Behavior
   |
+-+-+-++
| Yes | Yes | Same as built-in| Function in the specified 
database (built-in) executed |
| Yes | Yes | Different than built-in | Unknown function exception  
   |
| Yes | No  | Same as built-in| Function in the specified 
database executed|
| Yes | No  | Different than built-in | Function in the specified 
database executed|
| No  | Yes | Same as built-in| Built-in function executed  
   |
| No  | Yes | Different than built-in | Unknown function exception  
   |
| No  | No  | Same as built-in| Built-in function executed  
   |
| No  | No  | Different than built-in | Function in the current 
database executed  |
+-+-+-++

Testing:
- Ran front-end tests
- Added end-to-end DDL function tests

[Impala-ASF-CR] IMPALA-6587: free buffers before ScanRange::Cancel() returns

[Tim Armstrong (Code Review)]

Tim Armstrong has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9680 )

Change subject: IMPALA-6587: free buffers before ScanRange::Cancel() returns
..


Patch Set 14: Code-Review+2

carry


--
To view, visit http://gerrit.cloudera.org:8080/9680
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I87182b6bd51b5fb0b923e7e4c8d08a44e7617db2
Gerrit-Change-Number: 9680
Gerrit-PatchSet: 14
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Wed, 28 Mar 2018 20:33:05 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-4123 (prep): Parquet column reader cleanup

[Tim Armstrong (Code Review)]

Tim Armstrong has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9799 )

Change subject: IMPALA-4123 (prep): Parquet column reader cleanup
..


Patch Set 2:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/9799/2/be/src/exec/parquet-column-readers.cc
File be/src/exec/parquet-column-readers.cc:

http://gerrit.cloudera.org:8080/#/c/9799/2/be/src/exec/parquet-column-readers.cc@299
PS2, Line 299:   bool ReadValueBatch(MemPool* RESTRICT pool, int max_values, 
int tuple_size,
> I also found it quite hard to navigate in this file. I think that the real
Yeah I agree that would make sense (and probably is the way it should have been 
organised initially)



--
To view, visit http://gerrit.cloudera.org:8080/9799
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibc00352df3a0b2d605f872ae7e43db2dc90faab1
Gerrit-Change-Number: 9799
Gerrit-PatchSet: 2
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Wed, 28 Mar 2018 20:31:20 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-6747: Automate diagnostics collection.

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9815 )

Change subject: IMPALA-6747: Automate diagnostics collection.
..


Patch Set 3:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2199/


--
To view, visit http://gerrit.cloudera.org:8080/9815
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib29caec7c3be5b6a31e60461294979c318300f64
Gerrit-Change-Number: 9815
Gerrit-PatchSet: 3
Gerrit-Owner: Bharath Vissapragada 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Comment-Date: Wed, 28 Mar 2018 20:28:18 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6747: Automate diagnostics collection.

[Lars Volker (Code Review)]

Lars Volker has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9815 )

Change subject: IMPALA-6747: Automate diagnostics collection.
..


Patch Set 3: Code-Review+2

I'm +2ing this to allow us to get it into the next release. This change has 
seen a reasonable amount of reviewing and comments downstream before we decided 
to move it to the upstream repository. I am confident that it's up to our 
standards of code quality and has been vetted for defects reasonably well. In 
addition, it has been up for reviews by the community for 2 additional days.


--
To view, visit http://gerrit.cloudera.org:8080/9815
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib29caec7c3be5b6a31e60461294979c318300f64
Gerrit-Change-Number: 9815
Gerrit-PatchSet: 3
Gerrit-Owner: Bharath Vissapragada 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Comment-Date: Wed, 28 Mar 2018 20:26:27 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6724: Incorrect exception handling in create/drop function statements

[Bharath Vissapragada (Code Review)]

Bharath Vissapragada has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9800 )

Change subject: IMPALA-6724: Incorrect exception handling in create/drop 
function statements
..


Patch Set 7:

(3 comments)

Patch lgtm. I can +1 once my comments are addressed.

http://gerrit.cloudera.org:8080/#/c/9800/7//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/9800/7//COMMIT_MSG@7
PS7, Line 7: Incorrect exception handling in create/drop function statements
I think the more accurate title would be to allow add/drop of functions with 
the same name as built-ins?


http://gerrit.cloudera.org:8080/#/c/9800/7//COMMIT_MSG@61
PS7, Line 61: DLL
DDL?


http://gerrit.cloudera.org:8080/#/c/9800/7/testdata/workloads/functional-query/queries/QueryTest/functions-ddl.test
File testdata/workloads/functional-query/queries/QueryTest/functions-ddl.test:

http://gerrit.cloudera.org:8080/#/c/9800/7/testdata/workloads/functional-query/queries/QueryTest/functions-ddl.test@302
PS7, Line 302: 
Can you add SHOW functions in $DATABASE  after create and drop for these tests?



--
To view, visit http://gerrit.cloudera.org:8080/9800
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic30df56ac276970116715c14454a5a2477b185fa
Gerrit-Change-Number: 9800
Gerrit-PatchSet: 7
Gerrit-Owner: Fredy Wijaya 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Vuk Ercegovac 
Gerrit-Comment-Date: Wed, 28 Mar 2018 20:26:08 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-6724: Incorrect exception handling in create/drop function statements

[Fredy Wijaya (Code Review)]

Fredy Wijaya has uploaded a new patch set (#7). ( 
http://gerrit.cloudera.org:8080/9800 )

Change subject: IMPALA-6724: Incorrect exception handling in create/drop 
function statements
..

IMPALA-6724: Incorrect exception handling in create/drop function statements

This patch removes restriction on creating a function with the same name
as the built-in function. The reason for lifting the restriction is to
avoid a name clash when introducing new built-in functions. The patch
also fixes some inconsistent behavior when creating or dropping a function
when the name specified is fully-qualified or not.

Refer to the below tables for more information.

Create function:
+-+-+-+---+---+
| FQ Name | Built-in DB | Function Name   | Existing Behavior   
  | New Behavior  |
+-+-+-+---+---+
| Yes | Yes | Same as built-in| Same name exception 
  | Cannot modify system database |
| Yes | Yes | Different than built-in | Cannot modify system 
database | Cannot modify system database |
| Yes | No  | Same as built-in| Function created
  | Function created  |
| Yes | No  | Different than built-in | Function created
  | Function created  |
| No  | Yes | Same as built-in| Same name exception 
  | Cannot modify system database |
| No  | Yes | Different than built-in | Cannot modify system 
database | Cannot modify system database |
| No  | No  | Same as built-in| Same name exception 
  | Function created  |
| No  | No  | Different than built-in | Function created
  | Function created  |
+-+-+-+---+---+

Drop function:
+-+-+-+---+---+
| FQ Name | Built-in DB | Function Name   | Existing Behavior   
  | New Behavior  |
+-+-+-+---+---+
| Yes | Yes | Same as built-in| Cannot modify system 
database | Cannot modify system database |
| Yes | Yes | Different than built-in | Cannot modify system 
database | Cannot modify system database |
| Yes | No  | Same as built-in| Function dropped
  | Function dropped  |
| Yes | No  | Different than built-in | Function dropped
  | Function dropped  |
| No  | Yes | Same as built-in| Cannot modify system 
database | Cannot modify system database |
| No  | Yes | Different than built-in | Cannot modify system 
database | Cannot modify system database |
| No  | No  | Same as built-in| Cannot modify system 
database | Function dropped  |
| No  | No  | Different than built-in | Function dropped
  | Function dropped  |
+-+-+-+---+---+

Select function (no new behavior):
+-+-+-++
| FQ Name | Built-in DB | Function Name   | Behavior
   |
+-+-+-++
| Yes | Yes | Same as built-in| Function in the specified 
database (built-in) executed |
| Yes | Yes | Different than built-in | Unknown function exception  
   |
| Yes | No  | Same as built-in| Function in the specified 
database executed|
| Yes | No  | Different than built-in | Function in the specified 
database executed|
| No  | Yes | Same as built-in| Built-in function executed  
   |
| No  | Yes | Different than built-in | Unknown function exception  
   |
| No  | No  | Same as built-in| Built-in function executed  
   |
| No  | No  | Different than built-in | Function in the current 
database executed  |
+-+-+-++

Testing:
- Ran front-end tests
- Added end-to-end DLL function tests

Cherry-picks: 

[Impala-ASF-CR] IMPALA-6685: Improve profiles in KrpcDataStreamRecvr and KrpcDataStreamSender

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9690 )

Change subject: IMPALA-6685: Improve profiles in KrpcDataStreamRecvr and 
KrpcDataStreamSender
..


Patch Set 8: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/9690
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8ba405921b3df920c1e85b940ce9c8d02fc647cd
Gerrit-Change-Number: 9690
Gerrit-PatchSet: 8
Gerrit-Owner: Michael Ho 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Mostafa Mokhtar 
Gerrit-Comment-Date: Wed, 28 Mar 2018 19:21:45 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6685: Improve profiles in KrpcDataStreamRecvr and KrpcDataStreamSender

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/9690 )

Change subject: IMPALA-6685: Improve profiles in KrpcDataStreamRecvr and 
KrpcDataStreamSender
..

IMPALA-6685: Improve profiles in KrpcDataStreamRecvr and KrpcDataStreamSender

This change implements a couple of improvements to the profiles of
KrpcDataStreamRecvr and KrpcDataStreamSender:

- track pending number of deferred row batches over time in KrpcDataStreamRecvr
- track the number of bytes dequeued over time in KrpcDataStreamRecvr
- track the total time deferred RPCs queues are not empty
- track the number of bytes sent from KrpcDataStreamSender over time
- track the total amount of time spent in KrpcDataStreamSender, including time
  spent waiting for RPC completion.

Sample profile of an Exchange node instance:

  EXCHANGE_NODE (id=21):(Total: 2s284ms, non-child: 64.926ms, % 
non-child: 2.84%)
 - ConvertRowBatchTime: 44.380ms
 - PeakMemoryUsage: 124.04 KB (127021)
 - RowsReturned: 287.51K (287514)
 - RowsReturnedRate: 125.88 K/sec
Buffer pool:
   - AllocTime: 1.109ms
   - CumulativeAllocationBytes: 10.96 MB (11493376)
   - CumulativeAllocations: 562 (562)
   - PeakReservation: 112.00 KB (114688)
   - PeakUnpinnedBytes: 0
   - PeakUsedReservation: 112.00 KB (114688)
   - ReadIoBytes: 0
   - ReadIoOps: 0 (0)
   - ReadIoWaitTime: 0.000ns
   - WriteIoBytes: 0
   - WriteIoOps: 0 (0)
   - WriteIoWaitTime: 0.000ns
Dequeue:
  BytesDequeued(500.000ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 700.00 KB, 2.00 MB, 3.49 
MB, 4.39 MB, 5.86 MB, 6.85 MB
   - FirstBatchWaitTime: 0.000ns
   - TotalBytesDequeued: 6.85 MB (7187850)
   - TotalGetBatchTime: 2s237ms
 - DataWaitTime: 2s219ms
Enqueue:
  BytesReceived(500.000ms): 0, 23.36 KB, 23.36 KB, 23.36 KB, 23.36 
KB, 23.36 KB, 23.36 KB, 23.36 KB, 23.36 KB, 23.36 KB, 23.36 KB, 23.36 KB, 23.36 
KB, 23.36 KB, 23.36 KB, 23.36 KB, 23.36 KB, 23.36 KB, 23.36 KB, 23.36 KB, 23.36 
KB, 23.36 KB, 23.36 KB, 23.36 KB, 23.36 KB, 23.36 KB, 23.36 KB, 23.36 KB, 23.36 
KB, 23.36 KB, 23.36 KB, 328.73 KB, 963.79 KB, 1.64 MB, 2.09 MB, 2.76 MB, 3.23 MB
  DeferredQueueSize(500.000ms): 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0
   - DispatchTime: (Avg: 108.593us ; Min: 30.525us ; Max: 1.524ms ; 
Number of samples: 281)
   - DeserializeRowBatchTime: 8.395ms
   - TotalBatchesEnqueued: 281 (281)
   - TotalBatchesReceived: 281 (281)
   - TotalBytesReceived: 3.23 MB (3387144)
   - TotalEarlySenders: 0 (0)
   - TotalEosReceived: 1 (1)
   - TotalHasDeferredRPCsTime: 15s446ms
   - TotalRPCsDeferred: 38 (38)

Sample sender's profile:

KrpcDataStreamSender (dst_id=21):(Total: 17s923ms, non-child: 
604.494ms, % non-child: 3.37%)
  BytesSent(500.000ms): 0, 0, 0, 34.78 KB, 34.78 KB, 34.78 KB, 34.78 
KB, 34.78 KB, 34.78 KB, 34.78 KB, 34.78 KB, 34.78 KB, 34.78 KB, 34.78 KB, 34.78 
KB, 34.78 KB, 34.78 KB, 34.78 KB, 34.78 KB, 34.78 KB, 34.78 KB, 34.78 KB, 34.78 
KB, 46.54 KB, 46.54 KB, 46.54 KB, 58.31 KB, 58.31 KB, 58.31 KB, 58.31 KB, 58.31 
KB, 58.31 KB, 58.31 KB, 974.44 KB, 2.82 MB, 4.93 MB, 6.27 MB, 8.28 MB, 9.69 MB
   - EosSent: 3 (3)
   - NetworkThroughput: 4.61 MB/sec
   - PeakMemoryUsage: 22.57 KB (23112)
   - RowsSent: 287.51K (287514)
   - RpcFailure: 0 (0)
   - RpcRetry: 0 (0)
   - SerializeBatchTime: 329.162ms
   - TotalBytesSent: 9.69 MB (10161432)
   - UncompressedRowBatchSize: 20.56 MB (21563550)

Change-Id: I8ba405921b3df920c1e85b940ce9c8d02fc647cd
Reviewed-on: http://gerrit.cloudera.org:8080/9690
Reviewed-by: Michael Ho 
Tested-by: Impala Public Jenkins
---
M be/src/runtime/data-stream-test.cc
M be/src/runtime/krpc-data-stream-mgr.cc
M be/src/runtime/krpc-data-stream-recvr.cc
M be/src/runtime/krpc-data-stream-recvr.h
M be/src/runtime/krpc-data-stream-sender.cc
M be/src/runtime/krpc-data-stream-sender.h
M be/src/runtime/runtime-state.h
M be/src/service/data-stream-service.cc
M be/src/service/data-stream-service.h
M common/protobuf/data_stream_service.proto
10 files changed, 281 insertions(+), 138 deletions(-)

Approvals:
  Michael Ho: Looks good to me, approved
  Impala Public Jenkins: Verified

--
To view, visit http://gerrit.cloudera.org:8080/9690
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master

[Impala-ASF-CR] IMPALA-5721,IMPALA-6717,IMPALA-6738: improve stress test binary search

[Michael Brown (Code Review)]

Michael Brown has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9770 )

Change subject: IMPALA-5721,IMPALA-6717,IMPALA-6738: improve stress test binary 
search
..


Patch Set 7:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/9770/7/tests/stress/concurrent_select.py
File tests/stress/concurrent_select.py:

http://gerrit.cloudera.org:8080/#/c/9770/7/tests/stress/concurrent_select.py@141
PS7, Line 141:   if hasattr(self, curate_method_name):
> Yeah, __getattribute__ is the weird one. I think of implementing __getattr_
Somehow I missed this comment. I implemented your __getattr__, but not the args 
overwrite. So half-done in the latest patch set.



--
To view, visit http://gerrit.cloudera.org:8080/9770
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I33d036ec93df3016cd4703205078dbdba0168acb
Gerrit-Change-Number: 9770
Gerrit-PatchSet: 7
Gerrit-Owner: Michael Brown 
Gerrit-Reviewer: David Knupp 
Gerrit-Reviewer: Michael Brown 
Gerrit-Reviewer: Nithya Janarthanan 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Wed, 28 Mar 2018 18:34:23 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-5721,IMPALA-6717,IMPALA-6738: improve stress test binary search

[Michael Brown (Code Review)]

Hello Nithya Janarthanan, David Knupp, Tim Armstrong,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/9770

to look at the new patch set (#9).

Change subject: IMPALA-5721,IMPALA-6717,IMPALA-6738: improve stress test binary 
search
..

IMPALA-5721,IMPALA-6717,IMPALA-6738: improve stress test binary search

IMPALA-5721:
- Save profiles of queries at the end of both the spilling and
  non-spilling binary search. These were not being saved before. Note
  these profiles won't have ExecSummary until IMPALA-6640 is addressed.

- Save the profile of any query that produces incorrect results during
  binary search. These were not being saved before, either.

- Use descriptive names, like
  tpch_100_parquet_q12_profile_without_spilling.txt, for profiles
  mentioned above. We do this by introducing the concept of a
  "logical_query_id" whose values look like "tpch_100_parquet_q12".

- Use the logical_query_id in critical error paths and include the
  logical_query_id in result hash files.

IMPALA-6717:
- Plumb --common-query-options through to the binary search.

IMPALA-6738:
- Begin a refactoring to reduce the number of parameters used when doing
  the binary search.

- Introduce a notion of "converted args" via class that does the
  conversion (if needed) via property getters.

- Adjust populate_all_queries() to use converted_args

Change-Id: I33d036ec93df3016cd4703205078dbdba0168acb
---
M tests/stress/concurrent_select.py
1 file changed, 213 insertions(+), 76 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/70/9770/9
--
To view, visit http://gerrit.cloudera.org:8080/9770
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I33d036ec93df3016cd4703205078dbdba0168acb
Gerrit-Change-Number: 9770
Gerrit-PatchSet: 9
Gerrit-Owner: Michael Brown 
Gerrit-Reviewer: David Knupp 
Gerrit-Reviewer: Michael Brown 
Gerrit-Reviewer: Nithya Janarthanan 
Gerrit-Reviewer: Tim Armstrong 

[Impala-ASF-CR] IMPALA-5842: Write page index in Parquet files

[Zoltan Borok-Nagy (Code Review)]

Zoltan Borok-Nagy has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9693 )

Change subject: IMPALA-5842: Write page index in Parquet files
..


Patch Set 6:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/9693/6/tests/query_test/test_parquet_page_index.py
File tests/query_test/test_parquet_page_index.py:

http://gerrit.cloudera.org:8080/#/c/9693/6/tests/query_test/test_parquet_page_index.py@163
PS6, Line 163: for null_page in null_pages:
 :   assert null_page
> It would be nice to always add the column name and the page number (if appl
I added the column name to this specific error message.

Adding that information to all the asserts would need a lot of code 
modifications and I'm not sure that it would worth the effort.



--
To view, visit http://gerrit.cloudera.org:8080/9693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icbacf7fe3b7672e3ce719261ecef445b16f8dec9
Gerrit-Change-Number: 9693
Gerrit-PatchSet: 6
Gerrit-Owner: Zoltan Borok-Nagy 
Gerrit-Reviewer: Anonymous Coward #248
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Comment-Date: Wed, 28 Mar 2018 18:06:35 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-5842: Write page index in Parquet files

[Zoltan Borok-Nagy (Code Review)]

Hello Lars Volker, Anonymous Coward #248, Tim Armstrong, Csaba Ringhofer,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/9693

to look at the new patch set (#8).

Change subject: IMPALA-5842: Write page index in Parquet files
..

IMPALA-5842: Write page index in Parquet files

This commit builds on the previous work of
Pooja Nilangekar: https://gerrit.cloudera.org/#/c/7464/

The commit implements the write path of PARQUET-922:
"Add column indexes to parquet.thrift". As specified in the
parquet-format, Impala writes the page indexes just before
the footer. This allows much more efficient page filtering
than using the same information from the 'statistics' field
of DataPageHeader.

I updated Pooja's python tests as well.

Change-Id: Icbacf7fe3b7672e3ce719261ecef445b16f8dec9
---
M be/src/exec/hdfs-parquet-table-writer.cc
M be/src/exec/hdfs-parquet-table-writer.h
M be/src/exec/parquet-column-stats.h
M be/src/exec/parquet-column-stats.inline.h
M common/thrift/parquet.thrift
A tests/query_test/test_parquet_page_index.py
M tests/util/get_parquet_metadata.py
7 files changed, 566 insertions(+), 12 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/93/9693/8
--
To view, visit http://gerrit.cloudera.org:8080/9693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Icbacf7fe3b7672e3ce719261ecef445b16f8dec9
Gerrit-Change-Number: 9693
Gerrit-PatchSet: 8
Gerrit-Owner: Zoltan Borok-Nagy 
Gerrit-Reviewer: Anonymous Coward #248
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Zoltan Borok-Nagy 

[Impala-ASF-CR] IMPALA-6739: Incorrect exception in ALTER TABLE SET statements

[Adam Holley (Code Review)]

Adam Holley has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9819 )

Change subject: IMPALA-6739: Incorrect exception in ALTER TABLE SET statements
..


Patch Set 5:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/9819/5/fe/src/main/java/org/apache/impala/analysis/AlterTableSetStmt.java
File fe/src/main/java/org/apache/impala/analysis/AlterTableSetStmt.java:

http://gerrit.cloudera.org:8080/#/c/9819/5/fe/src/main/java/org/apache/impala/analysis/AlterTableSetStmt.java@57
PS5, Line 57: partitionSet_ != null
Don't need this check.



--
To view, visit http://gerrit.cloudera.org:8080/9819
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I793e827d5cf5b7986bd150dd9706df58da3417f3
Gerrit-Change-Number: 9819
Gerrit-PatchSet: 5
Gerrit-Owner: Fredy Wijaya 
Gerrit-Reviewer: Adam Holley 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Comment-Date: Wed, 28 Mar 2018 17:59:34 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-6694: fix "buffer pool" child profile order

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9749 )

Change subject: IMPALA-6694: fix "buffer pool" child profile order
..


Patch Set 4:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2198/


--
To view, visit http://gerrit.cloudera.org:8080/9749
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I230f0673edf20a846fdb13191b7a292d329c1bb8
Gerrit-Change-Number: 9749
Gerrit-PatchSet: 4
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Wed, 28 Mar 2018 17:43:19 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6694: fix "buffer pool" child profile order

[Lars Volker (Code Review)]

Lars Volker has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9749 )

Change subject: IMPALA-6694: fix "buffer pool" child profile order
..


Patch Set 4: Code-Review+2

With the additional explanation and after another look I feel confident to +2 
this one.


--
To view, visit http://gerrit.cloudera.org:8080/9749
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I230f0673edf20a846fdb13191b7a292d329c1bb8
Gerrit-Change-Number: 9749
Gerrit-PatchSet: 4
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Wed, 28 Mar 2018 17:42:52 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6587: free buffers before ScanRange::Cancel() returns

[Bikramjeet Vig (Code Review)]

Bikramjeet Vig has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9680 )

Change subject: IMPALA-6587: free buffers before ScanRange::Cancel() returns
..


Patch Set 14:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/9680/12/be/src/runtime/io/request-context.cc
File be/src/runtime/io/request-context.cc:

http://gerrit.cloudera.org:8080/#/c/9680/12/be/src/runtime/io/request-context.cc@74
PS12, Line 74:   if (outcome == ReadOutcome::SUCCESS_EOSR) {
 : // No more reads to do.
 : --disk_state->num_remaining_ranges();
 :   } else if (outcome == ReadOutcome::SUCCESS_NO_EOSR) {
 : // Schedule the next read.
 : if (state_ != RequestContext::Cancelled) {
 :   ScheduleScanRange(lock, range);
 : }
 :   } else if (outcome == ReadOutcome::BLOCKED_ON_BUFFER) {
 : // Do nothing - the caller must add a buffer to the range or 
cancel it.
 :   } else {
 : DCHECK(outcome == ReadOutcome::CANCELLED) << 
static_cast(outcome);
 : // No more reads - clean up the scan range.
 : --disk_state->num_remaining_ranges();
 : RemoveActiveScanRangeLocked(lock, range);
 :   }
> I tried both earlier and thought this was actually more readable (it was cl
I am good with this. dont have a strong preference to change it



--
To view, visit http://gerrit.cloudera.org:8080/9680
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I87182b6bd51b5fb0b923e7e4c8d08a44e7617db2
Gerrit-Change-Number: 9680
Gerrit-PatchSet: 14
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Wed, 28 Mar 2018 17:19:53 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-6739: Incorrect exception in ALTER TABLE SET statements

[Fredy Wijaya (Code Review)]

Fredy Wijaya has uploaded a new patch set (#5). ( 
http://gerrit.cloudera.org:8080/9819 )

Change subject: IMPALA-6739: Incorrect exception in ALTER TABLE SET statements
..

IMPALA-6739: Incorrect exception in ALTER TABLE SET statements

The patch fixes an issue with executing ALTER TABLE SET statements when
there are no matching partitions by throwing an AnalysisException.

Testing:
- Added a new test
- Ran all front-end tests

Change-Id: I793e827d5cf5b7986bd150dd9706df58da3417f3
---
M fe/src/main/java/org/apache/impala/analysis/AlterTableSetStmt.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
2 files changed, 18 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/9819/5
--
To view, visit http://gerrit.cloudera.org:8080/9819
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I793e827d5cf5b7986bd150dd9706df58da3417f3
Gerrit-Change-Number: 9819
Gerrit-PatchSet: 5
Gerrit-Owner: Fredy Wijaya 
Gerrit-Reviewer: Adam Holley 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Fredy Wijaya 

[Impala-ASF-CR] IMPALA-6587: free buffers before ScanRange::Cancel() returns

[Dan Hecht (Code Review)]

Dan Hecht has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9680 )

Change subject: IMPALA-6587: free buffers before ScanRange::Cancel() returns
..


Patch Set 12:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/9680/12/be/src/runtime/io/request-context.cc
File be/src/runtime/io/request-context.cc:

http://gerrit.cloudera.org:8080/#/c/9680/12/be/src/runtime/io/request-context.cc@74
PS12, Line 74:   if (outcome == ReadOutcome::SUCCESS_EOSR) {
 : // No more reads to do.
 : --disk_state->num_remaining_ranges();
 :   } else if (outcome == ReadOutcome::SUCCESS_NO_EOSR) {
 : // Schedule the next read.
 : if (state_ != RequestContext::Cancelled) {
 :   ScheduleScanRange(lock, range);
 : }
 :   } else if (outcome == ReadOutcome::BLOCKED_ON_BUFFER) {
 : // Do nothing - the caller must add a buffer to the range or 
cancel it.
 :   } else {
 : DCHECK(outcome == ReadOutcome::CANCELLED) << 
static_cast(outcome);
 : // No more reads - clean up the scan range.
 : --disk_state->num_remaining_ranges();
 : RemoveActiveScanRangeLocked(lock, range);
 :   }
> I tried both earlier and thought this was actually more readable (it was cl
FWIW I find the if-stmt easier to read also, though don't feel too strongly 
either.  Also, not that it matters in this case, but this structure implies 
(and optimizes for) the fast and slow paths, which can be nice.



--
To view, visit http://gerrit.cloudera.org:8080/9680
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I87182b6bd51b5fb0b923e7e4c8d08a44e7617db2
Gerrit-Change-Number: 9680
Gerrit-PatchSet: 12
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Wed, 28 Mar 2018 17:09:23 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-5384, part 1: introduce DmlExecState

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9793 )

Change subject: IMPALA-5384, part 1: introduce DmlExecState
..


Patch Set 7:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2197/


--
To view, visit http://gerrit.cloudera.org:8080/9793
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id4c025917620a7bff2acbeb46464f107ab4b7565
Gerrit-Change-Number: 9793
Gerrit-PatchSet: 7
Gerrit-Owner: Dan Hecht 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Wed, 28 Mar 2018 17:05:07 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5384, part 1: introduce DmlExecState

[Dan Hecht (Code Review)]

Dan Hecht has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9793 )

Change subject: IMPALA-5384, part 1: introduce DmlExecState
..


Patch Set 7: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/9793
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id4c025917620a7bff2acbeb46464f107ab4b7565
Gerrit-Change-Number: 9793
Gerrit-PatchSet: 7
Gerrit-Owner: Dan Hecht 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Wed, 28 Mar 2018 17:04:59 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-4835: switch I/O buffers to buffer pool

[Tim Armstrong (Code Review)]

Tim Armstrong has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9679 )

Change subject: IMPALA-4835: switch I/O buffers to buffer pool
..


Patch Set 9: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/9679
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3ef471dc0746f0ab93b572c34024fc7343161f00
Gerrit-Change-Number: 9679
Gerrit-PatchSet: 9
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Wed, 28 Mar 2018 17:04:33 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6587: free buffers before ScanRange::Cancel() returns

[Tim Armstrong (Code Review)]

Tim Armstrong has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9680 )

Change subject: IMPALA-6587: free buffers before ScanRange::Cancel() returns
..


Patch Set 12:

(4 comments)

http://gerrit.cloudera.org:8080/#/c/9680/12/be/src/runtime/io/disk-io-mgr.cc
File be/src/runtime/io/disk-io-mgr.cc:

http://gerrit.cloudera.org:8080/#/c/9680/12/be/src/runtime/io/disk-io-mgr.cc@a651
PS12, Line 651:
> any reason why we needed reader->Validate() at both the beginning and end o
I don't think there's a specific reason - it might catch that a precondition to 
the method was violated and make it slightly easier to debug, but I think the 
benefit is marginal. Usually it's easiest to debug if we check postconditions 
of methods so that we catch the problem in the method that caused it.


http://gerrit.cloudera.org:8080/#/c/9680/12/be/src/runtime/io/request-context.cc
File be/src/runtime/io/request-context.cc:

http://gerrit.cloudera.org:8080/#/c/9680/12/be/src/runtime/io/request-context.cc@74
PS12, Line 74:   if (outcome == ReadOutcome::SUCCESS_EOSR) {
 : // No more reads to do.
 : --disk_state->num_remaining_ranges();
 :   } else if (outcome == ReadOutcome::SUCCESS_NO_EOSR) {
 : // Schedule the next read.
 : if (state_ != RequestContext::Cancelled) {
 :   ScheduleScanRange(lock, range);
 : }
 :   } else if (outcome == ReadOutcome::BLOCKED_ON_BUFFER) {
 : // Do nothing - the caller must add a buffer to the range or 
cancel it.
 :   } else {
 : DCHECK(outcome == ReadOutcome::CANCELLED) << 
static_cast(outcome);
 : // No more reads - clean up the scan range.
 : --disk_state->num_remaining_ranges();
 : RemoveActiveScanRangeLocked(lock, range);
 :   }
> looks like a good candidate for a switch-case
I tried both earlier and thought this was actually more readable (it was close) 
- it's slightly shorter and if/else if cascades are a bit less error-prone than 
switch/case. I can change if there's a specific reason to.


http://gerrit.cloudera.org:8080/#/c/9680/12/be/src/runtime/io/request-ranges.h
File be/src/runtime/io/request-ranges.h:

http://gerrit.cloudera.org:8080/#/c/9680/12/be/src/runtime/io/request-ranges.h@277
PS12, Line 277:
> nit: extra space
Done


http://gerrit.cloudera.org:8080/#/c/9680/12/be/src/runtime/io/request-ranges.h@477
PS12, Line 477: StartRead() and set to false in EnqueueReadyBuffer() or 
ReadFailed()
> update comment, StartRead and ReadFailed dont exist anymore
Done



--
To view, visit http://gerrit.cloudera.org:8080/9680
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I87182b6bd51b5fb0b923e7e4c8d08a44e7617db2
Gerrit-Change-Number: 9680
Gerrit-PatchSet: 12
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Wed, 28 Mar 2018 17:03:40 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-6587: free buffers before ScanRange::Cancel() returns

[Tim Armstrong (Code Review)]

Hello Bikramjeet Vig, Impala Public Jenkins, Dan Hecht,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/9680

to look at the new patch set (#13).

Change subject: IMPALA-6587: free buffers before ScanRange::Cancel() returns
..

IMPALA-6587: free buffers before ScanRange::Cancel() returns

ScanRange::Cancel() now waits until an in-flight read finishes so
that the disk I/O buffer being processed by the disk thread is
freed when Cancel() returns.

The fix is to set a 'read_in_flight_' flag on the scan range
while the disk thread is doing the read. Cancel() blocks until
read_in_flight_ == false.

The code is refactored to move more logic into ScanRange and
to avoid holding RequestContext::lock_ for longer than necessary.

Testing:
Added query test that reproduces the issue.

Added a unit test and a stress option that reproduces the problem in a
targeted way.

Ran disk-io-mgr-stress test for a few hours. Ran it under TSAN and
inspected output to make sure there were no non-benign data races.

Change-Id: I87182b6bd51b5fb0b923e7e4c8d08a44e7617db2
---
M be/src/common/global-flags.cc
M be/src/runtime/io/disk-io-mgr-internal.h
M be/src/runtime/io/disk-io-mgr-stress.cc
M be/src/runtime/io/disk-io-mgr-test.cc
M be/src/runtime/io/disk-io-mgr.cc
M be/src/runtime/io/disk-io-mgr.h
M be/src/runtime/io/request-context.cc
M be/src/runtime/io/request-context.h
M be/src/runtime/io/request-ranges.h
M be/src/runtime/io/scan-range.cc
M testdata/workloads/functional-query/queries/QueryTest/scanners.test
11 files changed, 288 insertions(+), 174 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/9680/13
--
To view, visit http://gerrit.cloudera.org:8080/9680
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I87182b6bd51b5fb0b923e7e4c8d08a44e7617db2
Gerrit-Change-Number: 9680
Gerrit-PatchSet: 13
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong 

[Impala-ASF-CR] IMPALA-5384, part 1: introduce DmlExecState

[Tim Armstrong (Code Review)]

Tim Armstrong has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9793 )

Change subject: IMPALA-5384, part 1: introduce DmlExecState
..


Patch Set 6:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/9793/6/be/src/runtime/dml-exec-state.h
File be/src/runtime/dml-exec-state.h:

http://gerrit.cloudera.org:8080/#/c/9793/6/be/src/runtime/dml-exec-state.h@44
PS6, Line 44: status
> But the thing is called TInsertPartitionStatus. There's other metadata in a
Ah oh, I misunderstood



--
To view, visit http://gerrit.cloudera.org:8080/9793
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id4c025917620a7bff2acbeb46464f107ab4b7565
Gerrit-Change-Number: 9793
Gerrit-PatchSet: 6
Gerrit-Owner: Dan Hecht 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Wed, 28 Mar 2018 16:44:50 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-5384, part 1: introduce DmlExecState

[Dan Hecht (Code Review)]

Dan Hecht has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9793 )

Change subject: IMPALA-5384, part 1: introduce DmlExecState
..


Patch Set 6:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/9793/6/be/src/runtime/dml-exec-state.h
File be/src/runtime/dml-exec-state.h:

http://gerrit.cloudera.org:8080/#/c/9793/6/be/src/runtime/dml-exec-state.h@44
PS6, Line 44: status
> stats
But the thing is called TInsertPartitionStatus. There's other metadata in 
addition to stats in the hdfs case (e.g. base dir), so I guess that's where the 
terminology "status" came from. I hesitate to say "metadata" to keep it 
distinct from the table metadata concept.



--
To view, visit http://gerrit.cloudera.org:8080/9793
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id4c025917620a7bff2acbeb46464f107ab4b7565
Gerrit-Change-Number: 9793
Gerrit-PatchSet: 6
Gerrit-Owner: Dan Hecht 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Wed, 28 Mar 2018 16:42:10 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-5384, part 1: introduce DmlExecState

[Tim Armstrong (Code Review)]

Tim Armstrong has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9793 )

Change subject: IMPALA-5384, part 1: introduce DmlExecState
..


Patch Set 6: Code-Review+2

(4 comments)

Looks good, should be easier to understand now

http://gerrit.cloudera.org:8080/#/c/9793/4/be/src/runtime/coordinator.cc
File be/src/runtime/coordinator.cc:

http://gerrit.cloudera.org:8080/#/c/9793/4/be/src/runtime/coordinator.cc@a492
PS4, Line 492:
> Thanks for double checking. Yeah, I ended up regenerating the patch from th
That's good to know.


http://gerrit.cloudera.org:8080/#/c/9793/4/be/src/runtime/coordinator.cc@497
PS4, Line 497: HdfsTableDescriptor* hdfs_table;
> Then we'd have to pass in the desc_tbl anyway, right? So, I'd prefer to jus
Seems fine in that case.


http://gerrit.cloudera.org:8080/#/c/9793/6/be/src/runtime/dml-exec-state.h
File be/src/runtime/dml-exec-state.h:

http://gerrit.cloudera.org:8080/#/c/9793/6/be/src/runtime/dml-exec-state.h@44
PS6, Line 44: status
stats


http://gerrit.cloudera.org:8080/#/c/9793/4/be/src/runtime/insert-exec-state.h
File be/src/runtime/insert-exec-state.h:

http://gerrit.cloudera.org:8080/#/c/9793/4/be/src/runtime/insert-exec-state.h@47
PS4, Line 47:
> I renamed this class, but stopped short of trying to restructure the thrift
That seems reasonable, at least that's better encapsulated now.



--
To view, visit http://gerrit.cloudera.org:8080/9793
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id4c025917620a7bff2acbeb46464f107ab4b7565
Gerrit-Change-Number: 9793
Gerrit-PatchSet: 6
Gerrit-Owner: Dan Hecht 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Wed, 28 Mar 2018 16:34:58 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4025: Part 1: Add percentile disc aggregation function

[Vuk Ercegovac (Code Review)]

Vuk Ercegovac has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9777 )

Change subject: IMPALA-4025: Part 1: Add percentile_disc aggregation function
..


Patch Set 1:

(2 comments)

initial comment... still reading.

http://gerrit.cloudera.org:8080/#/c/9777/1//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/9777/1//COMMIT_MSG@17
PS1, Line 17: select min(case when _percentile_row_number_diff_0 >= 0 then
: _percentile_view.col end) from (select row_number() over (order 
by col)
: - 0.5 * count(col) over() _percentile_row_number_diff_0, * from 
tbl)
: _percentile_view
pls format this so its easier to read. for example, the ..._diff_0, * from tbl) 
_percentile_view is difficult to follow/looks like an error.


http://gerrit.cloudera.org:8080/#/c/9777/1/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
File fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java:

http://gerrit.cloudera.org:8080/#/c/9777/1/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java@363
PS1, Line 363: requiresPercentileRewrite()
first thing that got my attention is the way the rewrite for this case is 
handled. it seems fairly special: percentile is one specific type of aggregate 
function yet it has its unique handling here. I don't have a better suggestion 
right now-- looking into this more-- but can you comment on the other ways you 
might have tried this and why they didn't work (e.g., more generic rewrite rule 
approach)



--
To view, visit http://gerrit.cloudera.org:8080/9777
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iacef7b3fcd74c4c73d88400ce27307c3baa0121e
Gerrit-Change-Number: 9777
Gerrit-PatchSet: 1
Gerrit-Owner: Tianyi Wang 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Vuk Ercegovac 
Gerrit-Comment-Date: Wed, 28 Mar 2018 16:18:28 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-5384, part 1: introduce DmlExecState

[Dan Hecht (Code Review)]

Dan Hecht has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9793 )

Change subject: IMPALA-5384, part 1: introduce DmlExecState
..


Patch Set 5:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/9793/5//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/9793/5//COMMIT_MSG@7
PS5, Line 7: InsertExecState
> update
Done


http://gerrit.cloudera.org:8080/#/c/9793/5/be/src/runtime/dml-exec-state.cc
File be/src/runtime/dml-exec-state.cc:

http://gerrit.cloudera.org:8080/#/c/9793/5/be/src/runtime/dml-exec-state.cc@143
PS5, Line 143: FinalizeDml
> I might rename this FinalizeHdsInsert since it's really specific to that ca
Done, along with some related coordinator naming.



--
To view, visit http://gerrit.cloudera.org:8080/9793
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id4c025917620a7bff2acbeb46464f107ab4b7565
Gerrit-Change-Number: 9793
Gerrit-PatchSet: 5
Gerrit-Owner: Dan Hecht 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Wed, 28 Mar 2018 16:15:30 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-5384, part 1: introduce DmlExecState

[Dan Hecht (Code Review)]

Hello Thomas Tauber-Marshall, Tim Armstrong,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/9793

to look at the new patch set (#6).

Change subject: IMPALA-5384, part 1: introduce DmlExecState
..

IMPALA-5384, part 1: introduce DmlExecState

This change is based on a patch by Marcel Kornacker.

Move data structures that collect DML operation stats from the
RuntimeState and Coordinator into a new InsertExecState class, which
has it's own lock.  This removes a dependency on the coordinator's
lock, which will allow further coordinator locking cleanup in the next
patch.

Change-Id: Id4c025917620a7bff2acbeb46464f107ab4b7565
---
M be/src/benchmarks/expr-benchmark.cc
M be/src/exec/catalog-op-executor.cc
M be/src/exec/data-sink.cc
M be/src/exec/data-sink.h
M be/src/exec/hbase-table-sink.cc
M be/src/exec/hdfs-table-sink.cc
M be/src/exec/kudu-table-sink.cc
M be/src/exec/plan-root-sink.cc
M be/src/runtime/CMakeLists.txt
M be/src/runtime/coordinator-backend-state.cc
M be/src/runtime/coordinator-backend-state.h
M be/src/runtime/coordinator.cc
M be/src/runtime/coordinator.h
A be/src/runtime/dml-exec-state.cc
A be/src/runtime/dml-exec-state.h
M be/src/runtime/query-state.cc
M be/src/runtime/runtime-filter-bank.cc
M be/src/runtime/runtime-state.h
M be/src/service/client-request-state.cc
M be/src/service/client-request-state.h
M be/src/service/impala-beeswax-server.cc
M be/src/service/impala-hs2-server.cc
M be/src/service/impala-http-handler.cc
M be/src/service/impala-server.cc
M be/src/service/impala-server.h
M be/src/testutil/in-process-servers.cc
26 files changed, 750 insertions(+), 595 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/93/9793/6
--
To view, visit http://gerrit.cloudera.org:8080/9793
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id4c025917620a7bff2acbeb46464f107ab4b7565
Gerrit-Change-Number: 9793
Gerrit-PatchSet: 6
Gerrit-Owner: Dan Hecht 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Reviewer: Tim Armstrong 

[Impala-ASF-CR] IMPALA-4168: [DOCS] Adds Oracle-style hint placement for INSERT/UPSERT

[Alex Rodoni (Code Review)]

Alex Rodoni has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9030 )

Change subject: IMPALA-4168: [DOCS] Adds Oracle-style hint placement for 
INSERT/UPSERT
..


Patch Set 2:

Thank you @Jin Chul!

Could someone review the content and give +2?


--
To view, visit http://gerrit.cloudera.org:8080/9030
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I43e0a782087c2e67f2e012424fb9261be445efc9
Gerrit-Change-Number: 9030
Gerrit-PatchSet: 2
Gerrit-Owner: Kim Jin Chul 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Alex Rodoni 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: John Russell 
Gerrit-Reviewer: Kim Jin Chul 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Vuk Ercegovac 
Gerrit-Comment-Date: Wed, 28 Mar 2018 16:12:31 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6587: free buffers before ScanRange::Cancel() returns

[Dan Hecht (Code Review)]

Dan Hecht has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9680 )

Change subject: IMPALA-6587: free buffers before ScanRange::Cancel() returns
..


Patch Set 12: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/9680
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I87182b6bd51b5fb0b923e7e4c8d08a44e7617db2
Gerrit-Change-Number: 9680
Gerrit-PatchSet: 12
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Wed, 28 Mar 2018 16:07:53 +
Gerrit-HasComments: No