[Impala-ASF-CR] IMPALA-4924 addendum: Change result type to decimal in a TPCH query

2018-02-02 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9208 )

Change subject: IMPALA-4924 addendum: Change result type to decimal in a TPCH 
query
..


Patch Set 2: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/9208
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I46fc038d40969547622707ce77a037494f0ed0a9
Gerrit-Change-Number: 9208
Gerrit-PatchSet: 2
Gerrit-Owner: Taras Bobrovytsky 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Taras Bobrovytsky 
Gerrit-Comment-Date: Sat, 03 Feb 2018 05:22:23 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-4924 addendum: Change result type to decimal in a TPCH query

2018-02-02 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/9208 )

Change subject: IMPALA-4924 addendum: Change result type to decimal in a TPCH 
query
..

IMPALA-4924 addendum: Change result type to decimal in a TPCH query

Change the expected result type of Kudu TPCH Q17 to Decimal because
DECIMAL_V2 is now enabled by default. This was not done earlier because
we were not running TPCH on Kudu regularly.

Cherry-picks: not for 2.x

Change-Id: I46fc038d40969547622707ce77a037494f0ed0a9
Reviewed-on: http://gerrit.cloudera.org:8080/9208
Reviewed-by: Taras Bobrovytsky 
Tested-by: Impala Public Jenkins
---
M testdata/workloads/tpch/queries/tpch-kudu-q17.test
1 file changed, 2 insertions(+), 2 deletions(-)

Approvals:
  Taras Bobrovytsky: Looks good to me, approved
  Impala Public Jenkins: Verified

--
To view, visit http://gerrit.cloudera.org:8080/9208
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I46fc038d40969547622707ce77a037494f0ed0a9
Gerrit-Change-Number: 9208
Gerrit-PatchSet: 3
Gerrit-Owner: Taras Bobrovytsky 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Taras Bobrovytsky 


[Impala-ASF-CR] IMPALA-6476: Skip krpc mem test that uses stress options

2018-02-02 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9207 )

Change subject: IMPALA-6476: Skip krpc mem test that uses stress options
..


Patch Set 1: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/9207
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I89ae25ee8c1aca3833c2d98e902ddaad2dd01aad
Gerrit-Change-Number: 9207
Gerrit-PatchSet: 1
Gerrit-Owner: Lars Volker 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Sat, 03 Feb 2018 05:10:57 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-6476: Skip krpc mem test that uses stress options

2018-02-02 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/9207 )

Change subject: IMPALA-6476: Skip krpc mem test that uses stress options
..

IMPALA-6476: Skip krpc mem test that uses stress options

Some of the tests added in IMPALA-6193 rely on flags that are only
compiled for debug binaries. This change marks those tests as debug-only
so that they do not break the release tests.

Change-Id: I89ae25ee8c1aca3833c2d98e902ddaad2dd01aad
Reviewed-on: http://gerrit.cloudera.org:8080/9207
Reviewed-by: Alex Behm 
Tested-by: Impala Public Jenkins
---
M tests/custom_cluster/test_krpc_mem_usage.py
1 file changed, 2 insertions(+), 0 deletions(-)

Approvals:
  Alex Behm: Looks good to me, approved
  Impala Public Jenkins: Verified

--
To view, visit http://gerrit.cloudera.org:8080/9207
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I89ae25ee8c1aca3833c2d98e902ddaad2dd01aad
Gerrit-Change-Number: 9207
Gerrit-PatchSet: 2
Gerrit-Owner: Lars Volker 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Impala Public Jenkins


[Impala-ASF-CR] IMPALA-6475: Enable running TPCH on Kudu

2018-02-02 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9206 )

Change subject: IMPALA-6475: Enable running TPCH on Kudu
..


Patch Set 1: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/1870/


--
To view, visit http://gerrit.cloudera.org:8080/9206
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I88b66f5db105694b3bcf33360887265996f9059c
Gerrit-Change-Number: 9206
Gerrit-PatchSet: 1
Gerrit-Owner: Taras Bobrovytsky 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Brown 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Comment-Date: Sat, 03 Feb 2018 04:40:35 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-5269: Fix issue with final line of query followed by a comment

2018-02-02 Thread Fredy Wijaya (Code Review)
Fredy Wijaya has uploaded a new patch set (#4). ( 
http://gerrit.cloudera.org:8080/9191 )

Change subject: IMPALA-5269: Fix issue with final line of query followed by a 
comment
..

IMPALA-5269: Fix issue with final line of query followed by a comment

The patch is to remove any comments in a statement when checking if a
statement ends with a semicolon delimiter.

For example:

Before (semicolon delimiter is needed at the end):
select 1 + 1; -- comment\n;

After (semicolon delimiter is no longer needed):
select 1 + 1; -- comment

Change-Id: I54f9a8f65214023520eaa010fc462a663d02d258
---
M shell/impala_shell.py
M tests/shell/test_shell_interactive.py
2 files changed, 29 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/91/9191/4
--
To view, visit http://gerrit.cloudera.org:8080/9191
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I54f9a8f65214023520eaa010fc462a663d02d258
Gerrit-Change-Number: 9191
Gerrit-PatchSet: 4
Gerrit-Owner: Fredy Wijaya 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Thomas Tauber-Marshall 


[Impala-ASF-CR] IMPALA-6475: Enable running TPCH on Kudu

2018-02-02 Thread Michael Brown (Code Review)
Michael Brown has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9206 )

Change subject: IMPALA-6475: Enable running TPCH on Kudu
..


Patch Set 1: Code-Review+2

Yeah, this is basically the solution path I had started to go down when Alex 
had chatted with me about this.


--
To view, visit http://gerrit.cloudera.org:8080/9206
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I88b66f5db105694b3bcf33360887265996f9059c
Gerrit-Change-Number: 9206
Gerrit-PatchSet: 1
Gerrit-Owner: Taras Bobrovytsky 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Brown 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Comment-Date: Sat, 03 Feb 2018 03:57:34 +
Gerrit-HasComments: No


[Impala-ASF-CR](2.x) IMPALA-3916: Reserve SQL:2016 reserved words

2018-02-02 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9198 )

Change subject: IMPALA-3916: Reserve SQL:2016 reserved words
..


Patch Set 2:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1875/


--
To view, visit http://gerrit.cloudera.org:8080/9198
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: 2.x
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifd29d6aebe793d4eb37c092c1364f3acfebd91f8
Gerrit-Change-Number: 9198
Gerrit-PatchSet: 2
Gerrit-Owner: Tianyi Wang 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Tianyi Wang 
Gerrit-Comment-Date: Sat, 03 Feb 2018 03:49:52 +
Gerrit-HasComments: No


[Impala-ASF-CR](2.x) IMPALA-3916: Reserve SQL:2016 reserved words

2018-02-02 Thread Tianyi Wang (Code Review)
Tianyi Wang has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9198 )

Change subject: IMPALA-3916: Reserve SQL:2016 reserved words
..


Patch Set 2:

oops the regex_escape change is already in this branch.


--
To view, visit http://gerrit.cloudera.org:8080/9198
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: 2.x
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifd29d6aebe793d4eb37c092c1364f3acfebd91f8
Gerrit-Change-Number: 9198
Gerrit-PatchSet: 2
Gerrit-Owner: Tianyi Wang 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Tianyi Wang 
Gerrit-Comment-Date: Sat, 03 Feb 2018 03:49:28 +
Gerrit-HasComments: No


[Impala-ASF-CR](2.x) IMPALA-3916: Reserve SQL:2016 reserved words

2018-02-02 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9198 )

Change subject: IMPALA-3916: Reserve SQL:2016 reserved words
..


Patch Set 2: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/1869/


--
To view, visit http://gerrit.cloudera.org:8080/9198
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: 2.x
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifd29d6aebe793d4eb37c092c1364f3acfebd91f8
Gerrit-Change-Number: 9198
Gerrit-PatchSet: 2
Gerrit-Owner: Tianyi Wang 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Tianyi Wang 
Gerrit-Comment-Date: Sat, 03 Feb 2018 03:33:46 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-6113: Skip row groups with predicates on NULL columns

2018-02-02 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/9140 )

Change subject: IMPALA-6113: Skip row groups with predicates on NULL columns
..

IMPALA-6113: Skip row groups with predicates on NULL columns

Based on the existing Parquet column chunk level statistics null_count,
Impala's Parquet scanner is enhanced to skip an entire row group if the
null_count statistics indicate that all the values under the predicated
column are NULL as we wouldn't get any result rows from that row group
anyway.

Change-Id: I141317af0e0df30da8f220b29b0bfba364f40ddf
Reviewed-on: http://gerrit.cloudera.org:8080/9140
Reviewed-by: Tim Armstrong 
Tested-by: Impala Public Jenkins
---
M be/src/exec/hdfs-parquet-scanner.cc
M be/src/exec/parquet-column-stats.cc
M be/src/exec/parquet-column-stats.h
M testdata/workloads/functional-query/queries/QueryTest/parquet-stats.test
4 files changed, 59 insertions(+), 0 deletions(-)

Approvals:
  Tim Armstrong: Looks good to me, approved
  Impala Public Jenkins: Verified

--
To view, visit http://gerrit.cloudera.org:8080/9140
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I141317af0e0df30da8f220b29b0bfba364f40ddf
Gerrit-Change-Number: 9140
Gerrit-PatchSet: 9
Gerrit-Owner: Gabor Kaszab 
Gerrit-Reviewer: Gabor Kaszab 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: anujphadke 


[Impala-ASF-CR] IMPALA-6113: Skip row groups with predicates on NULL columns

2018-02-02 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9140 )

Change subject: IMPALA-6113: Skip row groups with predicates on NULL columns
..


Patch Set 8: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/9140
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I141317af0e0df30da8f220b29b0bfba364f40ddf
Gerrit-Change-Number: 9140
Gerrit-PatchSet: 8
Gerrit-Owner: Gabor Kaszab 
Gerrit-Reviewer: Gabor Kaszab 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: anujphadke 
Gerrit-Comment-Date: Sat, 03 Feb 2018 03:24:36 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-5293: Turn insert clustering on by default

2018-02-02 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9153 )

Change subject: IMPALA-5293: Turn insert clustering on by default
..


Patch Set 11:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1874/


--
To view, visit http://gerrit.cloudera.org:8080/9153
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idbf2368cf4415e6ecfa65058daf6ff87ef62f9d9
Gerrit-Change-Number: 9153
Gerrit-PatchSet: 11
Gerrit-Owner: Lars Volker 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Sat, 03 Feb 2018 02:11:02 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-5293: Turn insert clustering on by default

2018-02-02 Thread Lars Volker (Code Review)
Lars Volker has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9153 )

Change subject: IMPALA-5293: Turn insert clustering on by default
..


Patch Set 11: Code-Review+2

Hit IMPALA-6472, rebased, carrying Alex's +2.


--
To view, visit http://gerrit.cloudera.org:8080/9153
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idbf2368cf4415e6ecfa65058daf6ff87ef62f9d9
Gerrit-Change-Number: 9153
Gerrit-PatchSet: 11
Gerrit-Owner: Lars Volker 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Sat, 03 Feb 2018 02:10:57 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-4924 addendum: Change result type to decimal in a TPCH query

2018-02-02 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9208 )

Change subject: IMPALA-4924 addendum: Change result type to decimal in a TPCH 
query
..


Patch Set 2:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1873/


--
To view, visit http://gerrit.cloudera.org:8080/9208
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I46fc038d40969547622707ce77a037494f0ed0a9
Gerrit-Change-Number: 9208
Gerrit-PatchSet: 2
Gerrit-Owner: Taras Bobrovytsky 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Taras Bobrovytsky 
Gerrit-Comment-Date: Sat, 03 Feb 2018 01:42:47 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-4924 addendum: Change result type to decimal in a TPCH query

2018-02-02 Thread Taras Bobrovytsky (Code Review)
Taras Bobrovytsky has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9208 )

Change subject: IMPALA-4924 addendum: Change result type to decimal in a TPCH 
query
..


Patch Set 2: Code-Review+2

For some reason Change-Id was placed before "Cherry-picks" in the commit 
message. Fixed. Forwarding the +2


--
To view, visit http://gerrit.cloudera.org:8080/9208
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I46fc038d40969547622707ce77a037494f0ed0a9
Gerrit-Change-Number: 9208
Gerrit-PatchSet: 2
Gerrit-Owner: Taras Bobrovytsky 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Taras Bobrovytsky 
Gerrit-Comment-Date: Sat, 03 Feb 2018 01:42:26 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-5293: Turn insert clustering on by default

2018-02-02 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9153 )

Change subject: IMPALA-5293: Turn insert clustering on by default
..


Patch Set 10: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/1866/


--
To view, visit http://gerrit.cloudera.org:8080/9153
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idbf2368cf4415e6ecfa65058daf6ff87ef62f9d9
Gerrit-Change-Number: 9153
Gerrit-PatchSet: 10
Gerrit-Owner: Lars Volker 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Sat, 03 Feb 2018 01:41:36 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-4924 addendum: Change result type to decimal in a TPCH query

2018-02-02 Thread Taras Bobrovytsky (Code Review)
Taras Bobrovytsky has uploaded a new patch set (#2). ( 
http://gerrit.cloudera.org:8080/9208 )

Change subject: IMPALA-4924 addendum: Change result type to decimal in a TPCH 
query
..

IMPALA-4924 addendum: Change result type to decimal in a TPCH query

Change the expected result type of Kudu TPCH Q17 to Decimal because
DECIMAL_V2 is now enabled by default. This was not done earlier because
we were not running TPCH on Kudu regularly.

Cherry-picks: not for 2.x

Change-Id: I46fc038d40969547622707ce77a037494f0ed0a9
---
M testdata/workloads/tpch/queries/tpch-kudu-q17.test
1 file changed, 2 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/08/9208/2
--
To view, visit http://gerrit.cloudera.org:8080/9208
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I46fc038d40969547622707ce77a037494f0ed0a9
Gerrit-Change-Number: 9208
Gerrit-PatchSet: 2
Gerrit-Owner: Taras Bobrovytsky 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Impala Public Jenkins


[Impala-ASF-CR] IMPALA-4924 addendum: Change result type to decimal in a TPCH query

2018-02-02 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9208 )

Change subject: IMPALA-4924 addendum: Change result type to decimal in a TPCH 
query
..


Patch Set 1:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1872/


--
To view, visit http://gerrit.cloudera.org:8080/9208
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I46fc038d40969547622707ce77a037494f0ed0a9
Gerrit-Change-Number: 9208
Gerrit-PatchSet: 1
Gerrit-Owner: Taras Bobrovytsky 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Sat, 03 Feb 2018 01:35:48 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-4924 addendum: Change result type to decimal in a TPCH query

2018-02-02 Thread Alex Behm (Code Review)
Alex Behm has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9208 )

Change subject: IMPALA-4924 addendum: Change result type to decimal in a TPCH 
query
..


Patch Set 1: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/9208
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I46fc038d40969547622707ce77a037494f0ed0a9
Gerrit-Change-Number: 9208
Gerrit-PatchSet: 1
Gerrit-Owner: Taras Bobrovytsky 
Gerrit-Reviewer: Alex Behm 
Gerrit-Comment-Date: Sat, 03 Feb 2018 01:34:34 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-6476: Skip krpc mem test that uses stress options

2018-02-02 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9207 )

Change subject: IMPALA-6476: Skip krpc mem test that uses stress options
..


Patch Set 1:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1871/


--
To view, visit http://gerrit.cloudera.org:8080/9207
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I89ae25ee8c1aca3833c2d98e902ddaad2dd01aad
Gerrit-Change-Number: 9207
Gerrit-PatchSet: 1
Gerrit-Owner: Lars Volker 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Sat, 03 Feb 2018 01:32:33 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-4924 addendum: Change result type to decimal in a TPCH query

2018-02-02 Thread Taras Bobrovytsky (Code Review)
Taras Bobrovytsky has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/9208


Change subject: IMPALA-4924 addendum: Change result type to decimal in a TPCH 
query
..

IMPALA-4924 addendum: Change result type to decimal in a TPCH query

Change the expected result type of Kudu TPCH Q17 to Decimal because
DECIMAL_V2 is now enabled by default. This was not done earlier because
we were not running TPCH on Kudu regularly.

Change-Id: I46fc038d40969547622707ce77a037494f0ed0a9
Cherry-picks: not for 2.x
---
M testdata/workloads/tpch/queries/tpch-kudu-q17.test
1 file changed, 2 insertions(+), 2 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/08/9208/1
--
To view, visit http://gerrit.cloudera.org:8080/9208
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I46fc038d40969547622707ce77a037494f0ed0a9
Gerrit-Change-Number: 9208
Gerrit-PatchSet: 1
Gerrit-Owner: Taras Bobrovytsky 


[Impala-ASF-CR] IMPALA-6448: Re-enable kerberized testing with KRPC

2018-02-02 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9164 )

Change subject: IMPALA-6448: Re-enable kerberized testing with KRPC
..


Patch Set 3: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/1865/


--
To view, visit http://gerrit.cloudera.org:8080/9164
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6412978316de90875c98f8fbe51c8d215c227b18
Gerrit-Change-Number: 9164
Gerrit-PatchSet: 3
Gerrit-Owner: Sailesh Mukil 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Comment-Date: Sat, 03 Feb 2018 01:21:09 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-6475: Enable running TPCH on Kudu

2018-02-02 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9206 )

Change subject: IMPALA-6475: Enable running TPCH on Kudu
..


Patch Set 1:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1870/


--
To view, visit http://gerrit.cloudera.org:8080/9206
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I88b66f5db105694b3bcf33360887265996f9059c
Gerrit-Change-Number: 9206
Gerrit-PatchSet: 1
Gerrit-Owner: Taras Bobrovytsky 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Brown 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Comment-Date: Sat, 03 Feb 2018 00:53:45 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-6475: Enable running TPCH on Kudu

2018-02-02 Thread Alex Behm (Code Review)
Alex Behm has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9206 )

Change subject: IMPALA-6475: Enable running TPCH on Kudu
..


Patch Set 1: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/9206
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I88b66f5db105694b3bcf33360887265996f9059c
Gerrit-Change-Number: 9206
Gerrit-PatchSet: 1
Gerrit-Owner: Taras Bobrovytsky 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Michael Brown 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Comment-Date: Sat, 03 Feb 2018 00:50:42 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-5269: Fix issue with final line of query followed by a comment

2018-02-02 Thread Thomas Tauber-Marshall (Code Review)
Thomas Tauber-Marshall has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9191 )

Change subject: IMPALA-5269: Fix issue with final line of query followed by a 
comment
..


Patch Set 3:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/9191/3/shell/impala_shell.py
File shell/impala_shell.py:

http://gerrit.cloudera.org:8080/#/c/9191/3/shell/impala_shell.py@394
PS3, Line 394: statements = sqlparse.parse(line)
I'm concerned that this is a fairly heavy-weight operation to do for every 
query. Have you done any performance tests to see how much time this takes, 
particularly with a large query (eg. something from tpcds)?

Maybe sqlparse.format(line, strip_comments=True) would be work? I would expect 
it to be faster (you can check this) and it would also save you having to do 
the logic below where you check tokens.



--
To view, visit http://gerrit.cloudera.org:8080/9191
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I54f9a8f65214023520eaa010fc462a663d02d258
Gerrit-Change-Number: 9191
Gerrit-PatchSet: 3
Gerrit-Owner: Fredy Wijaya 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Comment-Date: Sat, 03 Feb 2018 00:50:15 +
Gerrit-HasComments: Yes


[Impala-ASF-CR] IMPALA-6475: Enable running TPCH on Kudu

2018-02-02 Thread Taras Bobrovytsky (Code Review)
Taras Bobrovytsky has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/9206


Change subject: IMPALA-6475: Enable running TPCH on Kudu
..

IMPALA-6475: Enable running TPCH on Kudu

Change-Id: I88b66f5db105694b3bcf33360887265996f9059c
---
M testdata/workloads/tpch/tpch_core.csv
M testdata/workloads/tpch/tpch_exhaustive.csv
M testdata/workloads/tpch/tpch_pairwise.csv
3 files changed, 3 insertions(+), 0 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/06/9206/1
--
To view, visit http://gerrit.cloudera.org:8080/9206
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I88b66f5db105694b3bcf33360887265996f9059c
Gerrit-Change-Number: 9206
Gerrit-PatchSet: 1
Gerrit-Owner: Taras Bobrovytsky 


[Impala-ASF-CR] IMPALA-5519: Allocate fragment's runtime filter memory from Buffer pool

2018-02-02 Thread Bikramjeet Vig (Code Review)
Bikramjeet Vig has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/8971 )

Change subject: IMPALA-5519: Allocate fragment's runtime filter memory from 
Buffer pool
..


Patch Set 9:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/8971/9/be/src/service/query-options.cc
File be/src/service/query-options.cc:

http://gerrit.cloudera.org:8080/#/c/8971/9/be/src/service/query-options.cc@376
PS9, Line 376: // last condition is to unblock the highly improbable case where 
the
 : // min_buffer_size is greater than 
RuntimeFilterBank::MAX_BLOOM_FILTER_SIZE.
 : && FLAGS_min_buffer_size <= 
RuntimeFilterBank::MAX_BLOOM_FILTER_SIZE
pretty ugly workaround, but had to put this so it doesn't block the query for 
this highly improbable case.
Any comments?



--
To view, visit http://gerrit.cloudera.org:8080/8971
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea2759665fb2e8bef9433014a8d42a7ebf99ce1f
Gerrit-Change-Number: 8971
Gerrit-PatchSet: 9
Gerrit-Owner: Bikramjeet Vig 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Sat, 03 Feb 2018 00:21:22 +
Gerrit-HasComments: Yes


[Impala-ASF-CR] IMPALA-5519: Allocate fragment's runtime filter memory from Buffer pool

2018-02-02 Thread Bikramjeet Vig (Code Review)
Bikramjeet Vig has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/8971 )

Change subject: IMPALA-5519: Allocate fragment's runtime filter memory from 
Buffer pool
..


Patch Set 8:

(5 comments)

> (5 comments)
 >
 > Did you look at the tests that Alex mentioned for creating a test
 > table with different cardinalities and nvds?

initially we discussed including those tests to check if the limits are being 
enforced in case the size calculated from ndvs is outside those limits. It 
turns out that we infact are already testing that code path when we check if 
the query limits are enforced (in bloom_filters.test). Also, since we are 
taking care of enforcing the hard limits({MIN,MAX}_BLOOM_FILTER_SIZE) on the 
query options itself, we dont need to write any tests with mock table stats 
anymore.

http://gerrit.cloudera.org:8080/#/c/8971/8/fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java
File fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java:

http://gerrit.cloudera.org:8080/#/c/8971/8/fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java@95
PS8, Line 95:   private static final long MIN_BLOOM_FILTER_SIZE = 4 * 1024;
:   private static final long MAX_BLOOM_FILTER_SIZE = 512 * 1024 * 
1024;
> You may remove them and add a check instead.
Done


http://gerrit.cloudera.org:8080/#/c/8971/8/fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java@109
PS8, Line 109: _
> nit: no need for '_' when the fields are public
Done


http://gerrit.cloudera.org:8080/#/c/8971/8/fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java@125
PS8, Line 125: bloomFilterSizeLimits_.max_ = 
tQueryOptions.getRuntime_filter_max_size();
 : bloomFilterSizeLimits_.max_ = 
Math.max(bloomFilterSizeLimits_.max_,
 : MIN_BLOOM_FILTER_SIZE);
 : bloomFilterSizeLimits_.max_ = 
Math.max(bloomFilterSizeLimits_.max_,
 : BackendConfig.INSTANCE.getMinBufferSize());
 : bloomFilterSizeLimits_.max_ = Math.min(
 : BitUtil.roundUpToPowerOf2(bloomFilterSizeLimits_.max_), 
MAX_BLOOM_FILTER_SIZE);
> You may want to see if it makes sense to add this logic to FilterSizeLimits
I moved this whole part into the FilterSizeLimits ctor, since i need to retain 
the order in which these are set.


http://gerrit.cloudera.org:8080/#/c/8971/8/fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java@482
PS8, Line 482: sizeLimits
> 'filterSizeLimits'
Done


http://gerrit.cloudera.org:8080/#/c/8971/8/fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java@494
PS8, Line 494:   filterSizeBytes_ = Math.min(filterSizeBytes_, 
filterSizeLimits.max_);
> There is always the case that the size needed to achieve the desired fp rat
I would not recommend logging it here because although this is a valid case, it 
still does not guarantee that the filter will be disabled, as the disabling 
logic uses the actual ndv to check for fpp during query execution. Also, there 
might be a lot of runtime filters generated here but might not end up in the 
plan eventually, so the log might be filled with extra messages.



--
To view, visit http://gerrit.cloudera.org:8080/8971
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea2759665fb2e8bef9433014a8d42a7ebf99ce1f
Gerrit-Change-Number: 8971
Gerrit-PatchSet: 8
Gerrit-Owner: Bikramjeet Vig 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Sat, 03 Feb 2018 00:18:46 +
Gerrit-HasComments: Yes


[Impala-ASF-CR] IMPALA-5519: Allocate fragment's runtime filter memory from Buffer pool

2018-02-02 Thread Bikramjeet Vig (Code Review)
Hello Dimitris Tsirogiannis, Tim Armstrong,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/8971

to look at the new patch set (#9).

Change subject: IMPALA-5519: Allocate fragment's runtime filter memory from 
Buffer pool
..

IMPALA-5519: Allocate fragment's runtime filter memory from Buffer pool

This patch adds changes to the planner to account for memory used by
bloom filters at the fragment instance level. Also adds changes to
allocate memory for those bloom filters from the buffer pool.

Testing:
- Modified Planner Tests and end to end tests to account for memory
  reservation for the runtime filters.
- Modified backend tests and benchmarks to use the bufferpool for
  bloom filter allocation.
- Add an end to end test.
- Ran rest of the core tests.

Change-Id: Iea2759665fb2e8bef9433014a8d42a7ebf99ce1f
---
M be/src/benchmarks/bloom-filter-benchmark.cc
M be/src/runtime/fragment-instance-state.cc
M be/src/runtime/runtime-filter-bank.cc
M be/src/runtime/runtime-filter-bank.h
M be/src/runtime/runtime-filter.h
M be/src/runtime/runtime-state.cc
M be/src/runtime/runtime-state.h
M be/src/service/fe-support.cc
M be/src/service/query-options-test.cc
M be/src/service/query-options.cc
M be/src/util/backend-gflag-util.cc
M be/src/util/bloom-filter-test.cc
M be/src/util/bloom-filter.cc
M be/src/util/bloom-filter.h
M common/thrift/BackendGflags.thrift
M common/thrift/ImpalaInternalService.thrift
M common/thrift/PlanNodes.thrift
M common/thrift/Planner.thrift
M fe/src/main/java/org/apache/impala/planner/PlanFragment.java
M fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/FeSupport.java
M testdata/workloads/functional-planner/queries/PlannerTest/disable-codegen.test
M 
testdata/workloads/functional-planner/queries/PlannerTest/fk-pk-join-detection.test
M testdata/workloads/functional-planner/queries/PlannerTest/max-row-size.test
M 
testdata/workloads/functional-planner/queries/PlannerTest/min-max-runtime-filters.test
M 
testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test
M 
testdata/workloads/functional-planner/queries/PlannerTest/spillable-buffer-sizing.test
M testdata/workloads/functional-planner/queries/PlannerTest/tablesample.test
M 
testdata/workloads/functional-query/queries/QueryTest/admission-reject-min-reservation.test
M testdata/workloads/functional-query/queries/QueryTest/bloom_filters.test
M testdata/workloads/functional-query/queries/QueryTest/bloom_filters_wait.test
M testdata/workloads/functional-query/queries/QueryTest/explain-level0.test
M testdata/workloads/functional-query/queries/QueryTest/explain-level1.test
M testdata/workloads/functional-query/queries/QueryTest/explain-level2.test
M testdata/workloads/functional-query/queries/QueryTest/explain-level3.test
M testdata/workloads/functional-query/queries/QueryTest/runtime_row_filters.test
M testdata/workloads/functional-query/queries/QueryTest/spilling.test
38 files changed, 906 insertions(+), 549 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/8971/9
--
To view, visit http://gerrit.cloudera.org:8080/8971
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iea2759665fb2e8bef9433014a8d42a7ebf99ce1f
Gerrit-Change-Number: 8971
Gerrit-PatchSet: 9
Gerrit-Owner: Bikramjeet Vig 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Tim Armstrong 


[Impala-ASF-CR] IMPALA-6228: Control stats extrapolation via tbl prop.

2018-02-02 Thread Alex Behm (Code Review)
Alex Behm has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9139 )

Change subject: IMPALA-6228: Control stats extrapolation via tbl prop.
..


Patch Set 4: Code-Review+2

Rebase and resolve conflicts


--
To view, visit http://gerrit.cloudera.org:8080/9139
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie49597bf1b93b7572106abc620d91f199cba0cfd
Gerrit-Change-Number: 9139
Gerrit-PatchSet: 4
Gerrit-Owner: Alex Behm 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Vuk Ercegovac 
Gerrit-Comment-Date: Fri, 02 Feb 2018 23:57:25 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-6228: Control stats extrapolation via tbl prop.

2018-02-02 Thread Alex Behm (Code Review)
Hello Vuk Ercegovac,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/9139

to look at the new patch set (#4).

Change subject: IMPALA-6228: Control stats extrapolation via tbl prop.
..

IMPALA-6228: Control stats extrapolation via tbl prop.

Introduces a new TBLPROPERTY for controlling stats
extrapolation on a per-table basis:

impala.enable.stats.extrapolation=true/false

The property key was chosen to be consistent with
the impalad startup flag --enable_stats_extrapolation
and to indicate that the property was set and is used
by Impala.

Behavior:
- If the property is not set, then the extrapolation
  behavior is determined by the impalad startup flag.
- If the property is set, it overrides the impalad
  startup flag, i.e., extrapolation can be explicitly
  enabled or disabled regardless of the startup flag.

Testing:
- added new unit tests
- code/hdfs run passed

Change-Id: Ie49597bf1b93b7572106abc620d91f199cba0cfd
---
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
M fe/src/test/java/org/apache/impala/planner/StatsExtrapolationTest.java
M testdata/workloads/functional-query/queries/QueryTest/stats-extrapolation.test
M tests/custom_cluster/test_stats_extrapolation.py
A tests/metadata/test_stats_extrapolation.py
9 files changed, 329 insertions(+), 278 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/39/9139/4
--
To view, visit http://gerrit.cloudera.org:8080/9139
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie49597bf1b93b7572106abc620d91f199cba0cfd
Gerrit-Change-Number: 9139
Gerrit-PatchSet: 4
Gerrit-Owner: Alex Behm 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Vuk Ercegovac 


[Impala-ASF-CR](2.x) IMPALA-3916: Reserve SQL:2016 reserved words

2018-02-02 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9198 )

Change subject: IMPALA-3916: Reserve SQL:2016 reserved words
..


Patch Set 2:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1869/


--
To view, visit http://gerrit.cloudera.org:8080/9198
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: 2.x
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifd29d6aebe793d4eb37c092c1364f3acfebd91f8
Gerrit-Change-Number: 9198
Gerrit-PatchSet: 2
Gerrit-Owner: Tianyi Wang 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Tianyi Wang 
Gerrit-Comment-Date: Fri, 02 Feb 2018 23:53:18 +
Gerrit-HasComments: No


[Impala-ASF-CR](2.x) IMPALA-3916: Reserve SQL:2016 reserved words

2018-02-02 Thread Tianyi Wang (Code Review)
Tianyi Wang has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9198 )

Change subject: IMPALA-3916: Reserve SQL:2016 reserved words
..


Patch Set 2: Code-Review+2

Carry over Philip's +2


--
To view, visit http://gerrit.cloudera.org:8080/9198
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: 2.x
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifd29d6aebe793d4eb37c092c1364f3acfebd91f8
Gerrit-Change-Number: 9198
Gerrit-PatchSet: 2
Gerrit-Owner: Tianyi Wang 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Tianyi Wang 
Gerrit-Comment-Date: Fri, 02 Feb 2018 23:52:47 +
Gerrit-HasComments: No


[Impala-ASF-CR](2.x) IMPALA-3916: Change reserved words version to 2.11.0

2018-02-02 Thread Alex Behm (Code Review)
Alex Behm has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9201 )

Change subject: IMPALA-3916: Change reserved_words_version to 2.11.0
..


Patch Set 1: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/9201
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: 2.x
Gerrit-MessageType: comment
Gerrit-Change-Id: I5a3e5eac8c09de4517beccf79dcea111264668c2
Gerrit-Change-Number: 9201
Gerrit-PatchSet: 1
Gerrit-Owner: Tianyi Wang 
Gerrit-Reviewer: Alex Behm 
Gerrit-Comment-Date: Fri, 02 Feb 2018 23:51:24 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-5269: Fix issue with final line of query followed by a comment

2018-02-02 Thread Fredy Wijaya (Code Review)
Fredy Wijaya has uploaded a new patch set (#3). ( 
http://gerrit.cloudera.org:8080/9191 )

Change subject: IMPALA-5269: Fix issue with final line of query followed by a 
comment
..

IMPALA-5269: Fix issue with final line of query followed by a comment

The patch is to fix a statement that is terminated with SQL terminated
delimiter, i.e. ; followed by a comment.

For example:

Before:
select 1 + 1; -- comment\n;

After:
select 1 + 1; -- comment

Change-Id: I54f9a8f65214023520eaa010fc462a663d02d258
---
M shell/impala_shell.py
M tests/shell/test_shell_interactive.py
2 files changed, 39 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/91/9191/3
--
To view, visit http://gerrit.cloudera.org:8080/9191
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I54f9a8f65214023520eaa010fc462a663d02d258
Gerrit-Change-Number: 9191
Gerrit-PatchSet: 3
Gerrit-Owner: Fredy Wijaya 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Thomas Tauber-Marshall 


[Impala-ASF-CR] IMPALA-6337: Fix infinite loop in Impala shell

2018-02-02 Thread Fredy Wijaya (Code Review)
Fredy Wijaya has uploaded a new patch set (#2). ( 
http://gerrit.cloudera.org:8080/9195 )

Change subject: IMPALA-6337: Fix infinite loop in Impala shell
..

IMPALA-6337: Fix infinite loop in Impala shell

The patch uses a workaround for a bug in the sqlparse.split() function
by joining the statements that contain error tokens into a single
statement

Change-Id: I9142f21a888189d351f00ce09baeba123bc0959b
---
M shell/impala_shell.py
M tests/shell/test_shell_interactive.py
2 files changed, 42 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/95/9195/2
--
To view, visit http://gerrit.cloudera.org:8080/9195
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9142f21a888189d351f00ce09baeba123bc0959b
Gerrit-Change-Number: 9195
Gerrit-PatchSet: 2
Gerrit-Owner: Fredy Wijaya 


[Impala-ASF-CR] IMPALA-6456: Add flags to configure rpc negotiation timeout ms and negotiation thread count in KRPC

2018-02-02 Thread Sailesh Mukil (Code Review)
Sailesh Mukil has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9186 )

Change subject: IMPALA-6456: Add flags to configure rpc_negotiation_timeout_ms 
and negotiation thread count in KRPC
..


Patch Set 1:

(4 comments)

http://gerrit.cloudera.org:8080/#/c/9186/1/be/src/rpc/rpc-mgr.cc
File be/src/rpc/rpc-mgr.cc:

http://gerrit.cloudera.org:8080/#/c/9186/1/be/src/rpc/rpc-mgr.cc@70
PS1, Line 70: Number of threads to dedicate to process connection negotiations
> Maximum number of threads dedicated to handling RPC connection negotiations
Done


http://gerrit.cloudera.org:8080/#/c/9186/1/be/src/rpc/rpc-mgr.cc@84
PS1, Line 84:   
bld.set_rpc_negotiation_timeout_ms(FLAGS_rpc_negotiation_timeout_ms);
> Does it make sense to also call bld.set_min_negotiation_threads(1) ?
The min number of threads is 0 by default. What that means is that there won't 
be any "permanent" thread that is always waiting to do negotiation. They will 
be spawned as required (up to max_thread_count) and live for some small period 
after which it times out and kills itself.

Given our workload, we would have bursts of negotiations at different points in 
time with potentially vast intervals in between. So maybe it's better to save 
the resources of that one thread. What do you think?


http://gerrit.cloudera.org:8080/#/c/9186/1/be/src/rpc/rpc-mgr.cc@84
PS1, Line 84: FLAGS_rpc_negotiation_timeout_ms
> What happens if this flag is set to negative by accident ? Will the code cr
There's a CHECK to make sure that this isn't negative:
https://github.com/apache/impala/blob/master/be/src/kudu/util/threadpool.cc#L79

So it will cause a crash.


http://gerrit.cloudera.org:8080/#/c/9186/1/be/src/rpc/rpc-mgr.cc@85
PS1, Line 85: FLAGS_rpc_negotiation_thread_count
> Will it be safer to do max(1, FLAGS_rpc_negotiation_thread_count) to guaran
Yes that makes sense. Done.



--
To view, visit http://gerrit.cloudera.org:8080/9186
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I108d700e7eac04b678e21a3a920aac81ba8eede5
Gerrit-Change-Number: 9186
Gerrit-PatchSet: 1
Gerrit-Owner: Sailesh Mukil 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Comment-Date: Fri, 02 Feb 2018 23:46:14 +
Gerrit-HasComments: Yes


[Impala-ASF-CR] IMPALA-5269: Fix issue with final line of query followed by a comment

2018-02-02 Thread Fredy Wijaya (Code Review)
Fredy Wijaya has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9191 )

Change subject: IMPALA-5269: Fix issue with final line of query followed by a 
comment
..


Patch Set 2:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/9191/2/shell/impala_shell.py
File shell/impala_shell.py:

http://gerrit.cloudera.org:8080/#/c/9191/2/shell/impala_shell.py@394
PS2, Line 394: line = line[0: line.index('--')].strip() if '--' in line 
else line
> I don't think that this is sufficient to determine the location of a commen
I changed the logic in my 3rd patch to use an SQL parser to remove the last 
comment token.



--
To view, visit http://gerrit.cloudera.org:8080/9191
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I54f9a8f65214023520eaa010fc462a663d02d258
Gerrit-Change-Number: 9191
Gerrit-PatchSet: 2
Gerrit-Owner: Fredy Wijaya 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Comment-Date: Fri, 02 Feb 2018 23:45:42 +
Gerrit-HasComments: Yes


[Impala-ASF-CR] IMPALA-6456: Add flags to configure rpc negotiation timeout ms and negotiation thread count in KRPC

2018-02-02 Thread Sailesh Mukil (Code Review)
Hello Michael Ho,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/9186

to look at the new patch set (#2).

Change subject: IMPALA-6456: Add flags to configure rpc_negotiation_timeout_ms 
and negotiation thread count in KRPC
..

IMPALA-6456: Add flags to configure rpc_negotiation_timeout_ms and negotiation 
thread count in KRPC

With the fix for KUDU-2228, the FLAGS_rpc_negotiation_timeout_ms was
retired in KRPC.

This patch introduces a flag to be able to configure that from the
Impala side (FLAGS_rpc_negotiation_timeout_ms).

It also introduces a flag to configure the negotiation
thread count (FLAGS_rpc_negotiation_thread_count).

Added a test to verify that setting FLAGS_rpc_negotiation_timeout_ms
to 0 causes negotiation failures. We unfortunately can't write a test
to check the same for FLAGS_rpc_negotiation_thread_count due to
DCHECKS present in the code.

Change-Id: I108d700e7eac04b678e21a3a920aac81ba8eede5
---
M be/src/rpc/rpc-mgr-test.cc
M be/src/rpc/rpc-mgr.cc
2 files changed, 31 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/9186/2
--
To view, visit http://gerrit.cloudera.org:8080/9186
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I108d700e7eac04b678e21a3a920aac81ba8eede5
Gerrit-Change-Number: 9186
Gerrit-PatchSet: 2
Gerrit-Owner: Sailesh Mukil 
Gerrit-Reviewer: Michael Ho 


[Impala-ASF-CR] IMPALA-6449: Use CLOCK MONOTONIC in ConditionVariable

2018-02-02 Thread Sailesh Mukil (Code Review)
Sailesh Mukil has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9158 )

Change subject: IMPALA-6449: Use CLOCK_MONOTONIC in ConditionVariable
..


Patch Set 1: Code-Review+2

(1 comment)

http://gerrit.cloudera.org:8080/#/c/9158/1//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/9158/1//COMMIT_MSG@13
PS1, Line 13: CLOCK_MONOTONIC
> Thanks for doing the research. I believe CLOCK_MONOTONIC_RAW is only availa
Makes sense. Let's just add that to the commit message then, so it's useful for 
future reference.



--
To view, visit http://gerrit.cloudera.org:8080/9158
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I81611cfd5e7c5347203fe7fa6b0f615602257f87
Gerrit-Change-Number: 9158
Gerrit-PatchSet: 1
Gerrit-Owner: Michael Ho 
Gerrit-Reviewer: Jim Apple 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Comment-Date: Fri, 02 Feb 2018 23:36:00 +
Gerrit-HasComments: Yes


[Impala-ASF-CR] IMPALA-6113: Skip row groups with predicates on NULL columns

2018-02-02 Thread Tim Armstrong (Code Review)
Tim Armstrong has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9140 )

Change subject: IMPALA-6113: Skip row groups with predicates on NULL columns
..


Patch Set 8: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/9140
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I141317af0e0df30da8f220b29b0bfba364f40ddf
Gerrit-Change-Number: 9140
Gerrit-PatchSet: 8
Gerrit-Owner: Gabor Kaszab 
Gerrit-Reviewer: Gabor Kaszab 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: anujphadke 
Gerrit-Comment-Date: Fri, 02 Feb 2018 23:31:44 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-6113: Skip row groups with predicates on NULL columns

2018-02-02 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9140 )

Change subject: IMPALA-6113: Skip row groups with predicates on NULL columns
..


Patch Set 8:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1868/


--
To view, visit http://gerrit.cloudera.org:8080/9140
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I141317af0e0df30da8f220b29b0bfba364f40ddf
Gerrit-Change-Number: 9140
Gerrit-PatchSet: 8
Gerrit-Owner: Gabor Kaszab 
Gerrit-Reviewer: Gabor Kaszab 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: anujphadke 
Gerrit-Comment-Date: Fri, 02 Feb 2018 23:31:57 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-6338: Fix flaky test profile fragment instances

2018-02-02 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/8997 )

Change subject: IMPALA-6338: Fix flaky test_profile_fragment_instances
..


Patch Set 6: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/8997
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3a1e3c62952003f37f88fe2b662bb11889ed
Gerrit-Change-Number: 8997
Gerrit-PatchSet: 6
Gerrit-Owner: Thomas Tauber-Marshall 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 02 Feb 2018 23:22:50 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-6338: Fix flaky test profile fragment instances

2018-02-02 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/8997 )

Change subject: IMPALA-6338: Fix flaky test_profile_fragment_instances
..

IMPALA-6338: Fix flaky test_profile_fragment_instances

test_profile_fragment_instances checks that, once all the results have
been returned, every fragment instance appears in the query profile
for a query that internally cancels fragment instances that are still
executing when the results have been fully returned.

Every fis is guaranteed to send a profile to the coordinator in
Finalize(), but previously fragment profiles were not applied by the
coordinator if the backend was 'done', defined as either all instances
having completed or one has entered an error state (including
cancelled).

So, the test could fail by the following sequence:
- Some fragment for a particular backend sends an update to the
  coordinator. 'returned_all_results_' is true, so the coordinator
  responds indicating the the backend should cancel its remaining
  fragments.
- Another fragment from that backend executes Finalize() and reports
  that it was cancelled. This causes the coordinator to consider the
  entire backend to be 'done'.
- A third fragment, which had not previously sent a report from the
  reporting thread, from the same backend executes Finalize(). This
  report will not be applied by the coordinator as the backend is
  considered 'done', so this fragment will not appear in the final
  profile.

The solution is to change the definition of 'done' to not include a
backend that has been cancelled but still has fragments that haven't
completed. This guarantees that for queries that complete successfully
and are cancelled internally, all fis will send a report and have it
applied by the coordinator before all results have been returned,
since if eos is true Coordinator::GetNext() calls
WaitForBackendCompletion(), which in this situation will now wait for
all fis to Finalize().

Returning results for queries that are cancelled by the user is
unaffected as the manual cancel path causes WaitForBackendCompletion().

Testing:
- Ran test_profile_fragment_instances in a loop with no failures.
  I can reliably repro the original problem with a few carefully
  placed sleeps.

Change-Id: I3a1e3c62952003f37f88fe2b662bb11889ed
Reviewed-on: http://gerrit.cloudera.org:8080/8997
Reviewed-by: Tim Armstrong 
Tested-by: Impala Public Jenkins
---
M be/src/common/status.h
M be/src/runtime/coordinator-backend-state.cc
M be/src/runtime/coordinator-backend-state.h
M be/src/runtime/coordinator.cc
M be/src/runtime/coordinator.h
M tests/query_test/test_observability.py
6 files changed, 30 insertions(+), 18 deletions(-)

Approvals:
  Tim Armstrong: Looks good to me, approved
  Impala Public Jenkins: Verified

--
To view, visit http://gerrit.cloudera.org:8080/8997
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I3a1e3c62952003f37f88fe2b662bb11889ed
Gerrit-Change-Number: 8997
Gerrit-PatchSet: 7
Gerrit-Owner: Thomas Tauber-Marshall 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Reviewer: Tim Armstrong 


[Impala-ASF-CR] IMPALA-5990: End-to-end compression of metadata

2018-02-02 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/8825 )

Change subject: IMPALA-5990: End-to-end compression of metadata
..


Patch Set 12: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/8825
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3a8819cad734b3a416eef6c954e55b73cc6023ae
Gerrit-Change-Number: 8825
Gerrit-PatchSet: 12
Gerrit-Owner: Tianyi Wang 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Tianyi Wang 
Gerrit-Comment-Date: Fri, 02 Feb 2018 23:17:46 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-6449: Use CLOCK MONOTONIC in ConditionVariable

2018-02-02 Thread Michael Ho (Code Review)
Michael Ho has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9158 )

Change subject: IMPALA-6449: Use CLOCK_MONOTONIC in ConditionVariable
..


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/9158/1//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/9158/1//COMMIT_MSG@13
PS1, Line 13: CLOCK_MONOTONIC
> Reading the man pages, it basically says that even CLOCK_MONOTONIC can be a
Thanks for doing the research. I believe CLOCK_MONOTONIC_RAW is only available 
after Linux kernel 2.6.28 so CLOCK_MONOTONIC seems more widely available.

https://www.systutorials.com/docs/linux/man/2-clock_settime/

FWIW, our monotonic stopwatch is also using CLOCK_MONOTONIC. The monotonicity 
of the clock is the major point here. I agree that CLOCK_MONOTONIC_RAW would be 
even better but we may need to do more work to detect whether it's available on 
the platform.



--
To view, visit http://gerrit.cloudera.org:8080/9158
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I81611cfd5e7c5347203fe7fa6b0f615602257f87
Gerrit-Change-Number: 9158
Gerrit-PatchSet: 1
Gerrit-Owner: Michael Ho 
Gerrit-Reviewer: Jim Apple 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Comment-Date: Fri, 02 Feb 2018 23:09:44 +
Gerrit-HasComments: Yes


[Impala-ASF-CR] IMPALA-6471: [docs] Corrected ALTER TABLE ADD PARTITION syntax for Kudu table

2018-02-02 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/9187 )

Change subject: IMPALA-6471: [docs] Corrected ALTER TABLE ADD PARTITION syntax 
for Kudu table
..

IMPALA-6471: [docs] Corrected ALTER TABLE ADD PARTITION syntax for Kudu table

Change-Id: I70c49286ed6e250707a6edb5ecd77448d1142d0c
Reviewed-on: http://gerrit.cloudera.org:8080/9187
Reviewed-by: Thomas Tauber-Marshall 
Tested-by: Impala Public Jenkins
---
M docs/topics/impala_alter_table.xml
1 file changed, 1 insertion(+), 1 deletion(-)

Approvals:
  Thomas Tauber-Marshall: Looks good to me, approved
  Impala Public Jenkins: Verified

--
To view, visit http://gerrit.cloudera.org:8080/9187
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I70c49286ed6e250707a6edb5ecd77448d1142d0c
Gerrit-Change-Number: 9187
Gerrit-PatchSet: 2
Gerrit-Owner: Alex Rodoni 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: John Russell 
Gerrit-Reviewer: Thomas Tauber-Marshall 


[Impala-ASF-CR] IMPALA-6471: [docs] Corrected ALTER TABLE ADD PARTITION syntax for Kudu table

2018-02-02 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9187 )

Change subject: IMPALA-6471: [docs] Corrected ALTER TABLE ADD PARTITION syntax 
for Kudu table
..


Patch Set 1: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/9187
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I70c49286ed6e250707a6edb5ecd77448d1142d0c
Gerrit-Change-Number: 9187
Gerrit-PatchSet: 1
Gerrit-Owner: Alex Rodoni 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: John Russell 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Comment-Date: Fri, 02 Feb 2018 22:37:29 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-6471: [docs] Corrected ALTER TABLE ADD PARTITION syntax for Kudu table

2018-02-02 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9187 )

Change subject: IMPALA-6471: [docs] Corrected ALTER TABLE ADD PARTITION syntax 
for Kudu table
..


Patch Set 1:

Build started: https://jenkins.impala.io/job/gerrit-docs-submit/194/


--
To view, visit http://gerrit.cloudera.org:8080/9187
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I70c49286ed6e250707a6edb5ecd77448d1142d0c
Gerrit-Change-Number: 9187
Gerrit-PatchSet: 1
Gerrit-Owner: Alex Rodoni 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: John Russell 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Comment-Date: Fri, 02 Feb 2018 22:28:35 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-6396: Exchange node's memory usage should include its receiver's

2018-02-02 Thread Michael Ho (Code Review)
Michael Ho has uploaded a new patch set (#2). ( 
http://gerrit.cloudera.org:8080/9202 )

Change subject: IMPALA-6396: Exchange node's memory usage should include its 
receiver's
..

IMPALA-6396: Exchange node's memory usage should include its receiver's

A DataStreamRecvr is co-owned by the DataStreamMgr and
an Exchange node. However, the life time of the memory
allocations (e.g. row batches) of a DataStreamRecvr never
exceeds that of its owning Exchange node. Previously, we
used the fragment instance's MemTracker as the parent of
the DataStreamRecvr's MemTracker. This change switches to
using the MemTracker of the owning Exchange node as the
parent tracker of the DataStreamRecvr. This makes it
easier to identify the peak memory usage of the receivers
of different exchange nodes in the runtime profile and
query summary.

Sample output from TPCH-Q21:

EXCHANGE_NODE (id=18):(Total: 1s448ms, non-child: 265.818ms, % non-child: 
18.35%)
   - ConvertRowBatchTime: 223.895ms
   - PeakMemoryUsage: 10.04 MB (10524943)
   - RowsReturned: 1.27M (1267464)
   - RowsReturnedRate: 875.19 K/sec
  RecvrSide:
BytesReceived(500.000ms): 0, 1.64 MB, 9.98 MB, 9.98 MB, 10.01 MB, 10.01 MB, 
10.01 MB, 31.79 MB, 60.19 MB, 87.84 MB
 - FirstBatchArrivalWaitTime: 0.000ns
 - TotalBytesReceived: 93.07 MB (97594728)
 - TotalGetBatchTime: 1s194ms
   - DataArrivalTimer: 1s183ms
   SenderSide:
  - DeserializeRowBatchTime: 344.343ms
  - NumBatchesAccepted: 3.80K (3796)
  - NumBatchesDeferred: 5 (5)
  - NumEarlySenders: 0 (0)

Change-Id: I8ca3c47d87bfcd221d34565eda1878f3c15d5c45
---
M be/src/exec/exchange-node.cc
M be/src/runtime/data-stream-mgr-base.h
M be/src/runtime/data-stream-mgr.cc
M be/src/runtime/data-stream-mgr.h
M be/src/runtime/data-stream-test.cc
M be/src/runtime/krpc-data-stream-mgr.cc
M be/src/runtime/krpc-data-stream-mgr.h
M tests/query_test/test_observability.py
8 files changed, 35 insertions(+), 36 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/02/9202/2
--
To view, visit http://gerrit.cloudera.org:8080/9202
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I8ca3c47d87bfcd221d34565eda1878f3c15d5c45
Gerrit-Change-Number: 9202
Gerrit-PatchSet: 2
Gerrit-Owner: Michael Ho 


[Impala-ASF-CR] IMPALA-6369: Exchange node's memory usage should include its receiver's

2018-02-02 Thread Michael Ho (Code Review)
Michael Ho has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/9202


Change subject: IMPALA-6369: Exchange node's memory usage should include its 
receiver's
..

IMPALA-6369: Exchange node's memory usage should include its receiver's

A DataStreamRecvr is co-owned by the DataStreamMgr and
an Exchange node. However, the life time of the memory
allocations (e.g. row batches) of a DataStreamRecvr never
exceeds that of its owning Exchange node. Previously, we
used the fragment instance's MemTracker as the parent of
the DataStreamRecvr's MemTracker. This change switches to
using the MemTracker of the owning Exchange node as the
parent tracker of the DataStreamRecvr. This makes it
easier to identify the peak memory usage of the receivers
of different exchange nodes in the runtime profile and
query summary.

Sample output from TPCH-Q21:

EXCHANGE_NODE (id=18):(Total: 1s448ms, non-child: 265.818ms, % non-child: 
18.35%)
   - ConvertRowBatchTime: 223.895ms
   - PeakMemoryUsage: 10.04 MB (10524943)
   - RowsReturned: 1.27M (1267464)
   - RowsReturnedRate: 875.19 K/sec
  RecvrSide:
BytesReceived(500.000ms): 0, 1.64 MB, 9.98 MB, 9.98 MB, 10.01 MB, 10.01 MB, 
10.01 MB, 31.79 MB, 60.19 MB, 87.84 MB
 - FirstBatchArrivalWaitTime: 0.000ns
 - TotalBytesReceived: 93.07 MB (97594728)
 - TotalGetBatchTime: 1s194ms
   - DataArrivalTimer: 1s183ms
   SenderSide:
  - DeserializeRowBatchTime: 344.343ms
  - NumBatchesAccepted: 3.80K (3796)
  - NumBatchesDeferred: 5 (5)
  - NumEarlySenders: 0 (0)

Change-Id: I8ca3c47d87bfcd221d34565eda1878f3c15d5c45
---
M be/src/exec/exchange-node.cc
M be/src/runtime/data-stream-mgr-base.h
M be/src/runtime/data-stream-mgr.cc
M be/src/runtime/data-stream-mgr.h
M be/src/runtime/data-stream-test.cc
M be/src/runtime/krpc-data-stream-mgr.cc
M be/src/runtime/krpc-data-stream-mgr.h
M tests/query_test/test_observability.py
8 files changed, 35 insertions(+), 36 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/02/9202/1
--
To view, visit http://gerrit.cloudera.org:8080/9202
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I8ca3c47d87bfcd221d34565eda1878f3c15d5c45
Gerrit-Change-Number: 9202
Gerrit-PatchSet: 1
Gerrit-Owner: Michael Ho 


[Impala-ASF-CR] IMPALA-6204: Remove external DataSource

2018-02-02 Thread Philip Zeyliger (Code Review)
Philip Zeyliger has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9192 )

Change subject: IMPALA-6204: Remove external DataSource
..


Patch Set 3:

I think this is now reviewable. I ran core tests successfully with this before 
rebasing. I'm running them again after rebase 
(https://jenkins.impala.io/job/gerrit-verify-dryrun-external/71/). There were 
some conflicts with the keyword change, but nothing interesting.

BTW: +95, -4667!!!


--
To view, visit http://gerrit.cloudera.org:8080/9192
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I02a3a6740466ed7372b71d948c705b30886dcfb6
Gerrit-Change-Number: 9192
Gerrit-PatchSet: 3
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Comment-Date: Fri, 02 Feb 2018 22:24:01 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-6204: Remove external DataSource

2018-02-02 Thread Philip Zeyliger (Code Review)
Hello Dimitris Tsirogiannis,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/9192

to look at the new patch set (#3).

Change subject: IMPALA-6204: Remove external DataSource
..

IMPALA-6204: Remove external DataSource

Removes DataSourceScanNode, external data sources, and all affiliated
code, tests, and documentation.

When a data source table is encountered, we now throw an exception. To
the user, this looks like:

  [pannier.ca.cloudera.com:21000] > create table t (x int) stored as textfile 
tblproperties('__IMPALA_DATA_SOURCE_NAME'='V1');
  Query: create table t (x int) stored as textfile 
tblproperties('__IMPALA_DATA_SOURCE_NAME'='V1')
  Fetched 0 row(s) in 0.11s
  [pannier.ca.cloudera.com:21000] > select * from t;
  Query: select * from t
  Query submitted at: 2018-02-01 17:16:26 (Coordinator: 
http://pannier.ca.cloudera.com:25000)
  ERROR: AnalysisException: Failed to load metadata for table: 't'
  CAUSED BY: TableLoadingException: Failed to load metadata for table: 
default.t. Running 'invalidate metadata default.t' may resolve this problem.
  CAUSED BY: UnsupportedOperationException: Eternal Data source table not 
supported.

A test has been added to capture this behavior.

For the most part, I deleted the unused code. In a few places, a renamed
the Thrift enums and threw errors if they're encountered. For Thrift
structs, I left a comment about the now-skipped id that used to
represent a data-source related entry.

Cherry-picks: not for 2.x

Change-Id: I02a3a6740466ed7372b71d948c705b30886dcfb6
---
M CMakeLists.txt
M be/generated-sources/gen-cpp/CMakeLists.txt
M be/src/catalog/catalog-util.cc
M be/src/exec/CMakeLists.txt
M be/src/exec/catalog-op-executor.cc
M be/src/exec/catalog-op-executor.h
D be/src/exec/data-source-scan-node.cc
D be/src/exec/data-source-scan-node.h
M be/src/exec/exec-node.cc
D be/src/exec/external-data-source-executor.cc
D be/src/exec/external-data-source-executor.h
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/scheduling/scheduler.cc
M be/src/service/client-request-state.cc
M be/src/service/frontend.cc
M be/src/service/frontend.h
M be/src/service/impala-server.cc
M bin/clean-cmake.sh
M bin/clean.sh
M buildall.sh
M common/thrift/CMakeLists.txt
M common/thrift/CatalogObjects.thrift
M common/thrift/CatalogService.thrift
M common/thrift/Data.thrift
M common/thrift/Descriptors.thrift
D common/thrift/ExternalDataSource.thrift
M common/thrift/Frontend.thrift
M common/thrift/JniCatalog.thrift
M common/thrift/PlanNodes.thrift
M docs/impala.ditamap
D docs/topics/impala_create_data_source.xml
D docs/topics/impala_data_sources.xml
D docs/topics/impala_drop_data_source.xml
D ext-data-source/.gitignore
D ext-data-source/CMakeLists.txt
D ext-data-source/api/pom.xml
D 
ext-data-source/api/src/main/java/org/apache/impala/extdatasource/util/SerializationUtils.java
D 
ext-data-source/api/src/main/java/org/apache/impala/extdatasource/v1/ExternalDataSource.java
D ext-data-source/pom.xml
D ext-data-source/sample/pom.xml
D 
ext-data-source/sample/src/main/java/org/apache/impala/extdatasource/sample/EchoDataSource.java
D ext-data-source/test/pom.xml
D 
ext-data-source/test/src/main/java/org/apache/impala/extdatasource/AllTypesDataSource.java
M fe/CMakeLists.txt
M fe/pom.xml
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
D fe/src/main/java/org/apache/impala/analysis/CreateDataSrcStmt.java
D fe/src/main/java/org/apache/impala/analysis/CreateTableDataSrcStmt.java
D fe/src/main/java/org/apache/impala/analysis/DropDataSrcStmt.java
M fe/src/main/java/org/apache/impala/analysis/PrivilegeSpec.java
D fe/src/main/java/org/apache/impala/analysis/ShowDataSrcsStmt.java
M fe/src/main/java/org/apache/impala/analysis/TableRef.java
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
D fe/src/main/java/org/apache/impala/catalog/DataSource.java
D fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java
M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
D fe/src/main/java/org/apache/impala/extdatasource/ApiVersion.java
D 
fe/src/main/java/org/apache/impala/extdatasource/ExternalDataSourceExecutor.java
D fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
M fe/src/main/jflex/sql-scanner.flex
M 

[Impala-ASF-CR] IMPALA-5293: Turn insert clustering on by default

2018-02-02 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9153 )

Change subject: IMPALA-5293: Turn insert clustering on by default
..


Patch Set 10:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1866/


--
To view, visit http://gerrit.cloudera.org:8080/9153
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idbf2368cf4415e6ecfa65058daf6ff87ef62f9d9
Gerrit-Change-Number: 9153
Gerrit-PatchSet: 10
Gerrit-Owner: Lars Volker 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 02 Feb 2018 22:04:55 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-5293: Turn insert clustering on by default

2018-02-02 Thread Lars Volker (Code Review)
Lars Volker has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9153 )

Change subject: IMPALA-5293: Turn insert clustering on by default
..


Patch Set 10: Code-Review+2

PS10 is a final rebase before submitting the change. Carrying Alex's +2.


--
To view, visit http://gerrit.cloudera.org:8080/9153
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idbf2368cf4415e6ecfa65058daf6ff87ef62f9d9
Gerrit-Change-Number: 9153
Gerrit-PatchSet: 10
Gerrit-Owner: Lars Volker 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 02 Feb 2018 22:04:49 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-6449: Use CLOCK MONOTONIC in ConditionVariable

2018-02-02 Thread Sailesh Mukil (Code Review)
Sailesh Mukil has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9158 )

Change subject: IMPALA-6449: Use CLOCK_MONOTONIC in ConditionVariable
..


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/9158/1//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/9158/1//COMMIT_MSG@13
PS1, Line 13: CLOCK_MONOTONIC
Reading the man pages, it basically says that even CLOCK_MONOTONIC can be 
affected by sources like NTP. However, the difference is that with 
CLOCK_MONOTONIC, the time will not jump backwards, but rather experience "time 
slew", which means that the frequency of clock ticks can change momentarily to 
reflect adjustments in NTP.

CLOCK_MONOTONIC_RAW on the other hand shows the absolute wall clock time that's 
not affected by NTP. But it could be affected by things like temperature, 
environment, etc. that will not be corrected (CLOCK_MONOTONIC would have that 
corrected apparently)

It seems a bit confusing which would be considered "right" while measuring time 
like we do. I'm okay with both, as long as we've considered both the options.

Whatever you think is the right option, I think we can just add to the commit 
message that we've considered the other one and give a reason as to why we 
didn't choose that.



--
To view, visit http://gerrit.cloudera.org:8080/9158
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I81611cfd5e7c5347203fe7fa6b0f615602257f87
Gerrit-Change-Number: 9158
Gerrit-PatchSet: 1
Gerrit-Owner: Michael Ho 
Gerrit-Reviewer: Jim Apple 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Comment-Date: Fri, 02 Feb 2018 22:03:42 +
Gerrit-HasComments: Yes


[Impala-ASF-CR] IMPALA-5293: Turn insert clustering on by default

2018-02-02 Thread Lars Volker (Code Review)
Hello Tim Armstrong, Alex Behm,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/9153

to look at the new patch set (#10).

Change subject: IMPALA-5293: Turn insert clustering on by default
..

IMPALA-5293: Turn insert clustering on by default

This change enables clustering by default. IMPALA-2521 introduced the
'clustered' hint which inserts a local sort by the partitioning columns
to a query plan. The hint is only effective for HDFS and Kudu tables.

Like before, the 'noclustered' hint prevents clustering. If a table has
ordering columns defined, the 'noclustered' hint is ignored and we
issue a warning.

This change removes some tests that were added specifically to test
that clustering can be enabled using the 'clustered' hint. It changes
some tests to use the 'noclustered' hint to make sure that clustering
can be disabled. It also adds tests to make sure that we cover the
'noclustered' case properly.

Cherry-picks: not for 2.x.

Change-Id: Idbf2368cf4415e6ecfa65058daf6ff87ef62f9d9
---
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/planner/Planner.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java
M 
testdata/workloads/functional-planner/queries/PlannerTest/constant-folding.test
M testdata/workloads/functional-planner/queries/PlannerTest/empty.test
M testdata/workloads/functional-planner/queries/PlannerTest/insert.test
M testdata/workloads/functional-planner/queries/PlannerTest/kudu.test
M testdata/workloads/functional-planner/queries/PlannerTest/order.test
M 
testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test
M testdata/workloads/functional-planner/queries/PlannerTest/with-clause.test
M testdata/workloads/functional-query/queries/QueryTest/insert.test
M testdata/workloads/functional-query/queries/QueryTest/stats-extrapolation.test
12 files changed, 248 insertions(+), 66 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/9153/10
--
To view, visit http://gerrit.cloudera.org:8080/9153
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Idbf2368cf4415e6ecfa65058daf6ff87ef62f9d9
Gerrit-Change-Number: 9153
Gerrit-PatchSet: 10
Gerrit-Owner: Lars Volker 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Tim Armstrong 


[Impala-ASF-CR] IMPALA-6456: Add flags to configure rpc negotiation timeout ms and negotiation thread count in KRPC

2018-02-02 Thread Michael Ho (Code Review)
Michael Ho has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9186 )

Change subject: IMPALA-6456: Add flags to configure rpc_negotiation_timeout_ms 
and negotiation thread count in KRPC
..


Patch Set 1:

(4 comments)

http://gerrit.cloudera.org:8080/#/c/9186/1/be/src/rpc/rpc-mgr.cc
File be/src/rpc/rpc-mgr.cc:

http://gerrit.cloudera.org:8080/#/c/9186/1/be/src/rpc/rpc-mgr.cc@70
PS1, Line 70: Number of threads to dedicate to process connection negotiations
Maximum number of threads dedicated to handling RPC connection negotiations


http://gerrit.cloudera.org:8080/#/c/9186/1/be/src/rpc/rpc-mgr.cc@84
PS1, Line 84:   
bld.set_rpc_negotiation_timeout_ms(FLAGS_rpc_negotiation_timeout_ms);
Does it make sense to also call bld.set_min_negotiation_threads(1) ?


http://gerrit.cloudera.org:8080/#/c/9186/1/be/src/rpc/rpc-mgr.cc@84
PS1, Line 84: FLAGS_rpc_negotiation_timeout_ms
What happens if this flag is set to negative by accident ? Will the code crash 
or does it translate to a very large unsigned value ?


http://gerrit.cloudera.org:8080/#/c/9186/1/be/src/rpc/rpc-mgr.cc@85
PS1, Line 85: FLAGS_rpc_negotiation_thread_count
Will it be safer to do max(1, FLAGS_rpc_negotiation_thread_count) to guarantee 
there is at least on negotiation thread available ?



--
To view, visit http://gerrit.cloudera.org:8080/9186
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I108d700e7eac04b678e21a3a920aac81ba8eede5
Gerrit-Change-Number: 9186
Gerrit-PatchSet: 1
Gerrit-Owner: Sailesh Mukil 
Gerrit-Reviewer: Michael Ho 
Gerrit-Comment-Date: Fri, 02 Feb 2018 21:43:52 +
Gerrit-HasComments: Yes


[Impala-ASF-CR] IMPALA-6448: Re-enable kerberized testing with KRPC

2018-02-02 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9164 )

Change subject: IMPALA-6448: Re-enable kerberized testing with KRPC
..


Patch Set 3:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1865/


--
To view, visit http://gerrit.cloudera.org:8080/9164
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6412978316de90875c98f8fbe51c8d215c227b18
Gerrit-Change-Number: 9164
Gerrit-PatchSet: 3
Gerrit-Owner: Sailesh Mukil 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Comment-Date: Fri, 02 Feb 2018 21:34:27 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-6448: Re-enable kerberized testing with KRPC

2018-02-02 Thread Sailesh Mukil (Code Review)
Sailesh Mukil has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9164 )

Change subject: IMPALA-6448: Re-enable kerberized testing with KRPC
..


Patch Set 3: Code-Review+2

(1 comment)

Rebase, carry +2.

http://gerrit.cloudera.org:8080/#/c/9164/2//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/9164/2//COMMIT_MSG@14
PS2, Line 14: .
: This behavior can be found in SaslInit() in 
kudu/rpc/sasl_common.cc
> nit: as SaslInit() which is called once per process will hard code some con
Done



--
To view, visit http://gerrit.cloudera.org:8080/9164
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6412978316de90875c98f8fbe51c8d215c227b18
Gerrit-Change-Number: 9164
Gerrit-PatchSet: 3
Gerrit-Owner: Sailesh Mukil 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Comment-Date: Fri, 02 Feb 2018 21:34:14 +
Gerrit-HasComments: Yes


[Impala-ASF-CR] IMPALA-6448: Re-enable kerberized testing with KRPC

2018-02-02 Thread Sailesh Mukil (Code Review)
Hello Michael Ho,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/9164

to look at the new patch set (#3).

Change subject: IMPALA-6448: Re-enable kerberized testing with KRPC
..

IMPALA-6448: Re-enable kerberized testing with KRPC

For the patch for IMPALA-5054, we realized that we needed to make
the kudu::rpc::Messenger configurable. A patch was done on the Kudu
side which is tracked by KUDU-2228. As part of that patch, one of
the design decisions taken was to only allow kerberos either on or
off for the entirety of the process life. This means that we cannot
switch kerberos on and off in the same process any more with KRPC.
This behavior can be found in SaslInit() in kudu/rpc/sasl_common.cc
as SaslInit() which is called once per process will hard code some
configuration which cannot be toggled.

This affected our kerberized rpc-mgr-tests. This patch splits out
the kerberized part of rpc-mgr-test into rpc-mgr-kerberized-test.

It also puts the common code between both the files into
rpc-mgr-test-base.h

Change-Id: I6412978316de90875c98f8fbe51c8d215c227b18
---
M be/src/rpc/CMakeLists.txt
A be/src/rpc/rpc-mgr-kerberized-test.cc
A be/src/rpc/rpc-mgr-test-base.h
M be/src/rpc/rpc-mgr-test.cc
4 files changed, 380 insertions(+), 316 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/64/9164/3
--
To view, visit http://gerrit.cloudera.org:8080/9164
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6412978316de90875c98f8fbe51c8d215c227b18
Gerrit-Change-Number: 9164
Gerrit-PatchSet: 3
Gerrit-Owner: Sailesh Mukil 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Sailesh Mukil 


[Impala-ASF-CR](2.x) IMPALA-3916: Reserve SQL:2016 reserved words

2018-02-02 Thread Tianyi Wang (Code Review)
Tianyi Wang has uploaded a new patch set (#2). ( 
http://gerrit.cloudera.org:8080/9198 )

Change subject: IMPALA-3916: Reserve SQL:2016 reserved words
..

IMPALA-3916: Reserve SQL:2016 reserved words

This patch reserves SQL:2016 reserved words, excluding:
1. Impala builtin function names.
2. Time unit words(year, month, etc.).
3. An exception list based on a discussion.

Some test cases are modified to avoid these words. A impalad and
catalogd startup option reserved_words_version is added. The words are
reserved if the option is set to "3.0.0".

Change-Id: If1b295e6a77e840cf1b794c2eb73e1b9d2b8ddd6
Reviewed-on: http://gerrit.cloudera.org:8080/9096
Reviewed-by: Alex Behm 
Reviewed-by: Philip Zeyliger 
Tested-by: Impala Public Jenkins
(cherry picked from commit f0b3d9d122f2c6eb4137bf93e3512a489ff8fab0)

Change-Id: Ifd29d6aebe793d4eb37c092c1364f3acfebd91f8
---
M be/src/common/global-flags.cc
M be/src/common/init.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/catalog/BuiltinsDb.java
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/Db.java
M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/jflex/sql-scanner.flex
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
M fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java
M fe/src/test/java/org/apache/impala/catalog/CatalogTest.java
M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java
M fe/src/test/java/org/apache/impala/planner/StatsExtrapolationTest.java
M fe/src/test/java/org/apache/impala/service/JdbcTest.java
M testdata/workloads/functional-query/queries/QueryTest/empty-build-joins.test
M testdata/workloads/functional-query/queries/QueryTest/exprs.test
M testdata/workloads/functional-query/queries/QueryTest/partition-col-types.test
M testdata/workloads/functional-query/queries/QueryTest/single-node-nlj.test
M testdata/workloads/functional-query/queries/QueryTest/spilling-large-rows.test
M testdata/workloads/functional-query/queries/QueryTest/values.test
A tests/custom_cluster/test_reserved_words_version.py
M tests/custom_cluster/test_stats_extrapolation.py
M tests/query_test/test_sort.py
28 files changed, 570 insertions(+), 442 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/98/9198/2
--
To view, visit http://gerrit.cloudera.org:8080/9198
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: 2.x
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ifd29d6aebe793d4eb37c092c1364f3acfebd91f8
Gerrit-Change-Number: 9198
Gerrit-PatchSet: 2
Gerrit-Owner: Tianyi Wang 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Tianyi Wang 


[Impala-ASF-CR](2.x) IMPALA-3916: Change reserved words version to 2.11.0

2018-02-02 Thread Tianyi Wang (Code Review)
Tianyi Wang has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/9201


Change subject: IMPALA-3916: Change reserved_words_version to 2.11.0
..

IMPALA-3916: Change reserved_words_version to 2.11.0

Impala 2.x doesn't reserve any unused words by default.

Change-Id: I5a3e5eac8c09de4517beccf79dcea111264668c2
---
M be/src/common/global-flags.cc
M fe/src/main/jflex/sql-scanner.flex
2 files changed, 2 insertions(+), 2 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/9201/1
--
To view, visit http://gerrit.cloudera.org:8080/9201
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: 2.x
Gerrit-MessageType: newchange
Gerrit-Change-Id: I5a3e5eac8c09de4517beccf79dcea111264668c2
Gerrit-Change-Number: 9201
Gerrit-PatchSet: 1
Gerrit-Owner: Tianyi Wang 


[Impala-ASF-CR] IMPALA-5269: Fix issue with final line of query followed by a comment

2018-02-02 Thread Thomas Tauber-Marshall (Code Review)
Thomas Tauber-Marshall has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9191 )

Change subject: IMPALA-5269: Fix issue with final line of query followed by a 
comment
..


Patch Set 2:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/9191/2/shell/impala_shell.py
File shell/impala_shell.py:

http://gerrit.cloudera.org:8080/#/c/9191/2/shell/impala_shell.py@394
PS2, Line 394: line = line[0: line.index('--')].strip() if '--' in line 
else line
I don't think that this is sufficient to determine the location of a comment. 
For example, consider the query:

select "--";



--
To view, visit http://gerrit.cloudera.org:8080/9191
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I54f9a8f65214023520eaa010fc462a663d02d258
Gerrit-Change-Number: 9191
Gerrit-PatchSet: 2
Gerrit-Owner: Fredy Wijaya 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Comment-Date: Fri, 02 Feb 2018 20:52:42 +
Gerrit-HasComments: Yes


[Impala-ASF-CR](2.x) IMPALA-3916: Reserve SQL:2016 reserved words

2018-02-02 Thread Philip Zeyliger (Code Review)
Philip Zeyliger has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9198 )

Change subject: IMPALA-3916: Reserve SQL:2016 reserved words
..


Patch Set 1: Code-Review+2

The conflict was boring, yes?

I'd prefer you change the default in a separate commit.


--
To view, visit http://gerrit.cloudera.org:8080/9198
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: 2.x
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifd29d6aebe793d4eb37c092c1364f3acfebd91f8
Gerrit-Change-Number: 9198
Gerrit-PatchSet: 1
Gerrit-Owner: Tianyi Wang 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Tianyi Wang 
Gerrit-Comment-Date: Fri, 02 Feb 2018 20:23:36 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-6113: Skip row groups with predicates on NULL columns

2018-02-02 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9140 )

Change subject: IMPALA-6113: Skip row groups with predicates on NULL columns
..


Patch Set 7: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/1862/


--
To view, visit http://gerrit.cloudera.org:8080/9140
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I141317af0e0df30da8f220b29b0bfba364f40ddf
Gerrit-Change-Number: 9140
Gerrit-PatchSet: 7
Gerrit-Owner: Gabor Kaszab 
Gerrit-Reviewer: Gabor Kaszab 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: anujphadke 
Gerrit-Comment-Date: Fri, 02 Feb 2018 19:59:36 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-6471: [docs] Corrected ALTER TABLE ADD PARTITION syntax for Kudu table

2018-02-02 Thread Alex Rodoni (Code Review)
Alex Rodoni has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/9187


Change subject: IMPALA-6471: [docs] Corrected ALTER TABLE ADD PARTITION syntax 
for Kudu table
..

IMPALA-6471: [docs] Corrected ALTER TABLE ADD PARTITION syntax for Kudu table

Change-Id: I70c49286ed6e250707a6edb5ecd77448d1142d0c
---
M docs/topics/impala_alter_table.xml
1 file changed, 1 insertion(+), 1 deletion(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/87/9187/1
--
To view, visit http://gerrit.cloudera.org:8080/9187
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I70c49286ed6e250707a6edb5ecd77448d1142d0c
Gerrit-Change-Number: 9187
Gerrit-PatchSet: 1
Gerrit-Owner: Alex Rodoni 
Gerrit-Reviewer: John Russell 
Gerrit-Reviewer: Thomas Tauber-Marshall 


[Impala-ASF-CR] IMPALA-5990: End-to-end compression of metadata

2018-02-02 Thread Tianyi Wang (Code Review)
Tianyi Wang has uploaded a new patch set (#12). ( 
http://gerrit.cloudera.org:8080/8825 )

Change subject: IMPALA-5990: End-to-end compression of metadata
..

IMPALA-5990: End-to-end compression of metadata

Currently the catalog data is compressed in the statestore, but
uncompressed when passed between FE and BE. It results in a ~2GB limit
on the metadata. IMPALA-3499 introduced a workaround in the impalad but
there isn't one in the catalogd. This patch aims to increase the size
limit for statestore updates, reduce the copying of the metadata and
reduce the memory footprint. With this patch, the catalog objects are
passed and (de)compressed between FE and BE one at a time. The new
limits are:
- A single catalog object cannot be larger than ~2GB.
- A statestore catalog update cannot be larger than ~4GB. It is
  compressed size if FLAGS_compact_catalog_topic is true.
The behavior of the catalog op executer is not changed. The data is not
compressed and the size limit is still 2GB.

Testing: Ran existing tests. A test for compressing and decompressing
catalog objects is added. Manually tested with a 1.95GB catalog object
and a 3.90 GB uncompressed statestore update.

Change-Id: I3a8819cad734b3a416eef6c954e55b73cc6023ae
---
M be/src/catalog/CMakeLists.txt
M be/src/catalog/catalog-server.cc
M be/src/catalog/catalog-server.h
A be/src/catalog/catalog-util-test.cc
M be/src/catalog/catalog-util.cc
M be/src/catalog/catalog-util.h
M be/src/catalog/catalog.cc
M be/src/catalog/catalog.h
M be/src/rpc/thrift-util.h
M be/src/service/fe-support.cc
M be/src/service/frontend.cc
M be/src/service/frontend.h
M be/src/service/impala-server.cc
M be/src/service/impalad-main.cc
M be/src/util/jni-util.cc
M be/src/util/jni-util.h
M common/thrift/CatalogInternalService.thrift
M common/thrift/Frontend.thrift
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java
M fe/src/main/java/org/apache/impala/service/FeSupport.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniCatalog.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
A fe/src/main/java/org/apache/impala/util/TByteBuffer.java
M testdata/workloads/functional-query/queries/QueryTest/exprs.test
27 files changed, 815 insertions(+), 563 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/8825/12
--
To view, visit http://gerrit.cloudera.org:8080/8825
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I3a8819cad734b3a416eef6c954e55b73cc6023ae
Gerrit-Change-Number: 8825
Gerrit-PatchSet: 12
Gerrit-Owner: Tianyi Wang 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Tianyi Wang 


[Impala-ASF-CR] IMPALA-5990: End-to-end compression of metadata

2018-02-02 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/8825 )

Change subject: IMPALA-5990: End-to-end compression of metadata
..


Patch Set 12:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1863/


--
To view, visit http://gerrit.cloudera.org:8080/8825
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3a8819cad734b3a416eef6c954e55b73cc6023ae
Gerrit-Change-Number: 8825
Gerrit-PatchSet: 12
Gerrit-Owner: Tianyi Wang 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Tianyi Wang 
Gerrit-Comment-Date: Fri, 02 Feb 2018 19:29:15 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-6337: Fix inifinte loop in Impala shell

2018-02-02 Thread Fredy Wijaya (Code Review)
Fredy Wijaya has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/9195


Change subject: IMPALA-6337: Fix inifinte loop in Impala shell
..

IMPALA-6337: Fix inifinte loop in Impala shell

The patch uses a workaround for a bug in the sqlparse.split() function
by joining the statements that contain error tokens into a single
statement

Change-Id: I9142f21a888189d351f00ce09baeba123bc0959b
---
M shell/impala_shell.py
M tests/shell/test_shell_interactive.py
2 files changed, 42 insertions(+), 1 deletion(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/95/9195/1
--
To view, visit http://gerrit.cloudera.org:8080/9195
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I9142f21a888189d351f00ce09baeba123bc0959b
Gerrit-Change-Number: 9195
Gerrit-PatchSet: 1
Gerrit-Owner: Fredy Wijaya 


[Impala-ASF-CR] IMPALA-6204: Remove external DataSource

2018-02-02 Thread Philip Zeyliger (Code Review)
Philip Zeyliger has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9192 )

Change subject: IMPALA-6204: Remove external DataSource
..


Patch Set 2: Code-Review-1

Sorry, hold off on this please. I rebased and then did some more grepping and 
found some more tentacles!


--
To view, visit http://gerrit.cloudera.org:8080/9192
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I02a3a6740466ed7372b71d948c705b30886dcfb6
Gerrit-Change-Number: 9192
Gerrit-PatchSet: 2
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Comment-Date: Fri, 02 Feb 2018 18:33:27 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-3651: [DOCS] Doc for MURMUR HASH() function

2018-02-02 Thread Alex Rodoni (Code Review)
Alex Rodoni has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9031 )

Change subject: IMPALA-3651: [DOCS] Doc for MURMUR_HASH() function
..


Patch Set 2: Code-Review+1


--
To view, visit http://gerrit.cloudera.org:8080/9031
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I784a5a080d7d13192aac2fca67f841d2d19fc99b
Gerrit-Change-Number: 9031
Gerrit-PatchSet: 2
Gerrit-Owner: Kim Jin Chul 
Gerrit-Reviewer: Alex Rodoni 
Gerrit-Reviewer: John Russell 
Gerrit-Reviewer: Kim Jin Chul 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 02 Feb 2018 18:28:54 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-5519: Allocate fragment's runtime filter memory from Buffer pool

2018-02-02 Thread Dimitris Tsirogiannis (Code Review)
Dimitris Tsirogiannis has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/8971 )

Change subject: IMPALA-5519: Allocate fragment's runtime filter memory from 
Buffer pool
..


Patch Set 8:

(5 comments)

Did you look at the tests that Alex mentioned for creating a test table with 
different cardinalities and nvds?

http://gerrit.cloudera.org:8080/#/c/8971/8/fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java
File fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java:

http://gerrit.cloudera.org:8080/#/c/8971/8/fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java@95
PS8, Line 95:   private static final long MIN_BLOOM_FILTER_SIZE = 4 * 1024;
:   private static final long MAX_BLOOM_FILTER_SIZE = 512 * 1024 * 
1024;
> the query option restrictions already take care of this limit, by making su
You may remove them and add a check instead.


http://gerrit.cloudera.org:8080/#/c/8971/8/fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java@109
PS8, Line 109: _
nit: no need for '_' when the fields are public


http://gerrit.cloudera.org:8080/#/c/8971/8/fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java@125
PS8, Line 125: bloomFilterSizeLimits_.max_ = 
tQueryOptions.getRuntime_filter_max_size();
 : bloomFilterSizeLimits_.max_ = 
Math.max(bloomFilterSizeLimits_.max_,
 : MIN_BLOOM_FILTER_SIZE);
 : bloomFilterSizeLimits_.max_ = 
Math.max(bloomFilterSizeLimits_.max_,
 : BackendConfig.INSTANCE.getMinBufferSize());
 : bloomFilterSizeLimits_.max_ = Math.min(
 : BitUtil.roundUpToPowerOf2(bloomFilterSizeLimits_.max_), 
MAX_BLOOM_FILTER_SIZE);
You may want to see if it makes sense to add this logic to FilterSizeLimits 
class. Something like setMax(), setMin(), setDefault() functions.


http://gerrit.cloudera.org:8080/#/c/8971/8/fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java@482
PS8, Line 482: sizeLimits
'filterSizeLimits'


http://gerrit.cloudera.org:8080/#/c/8971/8/fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java@494
PS8, Line 494:   filterSizeBytes_ = Math.min(filterSizeBytes_, 
filterSizeLimits.max_);
There is always the case that the size needed to achieve the desired fp rate is 
higher than the max limit size. In that case we should add a log message.



--
To view, visit http://gerrit.cloudera.org:8080/8971
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea2759665fb2e8bef9433014a8d42a7ebf99ce1f
Gerrit-Change-Number: 8971
Gerrit-PatchSet: 8
Gerrit-Owner: Bikramjeet Vig 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 02 Feb 2018 18:12:07 +
Gerrit-HasComments: Yes


[Impala-ASF-CR] IMPALA-6204: Remove external DataSource

2018-02-02 Thread Philip Zeyliger (Code Review)
Hello Dimitris Tsirogiannis,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/9192

to look at the new patch set (#2).

Change subject: IMPALA-6204: Remove external DataSource
..

IMPALA-6204: Remove external DataSource

Removes DataSourceScanNode, external data sources, and all affiliated
code, tests, and documentation.

When a data source table is encountered, we now throw an exception. To
the user, this looks like:

  [pannier.ca.cloudera.com:21000] > create table t (x int) stored as textfile 
tblproperties('__IMPALA_DATA_SOURCE_NAME'='V1');
  Query: create table t (x int) stored as textfile 
tblproperties('__IMPALA_DATA_SOURCE_NAME'='V1')
  Fetched 0 row(s) in 0.11s
  [pannier.ca.cloudera.com:21000] > select * from t;
  Query: select * from t
  Query submitted at: 2018-02-01 17:16:26 (Coordinator: 
http://pannier.ca.cloudera.com:25000)
  ERROR: AnalysisException: Failed to load metadata for table: 't'
  CAUSED BY: TableLoadingException: Failed to load metadata for table: 
default.t. Running 'invalidate metadata default.t' may resolve this problem.
  CAUSED BY: UnsupportedOperationException: Eternal Data source table not 
supported.

A test has been added to capture this behavior.

For the most part, I deleted the unused code. In a few places, a renamed
the Thrift enums and threw errors if they're encountered. For Thrift
structs, I left a comment about the now-skipped id that used to
represent a data-source related entry.

Cherry-picks: not for 2.x

Change-Id: I02a3a6740466ed7372b71d948c705b30886dcfb6
---
M CMakeLists.txt
M be/generated-sources/gen-cpp/CMakeLists.txt
M be/src/catalog/catalog-util.cc
M be/src/exec/CMakeLists.txt
M be/src/exec/catalog-op-executor.cc
M be/src/exec/catalog-op-executor.h
D be/src/exec/data-source-scan-node.cc
D be/src/exec/data-source-scan-node.h
M be/src/exec/exec-node.cc
D be/src/exec/external-data-source-executor.cc
D be/src/exec/external-data-source-executor.h
M be/src/runtime/descriptors.cc
M be/src/scheduling/scheduler.cc
M be/src/service/client-request-state.cc
M be/src/service/frontend.cc
M be/src/service/frontend.h
M be/src/service/impala-server.cc
M bin/clean-cmake.sh
M bin/clean.sh
M buildall.sh
M common/thrift/CMakeLists.txt
M common/thrift/CatalogObjects.thrift
M common/thrift/CatalogService.thrift
M common/thrift/Data.thrift
M common/thrift/Descriptors.thrift
D common/thrift/ExternalDataSource.thrift
M common/thrift/Frontend.thrift
M common/thrift/JniCatalog.thrift
M common/thrift/PlanNodes.thrift
M docs/impala.ditamap
D docs/topics/impala_create_data_source.xml
D docs/topics/impala_data_sources.xml
D docs/topics/impala_drop_data_source.xml
D ext-data-source/.gitignore
D ext-data-source/CMakeLists.txt
D ext-data-source/api/pom.xml
D 
ext-data-source/api/src/main/java/org/apache/impala/extdatasource/util/SerializationUtils.java
D 
ext-data-source/api/src/main/java/org/apache/impala/extdatasource/v1/ExternalDataSource.java
D ext-data-source/pom.xml
D ext-data-source/sample/pom.xml
D 
ext-data-source/sample/src/main/java/org/apache/impala/extdatasource/sample/EchoDataSource.java
D ext-data-source/test/pom.xml
D 
ext-data-source/test/src/main/java/org/apache/impala/extdatasource/AllTypesDataSource.java
M fe/CMakeLists.txt
M fe/pom.xml
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
D fe/src/main/java/org/apache/impala/analysis/CreateDataSrcStmt.java
D fe/src/main/java/org/apache/impala/analysis/CreateTableDataSrcStmt.java
D fe/src/main/java/org/apache/impala/analysis/DropDataSrcStmt.java
M fe/src/main/java/org/apache/impala/analysis/PrivilegeSpec.java
D fe/src/main/java/org/apache/impala/analysis/ShowDataSrcsStmt.java
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
D fe/src/main/java/org/apache/impala/catalog/DataSource.java
D fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java
M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
D fe/src/main/java/org/apache/impala/extdatasource/ApiVersion.java
D 
fe/src/main/java/org/apache/impala/extdatasource/ExternalDataSourceExecutor.java
D fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
M fe/src/main/jflex/sql-scanner.flex
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
M 

[Impala-ASF-CR] IMPALA-6204: Remove external DataSource

2018-02-02 Thread Philip Zeyliger (Code Review)
Philip Zeyliger has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/9192


Change subject: IMPALA-6204: Remove external DataSource
..

IMPALA-6204: Remove external DataSource

Removes DataSourceScanNode, external data sources, and all affiliated
code, tests, and documentation.

When a data source table is encountered, we now throw an exception. To
the user, this looks like:

  [pannier.ca.cloudera.com:21000] > create table t (x int) stored as textfile 
tblproperties('__IMPALA_DATA_SOURCE_NAME'='V1');
  Query: create table t (x int) stored as textfile 
tblproperties('__IMPALA_DATA_SOURCE_NAME'='V1')
  Fetched 0 row(s) in 0.11s
  [pannier.ca.cloudera.com:21000] > select * from t;
  Query: select * from t
  Query submitted at: 2018-02-01 17:16:26 (Coordinator: 
http://pannier.ca.cloudera.com:25000)
  ERROR: AnalysisException: Failed to load metadata for table: 't'
  CAUSED BY: TableLoadingException: Failed to load metadata for table: 
default.t. Running 'invalidate metadata default.t' may resolve this problem.
  CAUSED BY: UnsupportedOperationException: Eternal Data source table not 
supported.

A test has been added to capture this behavior.

For the most part, I deleted the unused code. In a few places, a renamed
the Thrift enums and threw errors if they're encountered. For Thrift
structs, I left a comment about the now-skipped id that used to
represent a data-source related entry.

Cherry-picks: not for 2.x

Change-Id: I02a3a6740466ed7372b71d948c705b30886dcfb6
---
M CMakeLists.txt
M be/generated-sources/gen-cpp/CMakeLists.txt
M be/src/catalog/catalog-util.cc
M be/src/exec/CMakeLists.txt
M be/src/exec/catalog-op-executor.cc
M be/src/exec/catalog-op-executor.h
D be/src/exec/data-source-scan-node.cc
D be/src/exec/data-source-scan-node.h
M be/src/exec/exec-node.cc
D be/src/exec/external-data-source-executor.cc
D be/src/exec/external-data-source-executor.h
M be/src/service/client-request-state.cc
M be/src/service/frontend.cc
M be/src/service/frontend.h
M be/src/service/impala-server.cc
M bin/clean-cmake.sh
M bin/clean.sh
M buildall.sh
M common/thrift/CMakeLists.txt
M common/thrift/CatalogObjects.thrift
M common/thrift/CatalogService.thrift
M common/thrift/Data.thrift
M common/thrift/Descriptors.thrift
D common/thrift/ExternalDataSource.thrift
M common/thrift/Frontend.thrift
M common/thrift/JniCatalog.thrift
M common/thrift/PlanNodes.thrift
M docs/impala.ditamap
D docs/topics/impala_create_data_source.xml
D docs/topics/impala_data_sources.xml
D docs/topics/impala_drop_data_source.xml
D ext-data-source/.gitignore
D ext-data-source/CMakeLists.txt
D ext-data-source/api/pom.xml
D 
ext-data-source/api/src/main/java/org/apache/impala/extdatasource/util/SerializationUtils.java
D 
ext-data-source/api/src/main/java/org/apache/impala/extdatasource/v1/ExternalDataSource.java
D ext-data-source/pom.xml
D ext-data-source/sample/pom.xml
D 
ext-data-source/sample/src/main/java/org/apache/impala/extdatasource/sample/EchoDataSource.java
D ext-data-source/test/pom.xml
D 
ext-data-source/test/src/main/java/org/apache/impala/extdatasource/AllTypesDataSource.java
M fe/CMakeLists.txt
M fe/pom.xml
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
D fe/src/main/java/org/apache/impala/analysis/CreateDataSrcStmt.java
D fe/src/main/java/org/apache/impala/analysis/CreateTableDataSrcStmt.java
D fe/src/main/java/org/apache/impala/analysis/DropDataSrcStmt.java
M fe/src/main/java/org/apache/impala/analysis/PrivilegeSpec.java
D fe/src/main/java/org/apache/impala/analysis/ShowDataSrcsStmt.java
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
D fe/src/main/java/org/apache/impala/catalog/DataSource.java
D fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java
M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
D fe/src/main/java/org/apache/impala/extdatasource/ApiVersion.java
D 
fe/src/main/java/org/apache/impala/extdatasource/ExternalDataSourceExecutor.java
D fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
M fe/src/main/jflex/sql-scanner.flex
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java
M fe/src/test/java/org/apache/impala/analysis/ParserTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java

[Impala-ASF-CR] IMPALA-6449: Use CLOCK MONOTONIC in ConditionVariable

2018-02-02 Thread Jim Apple (Code Review)
Jim Apple has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9158 )

Change subject: IMPALA-6449: Use CLOCK_MONOTONIC in ConditionVariable
..


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/9158/1/be/src/util/condition-variable.h
File be/src/util/condition-variable.h:

http://gerrit.cloudera.org:8080/#/c/9158/1/be/src/util/condition-variable.h@39
PS1, Line 39: pthread_condattr_setclock(, CLOCK_MONOTONIC);
> CLOCK_MONOTONIC_COARSE may be a bit too coarse for our purpose here given s
WFM



--
To view, visit http://gerrit.cloudera.org:8080/9158
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I81611cfd5e7c5347203fe7fa6b0f615602257f87
Gerrit-Change-Number: 9158
Gerrit-PatchSet: 1
Gerrit-Owner: Michael Ho 
Gerrit-Reviewer: Jim Apple 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Comment-Date: Fri, 02 Feb 2018 16:40:21 +
Gerrit-HasComments: Yes


[Impala-ASF-CR] IMPALA-5269: Fix issue with final line of query followed by a comment

2018-02-02 Thread Anonymous Coward (Code Review)
fwij...@cloudera.com has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/9191


Change subject: IMPALA-5269: Fix issue with final line of query followed by a 
comment
..

IMPALA-5269: Fix issue with final line of query followed by a comment

The patch is to fix a statement that is terminated with SQL terminated
delimiter, i.e. ; followed by a comment.

For example:

Before:
select 1 + 1; -- comment\n;

After:
select 1 + 1; -- comment

Change-Id: I54f9a8f65214023520eaa010fc462a663d02d258
---
M shell/impala_shell.py
M tests/shell/test_shell_interactive.py
2 files changed, 21 insertions(+), 0 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/91/9191/2
--
To view, visit http://gerrit.cloudera.org:8080/9191
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I54f9a8f65214023520eaa010fc462a663d02d258
Gerrit-Change-Number: 9191
Gerrit-PatchSet: 2
Gerrit-Owner: fwij...@cloudera.com


[Impala-ASF-CR] IMPALA-6113: Skip row groups with predicates on NULL columns

2018-02-02 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9140 )

Change subject: IMPALA-6113: Skip row groups with predicates on NULL columns
..


Patch Set 7:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1862/


--
To view, visit http://gerrit.cloudera.org:8080/9140
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I141317af0e0df30da8f220b29b0bfba364f40ddf
Gerrit-Change-Number: 9140
Gerrit-PatchSet: 7
Gerrit-Owner: Gabor Kaszab 
Gerrit-Reviewer: Gabor Kaszab 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: anujphadke 
Gerrit-Comment-Date: Fri, 02 Feb 2018 16:19:31 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-6113: Skip row groups with predicates on NULL columns

2018-02-02 Thread Tim Armstrong (Code Review)
Tim Armstrong has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9140 )

Change subject: IMPALA-6113: Skip row groups with predicates on NULL columns
..


Patch Set 7: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/9140
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I141317af0e0df30da8f220b29b0bfba364f40ddf
Gerrit-Change-Number: 9140
Gerrit-PatchSet: 7
Gerrit-Owner: Gabor Kaszab 
Gerrit-Reviewer: Gabor Kaszab 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: anujphadke 
Gerrit-Comment-Date: Fri, 02 Feb 2018 16:18:48 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-6113: Skip row groups with predicates on NULL columns

2018-02-02 Thread Tim Armstrong (Code Review)
Tim Armstrong has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9140 )

Change subject: IMPALA-6113: Skip row groups with predicates on NULL columns
..


Patch Set 6: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/9140
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I141317af0e0df30da8f220b29b0bfba364f40ddf
Gerrit-Change-Number: 9140
Gerrit-PatchSet: 6
Gerrit-Owner: Gabor Kaszab 
Gerrit-Reviewer: Gabor Kaszab 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: anujphadke 
Gerrit-Comment-Date: Fri, 02 Feb 2018 16:18:42 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-5717: Support for ORC data files

2018-02-02 Thread Quanlong Huang (Code Review)
Quanlong Huang has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9134 )

Change subject: IMPALA-5717: Support for ORC data files
..


Patch Set 3:

(24 comments)

Thanks, Tim! Your comments are really useful!
If we finally decide to move the ORC library into native-tool-chain project, is 
there any document about how to contribute to this? I think I may need the ORC 
library merged first than I can use it like other tools.

There're still comments I haven't deal with. Forgive me to reply them later.

http://gerrit.cloudera.org:8080/#/c/9134/3/be/src/exec/hdfs-orc-scanner.h
File be/src/exec/hdfs-orc-scanner.h:

http://gerrit.cloudera.org:8080/#/c/9134/3/be/src/exec/hdfs-orc-scanner.h@28
PS3, Line 28: class CollectionValueBuilder;
> Not needed?
Yeah, just added it when I try to support complex types. Will remove it.


http://gerrit.cloudera.org:8080/#/c/9134/3/be/src/exec/hdfs-orc-scanner.h@259
PS3, Line 259: ProcessFileTail
> It might be helpful to define what the "footer", "file tail" and "postscrip
I was confused as well at first :)
They're concepts in ORC. Here is their definitions: 
https://orc.apache.org/docs/file-tail.html
You can also find them in be/src/orc/orc_proto.proto


http://gerrit.cloudera.org:8080/#/c/9134/3/be/src/exec/hdfs-orc-scanner.cc
File be/src/exec/hdfs-orc-scanner.cc:

http://gerrit.cloudera.org:8080/#/c/9134/3/be/src/exec/hdfs-orc-scanner.cc@1
PS3, Line 1: // Copyright 2012 Cloudera Inc.
> Don't need cloudera copyrights!
Done


http://gerrit.cloudera.org:8080/#/c/9134/3/be/src/exec/hdfs-orc-scanner.cc@24
PS3, Line 24: #include "common/object-pool.h"
> Many of these headers look unused.
Yes, will remove them


http://gerrit.cloudera.org:8080/#/c/9134/3/be/src/exec/hdfs-orc-scanner.cc@53
PS3, Line 53: using boost::algorithm::split;
> Some of these boost "using" declarations don't seem to be needed.
Done


http://gerrit.cloudera.org:8080/#/c/9134/3/be/src/exec/hdfs-orc-scanner.cc@59
PS3, Line 59: DEFINE_double(orc_min_filter_reject_ratio, 0.1, "(Advanced) If 
the percentage of "
> I don't know why we made this flag option parquet-specific. Having an optio
agree with you. There're many logics in the parquet scanner that can share with 
the ORC scanner. Not only this var, but also functions like IssueInitialRanges 
and FindFooterSplit.


http://gerrit.cloudera.org:8080/#/c/9134/3/be/src/exec/hdfs-orc-scanner.cc@73
PS3, Line 73:   for (int i = 0; i < files.size(); ++i) {
> Can we convert this to a range for? We generally prefer that in new code.
Done


http://gerrit.cloudera.org:8080/#/c/9134/3/be/src/exec/hdfs-orc-scanner.cc@151
PS3, Line 151:   mem_tracker_->CloseAndUnregisterFromParent();
> We only want to use CloseAndUnregisterFromParent() for the query-level MemT
Just add the comment since I made it crashed when use Close at first...
I can remove the comment if you're all clear of it :)


http://gerrit.cloudera.org:8080/#/c/9134/3/be/src/exec/hdfs-orc-scanner.cc@157
PS3, Line 157: std::
> Don't need std:: prefix. We generally prefer avoiding it when it isn't need
We need this prefix because this class has a free function (see below) as well.


http://gerrit.cloudera.org:8080/#/c/9134/3/be/src/exec/hdfs-orc-scanner.cc@162
PS3, Line 162:   if (ImpaladMetrics::MEM_POOL_TOTAL_BYTES != nullptr) {
> You should be able to assume it's non-NULL. A lot of older code checks if m
So the non-NULL checks in mem-pool.cc are redundant too? I learn this from the 
impala::MemPool implememtation.

I found this metric useful when I ran test_failpoints.py individually. It won't 
come back to zero in 2 minutes. So I found bug mentioned in IMPALA-6423


http://gerrit.cloudera.org:8080/#/c/9134/3/be/src/exec/hdfs-orc-scanner.cc@196
PS3, Line 196: void HdfsOrcScanner::ScanRangeInputStream::read(void* buf, 
uint64_t length,
> It's unfortunate that the ORC code was designed to issue only synchronous r
yes, quite a pity. So I hope we can include the ORC codes and modify the logics 
of getting input.


http://gerrit.cloudera.org:8080/#/c/9134/3/be/src/exec/hdfs-orc-scanner.cc@223
PS3, Line 223: memset(buf, 0, length);
> Is the memset needed? If so, should document why in a comment.
Just let the orc-reader to throw an exception for parse error if it read this 
later. Not needed actually since we throw the following exception immediately.


http://gerrit.cloudera.org:8080/#/c/9134/3/be/src/exec/hdfs-orc-scanner.cc@224
PS3, Line 224: throw std::runtime_error("Cannot read from file: " + 
status.GetDetail());
> The fact we need to use exceptions is unfortunate. I need to think a bit mo
I don't like exceptions as well. One solution is to insert the stream context 
into the orc-reader, and check if cancel in the loops inside it.

This need to modify codes in the ORC-reader so I haven't started yet. If you 
guys decide to include the ORC codes, I can implement this.



[Impala-ASF-CR] IMPALA-6416: extend Thread::Create to track instance id

2018-02-02 Thread Zoltan Borok-Nagy (Code Review)
Zoltan Borok-Nagy has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9053 )

Change subject: IMPALA-6416: extend Thread::Create to track instance id
..


Patch Set 2:

(7 comments)

http://gerrit.cloudera.org:8080/#/c/9053/2/be/src/common/thread-debug-info.h
File be/src/common/thread-debug-info.h:

http://gerrit.cloudera.org:8080/#/c/9053/2/be/src/common/thread-debug-info.h@79
PS2, Line 79: ExtractInfoFromParent
> Extract... reads like something will be returned. How about "SetParentInfo"
yeah I like that. Done.


http://gerrit.cloudera.org:8080/#/c/9053/2/be/src/common/thread-debug-info.h@80
PS2, Line 80: if (parent != nullptr) {
> We usually early return instead of scoping the whole function, i.e. if (par
Done


http://gerrit.cloudera.org:8080/#/c/9053/2/be/src/common/thread-debug-info.h@93
PS2, Line 93: t
> nit: capital T
Done


http://gerrit.cloudera.org:8080/#/c/9053/2/be/src/common/thread-debug-info.h@121
PS2, Line 121:   struct ParentTdi {
> Can you add a comment to this struct, too? The test has a ThreadDebugInfo p
Added comment, and renamed it to "ParentInfo".


http://gerrit.cloudera.org:8080/#/c/9053/2/be/src/common/thread-debug-info.h@124
PS2, Line 124: char thread_name_[THREAD_NAME_SIZE] = {};
> I wonder if we can streamline the selective duplication of the parent threa
I chose the first alternative, i.e. to store the system thread id and a pointer 
to the parent TDI.

If it is a common case that the parent exits, we can switch to the second 
alternative, but also adding a pointer to the parent for convenience.


http://gerrit.cloudera.org:8080/#/c/9053/2/be/src/common/thread-debug-info.h@128
PS2, Line 128:
> nit: I think this newline and the next one don't add much to readability.
Done


http://gerrit.cloudera.org:8080/#/c/9053/2/be/src/util/thread.h
File be/src/util/thread.h:

http://gerrit.cloudera.org:8080/#/c/9053/2/be/src/util/thread.h@191
PS2, Line 191: const ThreadDebugInfo* parent_thread_info
> This should go to the front (behind category or functor) now since it's str
oh I see. I placed it behind functor.



--
To view, visit http://gerrit.cloudera.org:8080/9053
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I27de2962cf0b224c17b685d77dcba3bf2e9db187
Gerrit-Change-Number: 9053
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Comment-Date: Fri, 02 Feb 2018 14:38:14 +
Gerrit-HasComments: Yes


[Impala-ASF-CR] IMPALA-6416: extend Thread::Create to track instance id

2018-02-02 Thread Zoltan Borok-Nagy (Code Review)
Hello Lars Volker, Tim Armstrong,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/9053

to look at the new patch set (#3).

Change subject: IMPALA-6416: extend Thread::Create to track instance id
..

IMPALA-6416: extend Thread::Create to track instance id

This commit builds upon IMPALA-3703. Each thread that
was created through Thread::Create() has a ThreadDebugInfo
object on the stack frame of Thread::SuperviseThread().
This object has stack allocated char buffers that can be
read during a debug session even if we only have minidumps.

However, with the old solution ThreadDebugInfo::instance_id
was set manually for each thread. It is too easy to forget
to set instance_id every time we create a new thread.

This commit has the assumption that if a thread has an
instance id associated, then the threads spawned by it will
always work on the same instance id. In Thread::StartThread
the parent thread passes its ThreadDebugInfo object to
its child who copies the instance id and also saves its
parent's name.

This means if we set ThreadDebugInfo::instance_id in some
"root thread", then all descendant threads will annotate
themselves with the instance id automatically. Since threads
also record their parent's name, it might be also possible
to reconstruct the thread creation graph.

With GDB I tested if it copies the instance id at every
place where we previously needed to set it manually.

I added an automated test to thread-debug-info-test.cc

Change-Id: I27de2962cf0b224c17b685d77dcba3bf2e9db187
---
M be/src/common/thread-debug-info-test.cc
M be/src/common/thread-debug-info.cc
M be/src/common/thread-debug-info.h
M be/src/exec/blocking-join-node.cc
M be/src/exec/hdfs-scan-node.cc
M be/src/runtime/fragment-instance-state.cc
M be/src/util/thread.cc
M be/src/util/thread.h
8 files changed, 81 insertions(+), 25 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/9053/3
--
To view, visit http://gerrit.cloudera.org:8080/9053
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I27de2962cf0b224c17b685d77dcba3bf2e9db187
Gerrit-Change-Number: 9053
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Zoltan Borok-Nagy 


[Impala-ASF-CR] IMPALA-6113: Skip row groups with predicates on NULL columns

2018-02-02 Thread Zoltan Borok-Nagy (Code Review)
Zoltan Borok-Nagy has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9140 )

Change subject: IMPALA-6113: Skip row groups with predicates on NULL columns
..


Patch Set 6: Code-Review+1

LGTM!


--
To view, visit http://gerrit.cloudera.org:8080/9140
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I141317af0e0df30da8f220b29b0bfba364f40ddf
Gerrit-Change-Number: 9140
Gerrit-PatchSet: 6
Gerrit-Owner: Gabor Kaszab 
Gerrit-Reviewer: Gabor Kaszab 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: anujphadke 
Gerrit-Comment-Date: Fri, 02 Feb 2018 13:28:49 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-5237: Support a quoted string in date/time format

2018-02-02 Thread Gabor Kaszab (Code Review)
Gabor Kaszab has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/8508 )

Change subject: IMPALA-5237: Support a quoted string in date/time format
..


Patch Set 8: Code-Review+1

Thanks for taking care of this. For me it's fine.


--
To view, visit http://gerrit.cloudera.org:8080/8508
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie34055ac695748bcfb110bfa6ed5308f469ea178
Gerrit-Change-Number: 8508
Gerrit-PatchSet: 8
Gerrit-Owner: Kim Jin Chul 
Gerrit-Reviewer: Attila Jeges 
Gerrit-Reviewer: Gabor Kaszab 
Gerrit-Reviewer: Kim Jin Chul 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 02 Feb 2018 10:09:16 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-6113: Skip row groups with predicates on NULL columns

2018-02-02 Thread Gabor Kaszab (Code Review)
Gabor Kaszab has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9140 )

Change subject: IMPALA-6113: Skip row groups with predicates on NULL columns
..


Patch Set 6:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/9140/3/be/src/exec/parquet-column-stats.cc
File be/src/exec/parquet-column-stats.cc:

http://gerrit.cloudera.org:8080/#/c/9140/3/be/src/exec/parquet-column-stats.cc@132
PS3, Line 132: bool ColumnStatsBase::ReadNullCountStat(const 
parquet::ColumnChunk& col_chunk,
> Why do we need to return a "const int64_t*" to the caller?
Thanks Tim for the explanation. Makes sense.
Done.


http://gerrit.cloudera.org:8080/#/c/9140/3/tests/query_test/test_parquet_stats.py
File tests/query_test/test_parquet_stats.py:

http://gerrit.cloudera.org:8080/#/c/9140/3/tests/query_test/test_parquet_stats.py@49
PS3, Line 49: self.run_test_case('QueryTest/parquet-stats', vector, 
use_db=unique_database)
> Sorry about the confusion. I thought that adding the create table statement
Thanks Tim, Anuj!
Dropped the $DATABASE prefix.



--
To view, visit http://gerrit.cloudera.org:8080/9140
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I141317af0e0df30da8f220b29b0bfba364f40ddf
Gerrit-Change-Number: 9140
Gerrit-PatchSet: 6
Gerrit-Owner: Gabor Kaszab 
Gerrit-Reviewer: Gabor Kaszab 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: anujphadke 
Gerrit-Comment-Date: Fri, 02 Feb 2018 09:20:17 +
Gerrit-HasComments: Yes


[Impala-ASF-CR] IMPALA-6113: Skip row groups with predicates on NULL columns

2018-02-02 Thread Gabor Kaszab (Code Review)
Hello Lars Volker, Zoltan Borok-Nagy, anujphadke, Tim Armstrong,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/9140

to look at the new patch set (#6).

Change subject: IMPALA-6113: Skip row groups with predicates on NULL columns
..

IMPALA-6113: Skip row groups with predicates on NULL columns

Based on the existing Parquet column chunk level statistics null_count,
Impala's Parquet scanner is enhanced to skip an entire row group if the
null_count statistics indicate that all the values under the predicated
column are NULL as we wouldn't get any result rows from that row group
anyway.

Change-Id: I141317af0e0df30da8f220b29b0bfba364f40ddf
---
M be/src/exec/hdfs-parquet-scanner.cc
M be/src/exec/parquet-column-stats.cc
M be/src/exec/parquet-column-stats.h
M testdata/workloads/functional-query/queries/QueryTest/parquet-stats.test
4 files changed, 59 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/9140/6
--
To view, visit http://gerrit.cloudera.org:8080/9140
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I141317af0e0df30da8f220b29b0bfba364f40ddf
Gerrit-Change-Number: 9140
Gerrit-PatchSet: 6
Gerrit-Owner: Gabor Kaszab 
Gerrit-Reviewer: Gabor Kaszab 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: anujphadke