date:20200807

Hello Attila Jeges, Csaba Ringhofer, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/16301

to look at the new patch set (#3).

Change subject: IMPALA-10054: Fix flakiness in 
test_multiple_sort_run_bytes_limits
..

IMPALA-10054: Fix flakiness in test_multiple_sort_run_bytes_limits

test_multiple_sort_run_bytes_limits seems to become flaky in
ubuntu-16.04-dockerised-tests. This flakiness may come from accuracy
change in query estimates, the mem_limit specified in the test does not
fit anymore, or query concurrency in mini cluster that may disturb
expected memory allocation. This patch remove the second test case of
test_multiple_sort_run_bytes_limits due to variability in several test
run in the past. It does not compromise the test itself because the
basic feature of sort_run_bytes_limit is still verifiable by the
remaining test cases. The assertion is also changed a bit to allow
easier debugging in case test regression occurs again in the future.

Testing:
- Run and pass test_sort.py

Change-Id: I84a8b579c943cddba4432cf183f7f002ef8ec6ad
---
M tests/query_test/test_sort.py
1 file changed, 5 insertions(+), 5 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/16301/3
--
To view, visit http://gerrit.cloudera.org:8080/16301
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I84a8b579c943cddba4432cf183f7f002ef8ec6ad
Gerrit-Change-Number: 16301
Gerrit-PatchSet: 3
Gerrit-Owner: Riza Suminto 
Gerrit-Reviewer: Attila Jeges 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Riza Suminto

[Impala-ASF-CR] IMPALA-10017: Implement ds kll union() function

2020-08-07 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16267 )

Change subject: IMPALA-10017: Implement ds_kll_union() function
..


Patch Set 8: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6252/


--
To view, visit http://gerrit.cloudera.org:8080/16267
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I020aea28d36f9b6ef9fb57c08411f2170f5c24bf
Gerrit-Change-Number: 16267
Gerrit-PatchSet: 8
Gerrit-Owner: Gabor Kaszab 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Gabor Kaszab 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Sat, 08 Aug 2020 01:16:46 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10054: Fix flakiness in test multiple sort run bytes limits

Riza Suminto has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16301 )

Change subject: IMPALA-10054: Fix flakiness in 
test_multiple_sort_run_bytes_limits
..


Patch Set 2:

I reran the dockerised test with the new parameter and still failed to get the 
expected result.
https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/2921/testReport/junit/query_test.test_sort/TestQueryFullSort/test_multiple_sort_run_bytes_limits_protocol__beeswax___exec_optionbatch_size___0___num_nodes___0___disable_codegen_rows_threshold___0___disable_codegen___False___abort_on_error___1___exec_single_node_rows_threshold___0table_format__parquet_none_/

I thought it might overspill (SpilledRuns > 3), but turns out, the query finish 
with only 2 SpilledRuns.
It seems test case 2 is hard to get right since other concurrent running 
queries might disturb the memory allocation for this test.
At this point, I lean towards removing test case 2 to stabilize the build going 
forward.
In my opinion, test case 1 and 3 is sufficient to verify sort_run_bytes_limit 
feature.


--
To view, visit http://gerrit.cloudera.org:8080/16301
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I84a8b579c943cddba4432cf183f7f002ef8ec6ad
Gerrit-Change-Number: 16301
Gerrit-PatchSet: 2
Gerrit-Owner: Riza Suminto 
Gerrit-Reviewer: Attila Jeges 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Riza Suminto 
Gerrit-Comment-Date: Sat, 08 Aug 2020 01:15:15 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9859: Full ACID Milestone 4: Part 2 Reading modified tables (complex types)

2020-08-07 Thread Aman Sinha (Code Review)

Aman Sinha has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16228 )

Change subject: IMPALA-9859: Full ACID Milestone 4: Part 2 Reading modified 
tables (complex types)
..


Patch Set 6: Code-Review+1

LGTM.  Would be good to do a perf evaluation with the new plan when the rewrite 
is applied.


--
To view, visit http://gerrit.cloudera.org:8080/16228
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8b2c6cd3d87c452c5b96a913b14c90ada78d4c6f
Gerrit-Change-Number: 16228
Gerrit-PatchSet: 6
Gerrit-Owner: Zoltan Borok-Nagy 
Gerrit-Reviewer: Aman Sinha 
Gerrit-Reviewer: Gabor Kaszab 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Comment-Date: Sat, 08 Aug 2020 00:56:10 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10043: Keep more logs when using EE TEST SHARDS

2020-08-07 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16297 )

Change subject: IMPALA-10043: Keep more logs when using EE_TEST_SHARDS
..


Patch Set 2: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/16297
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie011b892cd2eb1a528012ec5600e72e44f281a88
Gerrit-Change-Number: 16297
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Laszlo Gaal 
Gerrit-Comment-Date: Sat, 08 Aug 2020 00:34:34 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10016: Remove unnecessary jar dependencies

2020-08-07 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16311 )

Change subject: IMPALA-10016: Remove unnecessary jar dependencies
..


Patch Set 1:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/6841/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/16311
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iadbb6142466f73f067dd7cf9d401ff81145c74cc
Gerrit-Change-Number: 16311
Gerrit-PatchSet: 1
Gerrit-Owner: Sahil Takiar 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Sat, 08 Aug 2020 00:32:43 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10016: Remove unnecessary jar dependencies

2020-08-07 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16311 )

Change subject: IMPALA-10016: Remove unnecessary jar dependencies
..


Patch Set 1:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/16311/1/fe/src/main/java/org/apache/impala/service/JniFrontend.java
File fe/src/main/java/org/apache/impala/service/JniFrontend.java:

http://gerrit.cloudera.org:8080/#/c/16311/1/fe/src/main/java/org/apache/impala/service/JniFrontend.java@768
PS1, Line 768: String domainSocketPath = 
conf.getTrimmed(HdfsClientConfigKeys.DFS_DOMAIN_SOCKET_PATH_KEY,
line too long (94 > 90)


http://gerrit.cloudera.org:8080/#/c/16311/1/fe/src/main/java/org/apache/impala/util/FsPermissionChecker.java
File fe/src/main/java/org/apache/impala/util/FsPermissionChecker.java:

http://gerrit.cloudera.org:8080/#/c/16311/1/fe/src/main/java/org/apache/impala/util/FsPermissionChecker.java@42
PS1, Line 42: import static 
org.apache.hadoop.hdfs.client.HdfsClientConfigKeys.DeprecatedKeys.DFS_PERMISSIONS_SUPERUSERGROUP_KEY;
line too long (115 > 90)


http://gerrit.cloudera.org:8080/#/c/16311/1/fe/src/test/java/org/apache/impala/service/JniFrontendTest.java
File fe/src/test/java/org/apache/impala/service/JniFrontendTest.java:

http://gerrit.cloudera.org:8080/#/c/16311/1/fe/src/test/java/org/apache/impala/service/JniFrontendTest.java@105
PS1, Line 105: 
HdfsClientConfigKeys.DFS_CLIENT_USE_LEGACY_BLOCKREADERLOCAL_DEFAULT)).thenReturn(false);
line too long (96 > 90)



--
To view, visit http://gerrit.cloudera.org:8080/16311
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iadbb6142466f73f067dd7cf9d401ff81145c74cc
Gerrit-Change-Number: 16311
Gerrit-PatchSet: 1
Gerrit-Owner: Sahil Takiar 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Sat, 08 Aug 2020 00:05:09 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10016: Remove unnecessary jar dependencies

Sahil Takiar has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/16311


Change subject: IMPALA-10016: Remove unnecessary jar dependencies
..

IMPALA-10016: Remove unnecessary jar dependencies

Remove the dependency on hadoop-hdfs, this jar file contains the core
code for implementing HDFS, and thus pulls in a bunch of unnecessary
transitive dependencies. Impala currently only requires this jar for
some configuration key names. Most of these configuration key names have
been moved to the appropriate HDFS client jars, and some others are
deprecated altogether. Removing this jar required making a few code
changes to move the location of the referenced configuration keys.

Removes all transitive Kafka dependencies from the Apache Ranger
dependency. Previously, Impala only excluded Kafka jars with binary
version kafka_2.11, however, it seems the Ranger recently upgraded the
dependency version to kafka_2.12. Now all Kafka dependencies are
excluded, regardless of artifact name.

Removes all transitive dependencies from the Apache Ozone dependency.
Impala has a dependency on the Ozone client shaded-jar, which already
includes all required transitive dependencies. For some reason, Ozone
still pulls in some transitive dependencies even though they are not
needed.

Made some other minor cleanup / improvements in the fe/pom.xml file.

This saves about 70 MB of space in the Docker images.

Testing:
* Ran core tests

Change-Id: Iadbb6142466f73f067dd7cf9d401ff81145c74cc
---
M fe/pom.xml
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
M fe/src/main/java/org/apache/impala/util/FsPermissionChecker.java
M fe/src/main/java/org/apache/impala/util/HdfsCachingUtil.java
M fe/src/test/java/org/apache/impala/service/JniFrontendTest.java
5 files changed, 40 insertions(+), 102 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/16311/1
--
To view, visit http://gerrit.cloudera.org:8080/16311
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Iadbb6142466f73f067dd7cf9d401ff81145c74cc
Gerrit-Change-Number: 16311
Gerrit-PatchSet: 1
Gerrit-Owner: Sahil Takiar

[Impala-ASF-CR] IMPALA-10039: Fixed Expr-test crash caused by thread unsafe function

2020-08-07 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/16299 )

Change subject: IMPALA-10039: Fixed Expr-test crash caused by thread unsafe 
function
..

IMPALA-10039: Fixed Expr-test crash caused by thread unsafe function

Recent patch for IMPALA-5746 registers a callback function for the
updating of cluster membership. The callback function cancels the
queries scheduled by the failed coordinators. This callback function
was called during Expr-test and caused crash.
This patch checks if the process running for tests and only registers
the callback function if it's not running for BE/FE tests.

Testing:
 - The issue could be reproduced by running expr-test for 10-20
   iterations. Verified the fixing by running expr-test over 1000
   iterations without crash.
 - Passed TestProcessFailures::test_kill_coordinator.
 - Passed core tests.

Change-Id: I85245bf4bffb469913d53741847e67773b7d4627
Reviewed-on: http://gerrit.cloudera.org:8080/16299
Reviewed-by: Thomas Tauber-Marshall 
Tested-by: Impala Public Jenkins 
---
M be/src/runtime/exec-env.cc
1 file changed, 1 insertion(+), 1 deletion(-)

Approvals:
  Thomas Tauber-Marshall: Looks good to me, approved
  Impala Public Jenkins: Verified

--
To view, visit http://gerrit.cloudera.org:8080/16299
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I85245bf4bffb469913d53741847e67773b7d4627
Gerrit-Change-Number: 16299
Gerrit-PatchSet: 5
Gerrit-Owner: Wenzhe Zhou 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Reviewer: Wenzhe Zhou

[Impala-ASF-CR] IMPALA-10039: Fixed Expr-test crash caused by thread unsafe function

2020-08-07 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16299 )

Change subject: IMPALA-10039: Fixed Expr-test crash caused by thread unsafe 
function
..


Patch Set 4: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/16299
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I85245bf4bffb469913d53741847e67773b7d4627
Gerrit-Change-Number: 16299
Gerrit-PatchSet: 4
Gerrit-Owner: Wenzhe Zhou 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Reviewer: Wenzhe Zhou 
Gerrit-Comment-Date: Fri, 07 Aug 2020 23:50:01 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10034: Add remaining TPC-DS queries to workload.

2020-08-07 Thread Shant Hovsepian (Code Review)

Shant Hovsepian has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16280 )

Change subject: IMPALA-10034: Add remaining TPC-DS queries to workload.
..


Patch Set 4:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/16280/4/testdata/workloads/tpcds/queries/tpcds-decimal_v2-q23-1.test
File testdata/workloads/tpcds/queries/tpcds-decimal_v2-q23-1.test:

http://gerrit.cloudera.org:8080/#/c/16280/4/testdata/workloads/tpcds/queries/tpcds-decimal_v2-q23-1.test@33
PS4, Line 33:   select  sum(sales)
> Is there any specific reason why we have two tests for query 23 that are al
The TPCDS query is written as two queries in one file. We separate it out like 
this to make the test diffing easier. The slight difference is just how TPC-DS 
designed this workload.


http://gerrit.cloudera.org:8080/#/c/16280/4/testdata/workloads/tpcds/queries/tpcds-decimal_v2-q24-1.test
File testdata/workloads/tpcds/queries/tpcds-decimal_v2-q24-1.test:

http://gerrit.cloudera.org:8080/#/c/16280/4/testdata/workloads/tpcds/queries/tpcds-decimal_v2-q24-1.test@45
PS4, Line 45: where i_color = 'peach'
> Is there any specific reason why we have two tests for query 24 that are al
Same as above, the TPC-DS specification has it written as two semi colon 
separated queries.



--
To view, visit http://gerrit.cloudera.org:8080/16280
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id5436689390f149694f14e6da1df624de4f5f7ad
Gerrit-Change-Number: 16280
Gerrit-PatchSet: 4
Gerrit-Owner: Shant Hovsepian 
Gerrit-Reviewer: Aman Sinha 
Gerrit-Reviewer: David Rorke 
Gerrit-Reviewer: Fang-Yu Rao 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Shant Hovsepian 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 07 Aug 2020 22:25:00 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10034: Add remaining TPC-DS queries to workload.

2020-08-07 Thread Fang-Yu Rao (Code Review)

Fang-Yu Rao has posted comments on this change. (
http://gerrit.cloudera.org:8080/16280 )

Change subject: IMPALA-10034: Add remaining TPC-DS queries to workload.
..

Patch Set 4:

(2 comments)

Thanks Shant for adding the remaining TPC-DS queries! I only have two very
minor questions out of curiosity and do not have any other comment.

http://gerrit.cloudera.org:8080/#/c/16280/4/testdata/workloads/tpcds/queries/tpcds-decimal_v2-q23-1.test
File testdata/workloads/tpcds/queries/tpcds-decimal_v2-q23-1.test:

http://gerrit.cloudera.org:8080/#/c/16280/4/testdata/workloads/tpcds/queries/tpcds-decimal_v2-q23-1.test@33
PS4, Line 33: select sum(sales)
Is there any specific reason why we have two tests for query 23 that are almost
the same? It seems the only difference is the selected column(s). In
TPCDS-Q23-2, the selected columns are 1) c_last_name, 2) c_first_name, and 3)
sales.

http://gerrit.cloudera.org:8080/#/c/16280/4/testdata/workloads/tpcds/queries/tpcds-decimal_v2-q24-1.test
File testdata/workloads/tpcds/queries/tpcds-decimal_v2-q24-1.test:

http://gerrit.cloudera.org:8080/#/c/16280/4/testdata/workloads/tpcds/queries/tpcds-decimal_v2-q24-1.test@45
PS4, Line 45: where i_color = 'peach'
Is there any specific reason why we have two tests for query 24 that are almost
the same? It seems the only difference is this condition. In TPCDS-Q24-2, the
condition is i_color = 'saddle'.

--
To view, visit http://gerrit.cloudera.org:8080/16280
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id5436689390f149694f14e6da1df624de4f5f7ad
Gerrit-Change-Number: 16280
Gerrit-PatchSet: 4
Gerrit-Owner: Shant Hovsepian
Gerrit-Reviewer: Aman Sinha
Gerrit-Reviewer: David Rorke
Gerrit-Reviewer: Fang-Yu Rao
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Shant Hovsepian
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Fri, 07 Aug 2020 22:04:24 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4364: Query option to refresh updated partitions

2020-08-07 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16308 )

Change subject: IMPALA-4364: Query option to refresh updated partitions
..


Patch Set 1:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/6840/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/16308
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I50e8680509f4eb0712e7bb3de44df5f2952179af
Gerrit-Change-Number: 16308
Gerrit-PatchSet: 1
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 07 Aug 2020 21:55:11 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-4364: Query option to refresh updated partitions

2020-08-07 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16308 )

Change subject: IMPALA-4364: Query option to refresh updated partitions
..


Patch Set 1:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/16308/1/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:

http://gerrit.cloudera.org:8080/#/c/16308/1/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@1326
PS1, Line 1326:  * the given partition.
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/16308/1/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@1388
PS1, Line 1388: loadTimeForFileMdNs_ += 
loadFileMetadataForPartitions(client_, partitionsToLoadFiles,
line too long (93 > 90)


http://gerrit.cloudera.org:8080/#/c/16308/1/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@1483
PS1, Line 1483: public long loadUpdatedPartitions(Map updatedPartBuilders)
line too long (93 > 90)



--
To view, visit http://gerrit.cloudera.org:8080/16308
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I50e8680509f4eb0712e7bb3de44df5f2952179af
Gerrit-Change-Number: 16308
Gerrit-PatchSet: 1
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 07 Aug 2020 21:32:34 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4364: Query option to refresh updated partitions

2020-08-07 Thread Vihang Karajgaonkar (Code Review)

Vihang Karajgaonkar has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/16308


Change subject: IMPALA-4364: Query option to refresh updated partitions
..

IMPALA-4364: Query option to refresh updated partitions

This patch introduces a new boolean query option
ENABLE_REFRESH_UPDATED_PARTITIONS. When this query option is set
the refresh table command reloads the partitions which have been
modified in HMS in addition to adding [removing] the new [removed]
partitions.

In order to do this the refresh table command needs to fetch all
the partitions instead of the just the partition names which can
cause the performance of refresh table to degrade when the query
option is set. However, for certain use-cases there is no way
currently the refresh table doesn't detect changed partitions.
For instance, if certain partition locations have been changed,
a refresh table will not update those partitions.

Testing:
1. Added a new test which sets the query option and makes sure
that the updated partitions from hive are reloaded after refresh
table command.
2. [In-progress] Running exhaustive tests with the patch.

Change-Id: I50e8680509f4eb0712e7bb3de44df5f2952179af
---
M be/src/service/client-request-state.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/CatalogService.thrift
M common/thrift/ImpalaInternalService.thrift
M common/thrift/ImpalaService.thrift
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java
M fe/src/main/java/org/apache/impala/catalog/HBaseTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/IncompleteTable.java
M fe/src/main/java/org/apache/impala/catalog/KuduTable.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/catalog/TableLoader.java
M fe/src/main/java/org/apache/impala/catalog/View.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M tests/metadata/test_reset_metadata.py
20 files changed, 443 insertions(+), 125 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/08/16308/1
--
To view, visit http://gerrit.cloudera.org:8080/16308
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I50e8680509f4eb0712e7bb3de44df5f2952179af
Gerrit-Change-Number: 16308
Gerrit-PatchSet: 1
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Tim Armstrong

[Impala-ASF-CR] IMPALA-10017: Implement ds kll union() function

2020-08-07 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16267 )

Change subject: IMPALA-10017: Implement ds_kll_union() function
..


Patch Set 7:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/6839/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/16267
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I020aea28d36f9b6ef9fb57c08411f2170f5c24bf
Gerrit-Change-Number: 16267
Gerrit-PatchSet: 7
Gerrit-Owner: Gabor Kaszab 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Gabor Kaszab 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Fri, 07 Aug 2020 21:28:17 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9995 Fix test alloc fail failed case on aarch64

2020-08-07 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16307 )

Change subject: IMPALA-9995 Fix test_alloc_fail failed case on aarch64
..


Patch Set 1:

Build Failed

https://jenkins.impala.io/job/gerrit-code-review-checks/6838/ : Initial code 
review checks failed. See linked job for details on the failure.


--
To view, visit http://gerrit.cloudera.org:8080/16307
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9a5a4ba19b225bdb4f18a68d6d9cb2c2d16f91fd
Gerrit-Change-Number: 16307
Gerrit-PatchSet: 1
Gerrit-Owner: Anonymous Coward 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Fri, 07 Aug 2020 21:19:21 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10061 Fix bugs of IMPALA-9645

2020-08-07 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16306 )

Change subject: IMPALA-10061 Fix bugs of IMPALA-9645
..


Patch Set 1:

Build Failed

https://jenkins.impala.io/job/gerrit-code-review-checks/6837/ : Initial code 
review checks failed. See linked job for details on the failure.


--
To view, visit http://gerrit.cloudera.org:8080/16306
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I219588992715b7d5c69cd7c0d48ff4d90b980338
Gerrit-Change-Number: 16306
Gerrit-PatchSet: 1
Gerrit-Owner: Anonymous Coward 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 07 Aug 2020 21:13:24 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9904 Fix bad cipher test failed case on aarch64

Tim Armstrong has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16172 )

Change subject: IMPALA-9904 Fix bad cipher test failed case on aarch64
..


Patch Set 5: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/16172
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I19751b6bf1045fd6d901c5a67f74e8bdd6bf65d3
Gerrit-Change-Number: 16172
Gerrit-PatchSet: 5
Gerrit-Owner: Anonymous Coward 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 07 Aug 2020 21:08:24 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9676 Add aarch64 compile options for clang

Tim Armstrong has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/15755 )

Change subject: IMPALA-9676 Add aarch64 compile options for clang
..


Patch Set 16: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/15755
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I69a5ff64bbd4427dd87ec6e884251e76d6a73122
Gerrit-Change-Number: 15755
Gerrit-PatchSet: 16
Gerrit-Owner: Anonymous Coward 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 07 Aug 2020 21:08:16 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10061 Fix bugs of IMPALA-9645

Tim Armstrong has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16306 )

Change subject: IMPALA-10061 Fix bugs of IMPALA-9645
..


Patch Set 1:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/16306/1/be/src/codegen/llvm-codegen.cc
File be/src/codegen/llvm-codegen.cc:

http://gerrit.cloudera.org:8080/#/c/16306/1/be/src/codegen/llvm-codegen.cc@1613
PS1, Line 1613:   if (IsCPUFeatureEnabled(CpuInfo::SSE4_2)) {
I think it's confusing for readers to #ifdef control flow structures like this. 
Can you reworked it, e.g. by defining a new constant or macro to something like:

  if (IS_AARCH64 || IsCPUFeatureEnabled(CpuInfo::SSE4_2).


http://gerrit.cloudera.org:8080/#/c/16306/1/be/src/exprs/scalar-fn-call.cc
File be/src/exprs/scalar-fn-call.cc:

http://gerrit.cloudera.org:8080/#/c/16306/1/be/src/exprs/scalar-fn-call.cc@351
PS1, Line 351: PrimitiveType col_type = children_[i]->type().type;
I think you need to undo this change - it caused a clang-tidy failure because 
it's an unused variable on x86



--
To view, visit http://gerrit.cloudera.org:8080/16306
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I219588992715b7d5c69cd7c0d48ff4d90b980338
Gerrit-Change-Number: 16306
Gerrit-PatchSet: 1
Gerrit-Owner: Anonymous Coward 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 07 Aug 2020 21:07:59 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-9906 Fix thread-pool-test failed case on aarch64

Tim Armstrong has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16173 )

Change subject: IMPALA-9906 Fix thread-pool-test failed case on aarch64
..


Patch Set 5: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/16173
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7b353f7eb9662995d9a8ae460bb1631933873d5d
Gerrit-Change-Number: 16173
Gerrit-PatchSet: 5
Gerrit-Owner: Anonymous Coward 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 07 Aug 2020 21:08:29 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10017: Implement ds kll union() function

2020-08-07 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16267 )

Change subject: IMPALA-10017: Implement ds_kll_union() function
..


Patch Set 8: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/16267
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I020aea28d36f9b6ef9fb57c08411f2170f5c24bf
Gerrit-Change-Number: 16267
Gerrit-PatchSet: 8
Gerrit-Owner: Gabor Kaszab 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Gabor Kaszab 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Fri, 07 Aug 2020 21:04:35 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10017: Implement ds kll union() function

2020-08-07 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16267 )

Change subject: IMPALA-10017: Implement ds_kll_union() function
..


Patch Set 8:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6252/ 
DRY_RUN=false


--
To view, visit http://gerrit.cloudera.org:8080/16267
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I020aea28d36f9b6ef9fb57c08411f2170f5c24bf
Gerrit-Change-Number: 16267
Gerrit-PatchSet: 8
Gerrit-Owner: Gabor Kaszab 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Gabor Kaszab 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Fri, 07 Aug 2020 21:04:36 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9926 base64decode % will not return error when in newer OS

2020-08-07 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16175 )

Change subject: IMPALA-9926  base64decode % will not return  error when in 
newer OS
..


Patch Set 5:

Build Failed

https://jenkins.impala.io/job/gerrit-code-review-checks/6836/ : Initial code 
review checks failed. See linked job for details on the failure.


--
To view, visit http://gerrit.cloudera.org:8080/16175
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib9bd9e03d5f744c18c957cdaf2064fa918086004
Gerrit-Change-Number: 16175
Gerrit-PatchSet: 5
Gerrit-Owner: Anonymous Coward 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 07 Aug 2020 21:04:07 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10017: Implement ds kll union() function

Gabor Kaszab has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16267 )

Change subject: IMPALA-10017: Implement ds_kll_union() function
..


Patch Set 7: Code-Review+2

Carry +2 from Csaba


--
To view, visit http://gerrit.cloudera.org:8080/16267
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I020aea28d36f9b6ef9fb57c08411f2170f5c24bf
Gerrit-Change-Number: 16267
Gerrit-PatchSet: 7
Gerrit-Owner: Gabor Kaszab 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Gabor Kaszab 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Fri, 07 Aug 2020 21:04:00 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10017: Implement ds kll union() function

Gabor Kaszab has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16267 )

Change subject: IMPALA-10017: Implement ds_kll_union() function
..


Patch Set 7:

PS7 is rebase with master


--
To view, visit http://gerrit.cloudera.org:8080/16267
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I020aea28d36f9b6ef9fb57c08411f2170f5c24bf
Gerrit-Change-Number: 16267
Gerrit-PatchSet: 7
Gerrit-Owner: Gabor Kaszab 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Gabor Kaszab 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Fri, 07 Aug 2020 21:03:12 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10017: Implement ds kll union() function

Hello Csaba Ringhofer, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/16267

to look at the new patch set (#7).

Change subject: IMPALA-10017: Implement ds_kll_union() function
..

IMPALA-10017: Implement ds_kll_union() function

This function receives a set of serialized Apache DataSketches KLL
sketches produced by ds_kll_sketch() and merges them into a single
sketch.

An example usage is to create a sketch for each partition of a table,
write these sketches to a separate table and based on which partition
the user is interested of the relevant sketches can be union-ed
together to get an estimate. E.g.:
  SELECT
  ds_kll_quantile(ds_kll_union(sketch_col), 0.5)
  FROM sketch_tbl
  WHERE partition_col=1 OR partition_col=5;

Testing:
  - Apart from the automated tests I added to this patch I also
tested ds_kll_union() on a bigger dataset to check that
serialization, deserialization and merging steps work well. I
took TPCH25.linelitem, created a number of sketches with grouping
by l_shipdate and called ds_kll_union() on those sketches.

Change-Id: I020aea28d36f9b6ef9fb57c08411f2170f5c24bf
---
M be/src/exprs/aggregate-functions-ir.cc
M be/src/exprs/aggregate-functions.h
M fe/src/main/java/org/apache/impala/catalog/BuiltinsDb.java
M testdata/data/README
A testdata/data/kll_sketches_from_impala.parquet
M testdata/workloads/functional-query/queries/QueryTest/datasketches-kll.test
M tests/query_test/test_datasketches.py
7 files changed, 204 insertions(+), 39 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/67/16267/7
--
To view, visit http://gerrit.cloudera.org:8080/16267
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I020aea28d36f9b6ef9fb57c08411f2170f5c24bf
Gerrit-Change-Number: 16267
Gerrit-PatchSet: 7
Gerrit-Owner: Gabor Kaszab 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Gabor Kaszab 
Gerrit-Reviewer: Impala Public Jenkins

[Impala-ASF-CR] IMPALA-9925 cast(pow(2, 31) as int) return 2147483647 on aarch64

2020-08-07 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16174 )

Change subject: IMPALA-9925 cast(pow(2, 31) as int) return 2147483647 on aarch64
..


Patch Set 5:

Build Failed

https://jenkins.impala.io/job/gerrit-code-review-checks/6835/ : Initial code 
review checks failed. See linked job for details on the failure.


--
To view, visit http://gerrit.cloudera.org:8080/16174
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I58ab52acebb9bcddbf298efa886fd30ce35f68bf
Gerrit-Change-Number: 16174
Gerrit-PatchSet: 5
Gerrit-Owner: Anonymous Coward 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Comment-Date: Fri, 07 Aug 2020 20:50:57 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10029: Strip debug symbols from libkudu client and libstdc++ binaries

Sahil Takiar has removed a vote on this change.

Change subject: IMPALA-10029: Strip debug symbols from libkudu_client and 
libstdc++ binaries
..


Removed Verified-1 by Impala Public Jenkins 
--
To view, visit http://gerrit.cloudera.org:8080/16263
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: I61fdf47041bd96248ecb48ae57dde143de2da294
Gerrit-Change-Number: 16263
Gerrit-PatchSet: 6
Gerrit-Owner: Sahil Takiar 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Sahil Takiar 
Gerrit-Reviewer: Tim Armstrong

[Impala-ASF-CR] IMPALA-10029: Strip debug symbols from libkudu client and libstdc++ binaries

2020-08-07 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16263 )

Change subject: IMPALA-10029: Strip debug symbols from libkudu_client and 
libstdc++ binaries
..


Patch Set 7:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6251/ 
DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/16263
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I61fdf47041bd96248ecb48ae57dde143de2da294
Gerrit-Change-Number: 16263
Gerrit-PatchSet: 7
Gerrit-Owner: Sahil Takiar 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Sahil Takiar 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 07 Aug 2020 20:39:27 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10029: Strip debug symbols from libkudu client and libstdc++ binaries

Sahil Takiar has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16263 )

Change subject: IMPALA-10029: Strip debug symbols from libkudu_client and 
libstdc++ binaries
..


Patch Set 6:

Failed because of IMPALA-10054


--
To view, visit http://gerrit.cloudera.org:8080/16263
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I61fdf47041bd96248ecb48ae57dde143de2da294
Gerrit-Change-Number: 16263
Gerrit-PatchSet: 6
Gerrit-Owner: Sahil Takiar 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Sahil Takiar 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 07 Aug 2020 20:39:13 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10029: Strip debug symbols from libkudu client and libstdc++ binaries

2020-08-07 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16263 )

Change subject: IMPALA-10029: Strip debug symbols from libkudu_client and 
libstdc++ binaries
..


Patch Set 7: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/16263
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I61fdf47041bd96248ecb48ae57dde143de2da294
Gerrit-Change-Number: 16263
Gerrit-PatchSet: 7
Gerrit-Owner: Sahil Takiar 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Sahil Takiar 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 07 Aug 2020 20:39:26 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9676 Add aarch64 compile options for clang

2020-08-07 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/15755 )

Change subject: IMPALA-9676 Add aarch64 compile options for clang
..


Patch Set 16:

Build Failed

https://jenkins.impala.io/job/gerrit-code-review-checks/6832/ : Initial code 
review checks failed. See linked job for details on the failure.


--
To view, visit http://gerrit.cloudera.org:8080/15755
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I69a5ff64bbd4427dd87ec6e884251e76d6a73122
Gerrit-Change-Number: 15755
Gerrit-PatchSet: 16
Gerrit-Owner: Anonymous Coward 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 07 Aug 2020 20:36:35 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9906 Fix thread-pool-test failed case on aarch64

2020-08-07 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16173 )

Change subject: IMPALA-9906 Fix thread-pool-test failed case on aarch64
..


Patch Set 5:

Build Failed

https://jenkins.impala.io/job/gerrit-code-review-checks/6834/ : Initial code 
review checks failed. See linked job for details on the failure.


--
To view, visit http://gerrit.cloudera.org:8080/16173
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7b353f7eb9662995d9a8ae460bb1631933873d5d
Gerrit-Change-Number: 16173
Gerrit-PatchSet: 5
Gerrit-Owner: Anonymous Coward 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 07 Aug 2020 20:36:23 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9744: Treat corrupt table stats as missing to avoid bad plans

2020-08-07 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16098 )

Change subject: IMPALA-9744: Treat corrupt table stats as missing to avoid bad 
plans
..


Patch Set 29: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/16098
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9f4c64616ff7c0b6d5a48f2b5331325feeff3576
Gerrit-Change-Number: 16098
Gerrit-PatchSet: 29
Gerrit-Owner: Qifan Chen 
Gerrit-Reviewer: Aman Sinha 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Sahil Takiar 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 07 Aug 2020 20:36:20 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9744: Treat corrupt table stats as missing to avoid bad plans

2020-08-07 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16098 )

Change subject: IMPALA-9744: Treat corrupt table stats as missing to avoid bad 
plans
..


Patch Set 29:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6250/ 
DRY_RUN=false


--
To view, visit http://gerrit.cloudera.org:8080/16098
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9f4c64616ff7c0b6d5a48f2b5331325feeff3576
Gerrit-Change-Number: 16098
Gerrit-PatchSet: 29
Gerrit-Owner: Qifan Chen 
Gerrit-Reviewer: Aman Sinha 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Sahil Takiar 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 07 Aug 2020 20:36:21 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9744: Treat corrupt table stats as missing to avoid bad plans

Sahil Takiar has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16098 )

Change subject: IMPALA-9744: Treat corrupt table stats as missing to avoid bad 
plans
..


Patch Set 28: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/16098
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9f4c64616ff7c0b6d5a48f2b5331325feeff3576
Gerrit-Change-Number: 16098
Gerrit-PatchSet: 28
Gerrit-Owner: Qifan Chen 
Gerrit-Reviewer: Aman Sinha 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Sahil Takiar 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 07 Aug 2020 20:35:56 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9904 Fix bad cipher test failed case on aarch64

2020-08-07 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16172 )

Change subject: IMPALA-9904 Fix bad cipher test failed case on aarch64
..


Patch Set 5:

Build Failed

https://jenkins.impala.io/job/gerrit-code-review-checks/6833/ : Initial code 
review checks failed. See linked job for details on the failure.


--
To view, visit http://gerrit.cloudera.org:8080/16172
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I19751b6bf1045fd6d901c5a67f74e8bdd6bf65d3
Gerrit-Change-Number: 16172
Gerrit-PatchSet: 5
Gerrit-Owner: Anonymous Coward 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 07 Aug 2020 20:36:26 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9544 Replace Intel's SSE instructions with ARM's NEON instructions

2020-08-07 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/15531 )

Change subject: IMPALA-9544 Replace Intel's SSE instructions with ARM's NEON 
instructions
..


Patch Set 39:

Build Failed

https://jenkins.impala.io/job/gerrit-code-review-checks/6831/ : Initial code 
review checks failed. See linked job for details on the failure.


--
To view, visit http://gerrit.cloudera.org:8080/15531
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id7dfe17125b2910ece54e7dd18b4e4b25d7de8b9
Gerrit-Change-Number: 15531
Gerrit-PatchSet: 39
Gerrit-Owner: Anonymous Coward 
Gerrit-Reviewer: Anonymous Coward 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 07 Aug 2020 20:34:17 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9995 Fix test alloc fail failed case on aarch64

zhaoren...@hotmail.com has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/16307


Change subject: IMPALA-9995 Fix test_alloc_fail failed case on aarch64
..

IMPALA-9995 Fix test_alloc_fail failed case on aarch64

Length of Json object '{"a": 1}", '$.a' is 32 bytes on x86,
but is 48 bytes on aarch64

Change-Id: I9a5a4ba19b225bdb4f18a68d6d9cb2c2d16f91fd
---
M testdata/workloads/functional-query/queries/QueryTest/alloc-fail-init.test
1 file changed, 1 insertion(+), 1 deletion(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/16307/1
--
To view, visit http://gerrit.cloudera.org:8080/16307
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I9a5a4ba19b225bdb4f18a68d6d9cb2c2d16f91fd
Gerrit-Change-Number: 16307
Gerrit-PatchSet: 1
Gerrit-Owner: Anonymous Coward

[Impala-ASF-CR] IMPALA-9544 Replace Intel's SSE instructions with ARM's NEON instructions

2020-08-07 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/15531 )

Change subject: IMPALA-9544 Replace Intel's SSE instructions with ARM's NEON 
instructions
..


Patch Set 39:

(8 comments)

http://gerrit.cloudera.org:8080/#/c/15531/39/be/src/util/sse2neon.h
File be/src/util/sse2neon.h:

http://gerrit.cloudera.org:8080/#/c/15531/39/be/src/util/sse2neon.h@213
PS39, Line 213: // 
https://msdn.microsoft.com/en-us/library/bb514059%28v=vs.120%29.aspx?f=255=-2147217396
line too long (99 > 90)


http://gerrit.cloudera.org:8080/#/c/15531/39/be/src/util/sse2neon.h@406
PS39, Line 406: // 
https://docs.microsoft.com/en-us/previous-versions/visualstudio/visual-studio-2010/whtfzhzk(v=vs.100)
line too long (104 > 90)


http://gerrit.cloudera.org:8080/#/c/15531/39/be/src/util/sse2neon.h@413
PS39, Line 413: // 
https://software.intel.com/sites/landingpage/IntrinsicsGuide/#text=_mm_set1_epi64x=4961
line too long (97 > 90)


http://gerrit.cloudera.org:8080/#/c/15531/39/be/src/util/sse2neon.h@1054
PS39, Line 1054: // 
https://software.intel.com/sites/landingpage/IntrinsicsGuide/#text=_mm_shuffle_epi8=5146
line too long (98 > 90)


http://gerrit.cloudera.org:8080/#/c/15531/39/be/src/util/sse2neon.h@1199
PS39, Line 1199: // 
https://docs.microsoft.com/en-us/previous-versions/visualstudio/visual-studio-2010/y41dkk37(v=vs.100)
line too long (104 > 90)


http://gerrit.cloudera.org:8080/#/c/15531/39/be/src/util/sse2neon.h@1645
PS39, Line 1645: // 
https://software.intel.com/sites/landingpage/IntrinsicsGuide/#text=_mm_test_all_zeros=5871
line too long (100 > 90)


http://gerrit.cloudera.org:8080/#/c/15531/39/be/src/util/sse2neon.h@3581
PS39, Line 3581: // 
https://github.com/ColinIanKing/linux-next-mirror/blob/b5f466091e130caaf0735976648f72bd5e09aa84/crypto/aegis128-neon-inner.c#L52
line too long (131 > 90)


http://gerrit.cloudera.org:8080/#/c/15531/39/be/src/util/sse2neon.h@3681
PS39, Line 3681: // 
cpp-compiler-developer-guide-and-reference-allocating-and-freeing-aligned-memory-blocks
line too long (98 > 90)



--
To view, visit http://gerrit.cloudera.org:8080/15531
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id7dfe17125b2910ece54e7dd18b4e4b25d7de8b9
Gerrit-Change-Number: 15531
Gerrit-PatchSet: 39
Gerrit-Owner: Anonymous Coward 
Gerrit-Reviewer: Anonymous Coward 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 07 Aug 2020 20:08:56 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-9925 cast(pow(2, 31) as int) return 2147483647 on aarch64

zhaoren...@hotmail.com has uploaded a new patch set (#5). ( 
http://gerrit.cloudera.org:8080/16174 )

Change subject: IMPALA-9925 cast(pow(2, 31) as int) return 2147483647 on aarch64
..

IMPALA-9925 cast(pow(2, 31) as int) return 2147483647 on aarch64

cast(pow(2, 31) as int) return 2147483647 on aarch64
but return 2147483648 on x86
I think aarch64 is correct.
So here I will not convert it, just use aarch64's value

Change-Id: I58ab52acebb9bcddbf298efa886fd30ce35f68bf
---
M be/src/exprs/expr-test.cc
1 file changed, 5 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/74/16174/5
--
To view, visit http://gerrit.cloudera.org:8080/16174
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I58ab52acebb9bcddbf298efa886fd30ce35f68bf
Gerrit-Change-Number: 16174
Gerrit-PatchSet: 5
Gerrit-Owner: Anonymous Coward 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Zoltan Borok-Nagy

[Impala-ASF-CR] IMPALA-9926 base64decode % will not return error when in newer OS

zhaoren...@hotmail.com has uploaded a new patch set (#5). ( 
http://gerrit.cloudera.org:8080/16175 )

Change subject: IMPALA-9926  base64decode % will not return  error when in 
newer OS
..

IMPALA-9926  base64decode % will not return  error when in newer OS

for example, base64decode('YWxwaGE%') will return
'alpha\377' in newer os which has newer sasl library.
I tested it on Ubuntu 18.04 aarch64 version.

Change-Id: Ib9bd9e03d5f744c18c957cdaf2064fa918086004
---
M be/src/exprs/expr-test.cc
M testdata/workloads/functional-query/queries/QueryTest/exprs.test
2 files changed, 7 insertions(+), 5 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/16175/5
--
To view, visit http://gerrit.cloudera.org:8080/16175
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib9bd9e03d5f744c18c957cdaf2064fa918086004
Gerrit-Change-Number: 16175
Gerrit-PatchSet: 5
Gerrit-Owner: Anonymous Coward 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Tim Armstrong

[Impala-ASF-CR] IMPALA-9478: Profiles should indicate if custom UDFs are being used

2020-08-07 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/16188 )

Change subject: IMPALA-9478: Profiles should indicate if custom UDFs are being 
used
..

IMPALA-9478: Profiles should indicate if custom UDFs are being used

Adds a marker to runtime profiles and explain plans indicating if custom
(e.g. non-built in) user-defined functions are being used. For explain
plans, a SQL-style comment is added after any function call. For runtime
profiles, a new Frontend entry called "User Defined Functions (UDFs)"
lists out all UDFs analyzed during planning.

Take the following example:

  create function hive_lower(string) returns string location
  '/test-warehouse/hive-exec.jar'
  symbol='org.apache.hadoop.hive.ql.udf.UDFLower';
  set explain_level=3;
  explain select * from functional.alltypes order by hive_lower(string_col);
  ...
  01:SORT
order by: default.hive_lower(string_col) /* JAVA UDF */ ASC
materialized: default.hive_lower(string_col) /* JAVA UDF */
  ...

This shows up in the runtime profile as well.

When the above query is actually run, the runtime profile includes the
following entry:

  Frontend
User Defined Functions (UDFs): default.hive_lower

Error messages will also include SQL-style comments about any UDFs used.
For example:

  select aggfn(int_col) over (partition by int_col) from
  functional.alltypesagg

Throws:

  Aggregate function 'default.aggfn(int_col) /* NATIVE UDF */' not
  supported with OVER clause.

Testing:
* Added tests to test_udfs.py
* Ran core tests

Change-Id: I79122e6cc74fd5a62c76962289a1615fbac2f345
Reviewed-on: http://gerrit.cloudera.org:8080/16188
Reviewed-by: Impala Public Jenkins 
Tested-by: Impala Public Jenkins 
---
M fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java
M fe/src/main/java/org/apache/impala/service/FrontendProfile.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeSubqueriesTest.java
M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java
M 
testdata/workloads/functional-planner/queries/PlannerTest/sort-expr-materialization.test
M tests/query_test/test_udfs.py
7 files changed, 88 insertions(+), 21 deletions(-)

Approvals:
  Impala Public Jenkins: Looks good to me, approved; Verified

--
To view, visit http://gerrit.cloudera.org:8080/16188
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I79122e6cc74fd5a62c76962289a1615fbac2f345
Gerrit-Change-Number: 16188
Gerrit-PatchSet: 9
Gerrit-Owner: Sahil Takiar 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Sahil Takiar 
Gerrit-Reviewer: Tim Armstrong

[Impala-ASF-CR] IMPALA-9906 Fix thread-pool-test failed case on aarch64

zhaoren...@hotmail.com has uploaded a new patch set (#5). ( 
http://gerrit.cloudera.org:8080/16173 )

Change subject: IMPALA-9906 Fix thread-pool-test failed case on aarch64
..

IMPALA-9906 Fix thread-pool-test failed case on aarch64

Threads switch frequency is not so fast as x86.
So here change the sleep task time from 100ms to 500ms

Change-Id: I7b353f7eb9662995d9a8ae460bb1631933873d5d
---
M be/src/util/thread-pool-test.cc
1 file changed, 9 insertions(+), 3 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/73/16173/5
--
To view, visit http://gerrit.cloudera.org:8080/16173
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7b353f7eb9662995d9a8ae460bb1631933873d5d
Gerrit-Change-Number: 16173
Gerrit-PatchSet: 5
Gerrit-Owner: Anonymous Coward 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Tim Armstrong

[Impala-ASF-CR] IMPALA-10061 Fix bugs of IMPALA-9645

zhaoren...@hotmail.com has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/16306


Change subject: IMPALA-10061 Fix bugs of IMPALA-9645
..

IMPALA-10061 Fix bugs of IMPALA-9645

Fix one bug of IMPALA-9645.

And fix issue when return type is decimal,
codegen code lack a 'StructRet' attribute,
this is not a issue on x86, but on aarch64,
the "StructRet" attribute is necessary.

And fix the hash function on aarch64.

Change-Id: I219588992715b7d5c69cd7c0d48ff4d90b980338
---
M be/src/codegen/llvm-codegen.cc
M be/src/exprs/scalar-fn-call.cc
2 files changed, 39 insertions(+), 3 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/06/16306/1
--
To view, visit http://gerrit.cloudera.org:8080/16306
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I219588992715b7d5c69cd7c0d48ff4d90b980338
Gerrit-Change-Number: 16306
Gerrit-PatchSet: 1
Gerrit-Owner: Anonymous Coward

[Impala-ASF-CR] IMPALA-9478: Profiles should indicate if custom UDFs are being used

2020-08-07 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16188 )

Change subject: IMPALA-9478: Profiles should indicate if custom UDFs are being 
used
..


Patch Set 8: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/16188
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I79122e6cc74fd5a62c76962289a1615fbac2f345
Gerrit-Change-Number: 16188
Gerrit-PatchSet: 8
Gerrit-Owner: Sahil Takiar 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Sahil Takiar 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 07 Aug 2020 20:08:20 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9904 Fix bad cipher test failed case on aarch64

zhaoren...@hotmail.com has uploaded a new patch set (#5). ( 
http://gerrit.cloudera.org:8080/16172 )

Change subject: IMPALA-9904 Fix bad cipher test failed case on aarch64
..

IMPALA-9904 Fix bad cipher test failed case on aarch64

On aarch64 and os ubuntu 18.04, the openssl version is 1.1.1,
the server which used openssl can start  successfully even ciphers is bad.
So here just don't test bad ciphers cases on aarch64.
On x86, the server cannot start successfully  because the lower openssl version,
not because the bad cipher.
Change-Id: I19751b6bf1045fd6d901c5a67f74e8bdd6bf65d3
---
M be/src/rpc/rpc-mgr-test.cc
M be/src/rpc/thrift-server-test.cc
M be/src/util/webserver-test.cc
3 files changed, 10 insertions(+), 5 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/72/16172/5
--
To view, visit http://gerrit.cloudera.org:8080/16172
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I19751b6bf1045fd6d901c5a67f74e8bdd6bf65d3
Gerrit-Change-Number: 16172
Gerrit-PatchSet: 5
Gerrit-Owner: Anonymous Coward 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Tim Armstrong

[Impala-ASF-CR] IMPALA-9544 Replace Intel's SSE instructions with ARM's NEON instructions

zhaoren...@hotmail.com has uploaded a new patch set (#39). ( 
http://gerrit.cloudera.org:8080/15531 )

Change subject: IMPALA-9544 Replace Intel's SSE instructions with ARM's NEON 
instructions
..

IMPALA-9544 Replace Intel's SSE instructions with ARM's NEON instructions

Replace Intel's SSE instructions with ARM's NEON instructions
Replace Intel's crc32 instructions with ARM's instructions
Replace Intel's popcntq instruction with ARM's mechanism
Replace Intel's pcmpestri and pcmpestrm instructions
with ARM mechanism

Change-Id: Id7dfe17125b2910ece54e7dd18b4e4b25d7de8b9
---
M CMakeLists.txt
M be/CMakeLists.txt
M be/src/benchmarks/bswap-benchmark.cc
M be/src/benchmarks/int-hash-benchmark.cc
M be/src/codegen/CMakeLists.txt
M be/src/codegen/gen_ir_descriptions.py
M be/src/codegen/llvm-codegen-test.cc
M be/src/exec/delimited-text-parser.inline.h
M be/src/kudu/util/block_bloom_filter.cc
M be/src/kudu/util/group_varint-inl.h
M be/src/kudu/util/group_varint-test.cc
A be/src/kudu/util/sse2neon.h
M be/src/util/bit-util-test.cc
M be/src/util/bit-util.cc
M be/src/util/bit-util.h
M be/src/util/bloom-filter.cc
M be/src/util/bloom-filter.h
M be/src/util/cpu-info.cc
M be/src/util/hash-util-ir.cc
M be/src/util/hash-util.h
M be/src/util/sse-util.h
A be/src/util/sse2neon.h
22 files changed, 3,994 insertions(+), 25 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/15531/39
--
To view, visit http://gerrit.cloudera.org:8080/15531
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id7dfe17125b2910ece54e7dd18b4e4b25d7de8b9
Gerrit-Change-Number: 15531
Gerrit-PatchSet: 39
Gerrit-Owner: Anonymous Coward 
Gerrit-Reviewer: Anonymous Coward 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Tim Armstrong

[Impala-ASF-CR] IMPALA-9676 Add aarch64 compile options for clang

zhaoren...@hotmail.com has uploaded a new patch set (#16). ( 
http://gerrit.cloudera.org:8080/15755 )

Change subject: IMPALA-9676 Add aarch64 compile options for clang
..

IMPALA-9676 Add aarch64 compile options for clang

Add signed-char and armv8a and crc compile options to clang

Change-Id: I69a5ff64bbd4427dd87ec6e884251e76d6a73122
---
M be/CMakeLists.txt
1 file changed, 5 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/55/15755/16
--
To view, visit http://gerrit.cloudera.org:8080/15755
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I69a5ff64bbd4427dd87ec6e884251e76d6a73122
Gerrit-Change-Number: 15755
Gerrit-PatchSet: 16
Gerrit-Owner: Anonymous Coward 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Tim Armstrong

[Impala-ASF-CR] IMPALA-10029: Strip debug symbols from libkudu client and libstdc++ binaries

2020-08-07 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16263 )

Change subject: IMPALA-10029: Strip debug symbols from libkudu_client and 
libstdc++ binaries
..


Patch Set 6: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6246/


--
To view, visit http://gerrit.cloudera.org:8080/16263
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I61fdf47041bd96248ecb48ae57dde143de2da294
Gerrit-Change-Number: 16263
Gerrit-PatchSet: 6
Gerrit-Owner: Sahil Takiar 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Sahil Takiar 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 07 Aug 2020 19:50:13 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10043: Keep more logs when using EE TEST SHARDS

2020-08-07 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16297 )

Change subject: IMPALA-10043: Keep more logs when using EE_TEST_SHARDS
..


Patch Set 2:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6249/ 
DRY_RUN=true


--
To view, visit http://gerrit.cloudera.org:8080/16297
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie011b892cd2eb1a528012ec5600e72e44f281a88
Gerrit-Change-Number: 16297
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Laszlo Gaal 
Gerrit-Comment-Date: Fri, 07 Aug 2020 19:16:31 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10039: Fixed Expr-test crash caused by thread unsafe function

2020-08-07 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16299 )

Change subject: IMPALA-10039: Fixed Expr-test crash caused by thread unsafe 
function
..


Patch Set 4:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/6830/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/16299
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I85245bf4bffb469913d53741847e67773b7d4627
Gerrit-Change-Number: 16299
Gerrit-PatchSet: 4
Gerrit-Owner: Wenzhe Zhou 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Reviewer: Wenzhe Zhou 
Gerrit-Comment-Date: Fri, 07 Aug 2020 18:45:24 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10039: Fixed Expr-test crash caused by thread unsafe function

2020-08-07 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16299 )

Change subject: IMPALA-10039: Fixed Expr-test crash caused by thread unsafe 
function
..


Patch Set 4:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6248/ 
DRY_RUN=false


--
To view, visit http://gerrit.cloudera.org:8080/16299
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I85245bf4bffb469913d53741847e67773b7d4627
Gerrit-Change-Number: 16299
Gerrit-PatchSet: 4
Gerrit-Owner: Wenzhe Zhou 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Reviewer: Wenzhe Zhou 
Gerrit-Comment-Date: Fri, 07 Aug 2020 18:31:26 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10039: Fixed Expr-test crash caused by thread unsafe function

2020-08-07 Thread Thomas Tauber-Marshall (Code Review)

Thomas Tauber-Marshall has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16299 )

Change subject: IMPALA-10039: Fixed Expr-test crash caused by thread unsafe 
function
..


Patch Set 4: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/16299
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I85245bf4bffb469913d53741847e67773b7d4627
Gerrit-Change-Number: 16299
Gerrit-PatchSet: 4
Gerrit-Owner: Wenzhe Zhou 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Reviewer: Wenzhe Zhou 
Gerrit-Comment-Date: Fri, 07 Aug 2020 18:31:11 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10029: Strip debug symbols from libkudu client and libstdc++ binaries

Tim Armstrong has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16263 )

Change subject: IMPALA-10029: Strip debug symbols from libkudu_client and 
libstdc++ binaries
..


Patch Set 6: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/16263
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I61fdf47041bd96248ecb48ae57dde143de2da294
Gerrit-Change-Number: 16263
Gerrit-PatchSet: 6
Gerrit-Owner: Sahil Takiar 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Sahil Takiar 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 07 Aug 2020 18:18:29 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10039: Fixed Expr-test crash caused by thread unsafe function

Wenzhe Zhou has uploaded a new patch set (#4). ( 
http://gerrit.cloudera.org:8080/16299 )

Change subject: IMPALA-10039: Fixed Expr-test crash caused by thread unsafe 
function
..

IMPALA-10039: Fixed Expr-test crash caused by thread unsafe function

Recent patch for IMPALA-5746 registers a callback function for the
updating of cluster membership. The callback function cancels the
queries scheduled by the failed coordinators. This callback function
was called during Expr-test and caused crash.
This patch checks if the process running for tests and only registers
the callback function if it's not running for BE/FE tests.

Testing:
 - The issue could be reproduced by running expr-test for 10-20
   iterations. Verified the fixing by running expr-test over 1000
   iterations without crash.
 - Passed TestProcessFailures::test_kill_coordinator.
 - Passed core tests.

Change-Id: I85245bf4bffb469913d53741847e67773b7d4627
---
M be/src/runtime/exec-env.cc
1 file changed, 1 insertion(+), 1 deletion(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/99/16299/4
--
To view, visit http://gerrit.cloudera.org:8080/16299
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I85245bf4bffb469913d53741847e67773b7d4627
Gerrit-Change-Number: 16299
Gerrit-PatchSet: 4
Gerrit-Owner: Wenzhe Zhou 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Reviewer: Wenzhe Zhou

[Impala-ASF-CR] IMPALA-10039: Fixed Expr-test crash caused by thread unsafe function

Wenzhe Zhou has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16299 )

Change subject: IMPALA-10039: Fixed Expr-test crash caused by thread unsafe 
function
..


Patch Set 3:

(1 comment)

TestInfo part is required to stop expr-test from crashing. I will check in this 
part first to stop the builds from breaking, and fix the other part in a 
separate patch.

http://gerrit.cloudera.org:8080/#/c/16299/3/be/src/runtime/query-state.cc
File be/src/runtime/query-state.cc:

http://gerrit.cloudera.org:8080/#/c/16299/3/be/src/runtime/query-state.cc@863
PS3, Line 863:   discard_result(initialized_.Get(1, _out));
> So using this timeout here is less than ideal (for example because how do w
agree



--
To view, visit http://gerrit.cloudera.org:8080/16299
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I85245bf4bffb469913d53741847e67773b7d4627
Gerrit-Change-Number: 16299
Gerrit-PatchSet: 3
Gerrit-Owner: Wenzhe Zhou 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Reviewer: Wenzhe Zhou 
Gerrit-Comment-Date: Fri, 07 Aug 2020 18:17:00 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] WIP IMPALA-9180 (part 1): Remove legacy ImpalaInternalService

2020-08-07 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16291 )

Change subject: WIP IMPALA-9180 (part 1): Remove legacy ImpalaInternalService
..


Patch Set 2:

Build Failed

https://jenkins.impala.io/job/gerrit-code-review-checks/6829/ : Initial code 
review checks failed. See linked job for details on the failure.


--
To view, visit http://gerrit.cloudera.org:8080/16291
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5fa83c8009590124dded4783f77ef70fa30119e6
Gerrit-Change-Number: 16291
Gerrit-PatchSet: 2
Gerrit-Owner: Wenzhe Zhou 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Reviewer: Wenzhe Zhou 
Gerrit-Comment-Date: Fri, 07 Aug 2020 18:07:03 +
Gerrit-HasComments: No

[Impala-ASF-CR] WIP IMPALA-9180 (part 1): Remove legacy ImpalaInternalService

Wenzhe Zhou has uploaded a new patch set (#2). ( 
http://gerrit.cloudera.org:8080/16291 )

Change subject: WIP IMPALA-9180 (part 1): Remove legacy ImpalaInternalService
..

WIP IMPALA-9180 (part 1): Remove legacy ImpalaInternalService

The legacy Thrift based Impala internal service has been deprecated
and can be removed now. The port 22000 can also be freed up.
This patch removes ImpalaInternalService related code. The flag
be_port is made as a REMOVED_FLAG and all infrastructures around it
are cleaned up. TQueryCtx.coord_address is changed to
TQueryCtx.coord_hostname since the port in TQueryCtx.coord_address
is set as be_port and is unused now. Rename
TQueryCtx.coord_krpc_address as TQueryCtx.coord_ip_address.

Testing:
 - TODO: Pass the exhaustive test.

Change-Id: I5fa83c8009590124dded4783f77ef70fa30119e6
---
M be/generated-sources/gen-cpp/CMakeLists.txt
M be/src/benchmarks/expr-benchmark.cc
M be/src/common/global-flags.cc
M be/src/exprs/expr-test.cc
M be/src/exprs/utility-functions-ir.cc
M be/src/rpc/thrift-server-test.cc
D be/src/runtime/backend-client.h
M be/src/runtime/client-cache-types.h
M be/src/runtime/coordinator-backend-state.cc
M be/src/runtime/data-stream-test.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/exec-env.h
M be/src/runtime/fragment-instance-state.cc
M be/src/runtime/fragment-instance-state.h
M be/src/runtime/initial-reservations.cc
M be/src/runtime/query-exec-mgr.cc
M be/src/runtime/query-state.cc
M be/src/runtime/runtime-filter-bank.cc
M be/src/runtime/test-env.cc
M be/src/scheduling/executor-blacklist.cc
M be/src/scheduling/scheduler-test-util.h
M be/src/service/CMakeLists.txt
M be/src/service/client-request-state.cc
M be/src/service/control-service.cc
D be/src/service/impala-internal-service.cc
D be/src/service/impala-internal-service.h
M be/src/service/impala-server.cc
M be/src/service/impala-server.h
M be/src/service/impalad-main.cc
M be/src/service/session-expiry-test.cc
M be/src/testutil/in-process-servers.cc
M be/src/testutil/in-process-servers.h
M be/src/util/debug-util.cc
M bin/generate_minidump_collection_testdata.py
M bin/start-impala-cluster.py
M common/thrift/ImpalaInternalService.thrift
M infra/deploy/deploy.py
M tests/common/impala_cluster.py
M tests/common/impala_service.py
M tests/custom_cluster/test_blacklist.py
M tests/custom_cluster/test_process_failures.py
M tests/custom_cluster/test_query_retries.py
M tests/custom_cluster/test_restart_services.py
M tests/shell/test_shell_interactive.py
M tests/webserver/test_web_pages.py
45 files changed, 100 insertions(+), 352 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/91/16291/2
--
To view, visit http://gerrit.cloudera.org:8080/16291
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I5fa83c8009590124dded4783f77ef70fa30119e6
Gerrit-Change-Number: 16291
Gerrit-PatchSet: 2
Gerrit-Owner: Wenzhe Zhou 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Reviewer: Wenzhe Zhou

[Impala-ASF-CR] IMPALA-10039: Fixed Expr-test crash caused by thread unsafe function

2020-08-07 Thread Thomas Tauber-Marshall (Code Review)

Thomas Tauber-Marshall has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16299 )

Change subject: IMPALA-10039: Fixed Expr-test crash caused by thread unsafe 
function
..


Patch Set 3:

(1 comment)

If I'm understanding correctly, just the TestInfo part is required to stop 
expr-test from crashing, so if you want to submit that as a patch by itself and 
then take my suggestion for fixing the other part and submit it separately, 
that might be nice so that we can stop the builds from breaking as much. 
Depends a bit on how quickly you'll be able to get the rest of the stuff 
updated, tested, and resubmitted.

http://gerrit.cloudera.org:8080/#/c/16299/3/be/src/runtime/query-state.cc
File be/src/runtime/query-state.cc:

http://gerrit.cloudera.org:8080/#/c/16299/3/be/src/runtime/query-state.cc@863
PS3, Line 863:   discard_result(initialized_.Get(1, _out));
So using this timeout here is less than ideal (for example because how do we 
choose a reasonable value for how long to wait?), and I'm concerned this may 
still not be correct, eg. it seems that you're assuming that if this times out 
then Init() won't get called at some point after that, which I don't think is 
always true.

I think that there's a better way to do this that doesn't require the timeout: 
something like have a 'bool is_initialized_' and a 'std::mutex init_lock_'. 
Have Init() take 'init_lock_', check if is_cancelled_ is true and if it is 
return and don't init, otherwise continue holding init_lock_ until Init() is 
done and 'is_initialized_' is set to true.

Then Cancel() will do the equivalent: also take 'init_lock_', check if 
is_initialized is true, if not just sset is_cancelled_ true and return without 
doing the cancel stuff.



--
To view, visit http://gerrit.cloudera.org:8080/16299
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I85245bf4bffb469913d53741847e67773b7d4627
Gerrit-Change-Number: 16299
Gerrit-PatchSet: 3
Gerrit-Owner: Wenzhe Zhou 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Comment-Date: Fri, 07 Aug 2020 17:24:34 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-9744: Treat corrupt table stats as missing to avoid bad plans

Sahil Takiar has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16098 )

Change subject: IMPALA-9744: Treat corrupt table stats as missing to avoid bad 
plans
..


Patch Set 25:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/16098/25/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
File fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java:

http://gerrit.cloudera.org:8080/#/c/16098/25/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java@1199
PS25, Line 1199: hasCorruptTableStats_
> > Tables and partitions form a hierarchy and if the table has a valid RC (a
Actually, thinking through this some more. Even the previous behavior doesn't 
really honor the table level setting for the RC (for partitioned tables), 
unless *all* partitions have missing / corrupt stats.

The previous behavior will just use whatever partitions stats are available, 
and ignore any missing / corrupt stats. It won't look at the table level stats 
at all.

So I think it would actually make more sense to go with my original idea, to 
change the check to use 'partitionsWithCorruptOrMissingStats.size() != 0' 
instead of hasCorruptTableStats_. And maybe you add another if statement before 
this that checks 'if numRows != -1L && 
partitionsWithCorruptOrMissingStats.size() == partitions_.size() return 
numRows' - that would be more consistent with the previous behavior.



--
To view, visit http://gerrit.cloudera.org:8080/16098
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9f4c64616ff7c0b6d5a48f2b5331325feeff3576
Gerrit-Change-Number: 16098
Gerrit-PatchSet: 25
Gerrit-Owner: Qifan Chen 
Gerrit-Reviewer: Aman Sinha 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Sahil Takiar 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 07 Aug 2020 17:16:33 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-9744: Treat corrupt table stats as missing to avoid bad plans

Sahil Takiar has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16098 )

Change subject: IMPALA-9744: Treat corrupt table stats as missing to avoid bad 
plans
..


Patch Set 25:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/16098/25/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
File fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java:

http://gerrit.cloudera.org:8080/#/c/16098/25/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java@1199
PS25, Line 1199: hasCorruptTableStats_
> Tables and partitions form a hierarchy and if the table has a valid RC (as in 
> this case through ALTER TABLE), then the table does not have missing stats, 
> regardless of the situations at the partition level.

Agree, but again I don't understand how that is a reason to only trigger the if 
condition if stats are corrupt, but not if stats are missing.

> Maybe this is a good comprise such that the intention to set RC at the table 
> level is preserved?

Again agree, it is nice to preserve the intention of setting the RC at the 
table level, but again if the goal is preserve the setting of the table level 
RC, why does it matter is partition starts are missing vs. corrupt.



--
To view, visit http://gerrit.cloudera.org:8080/16098
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9f4c64616ff7c0b6d5a48f2b5331325feeff3576
Gerrit-Change-Number: 16098
Gerrit-PatchSet: 25
Gerrit-Owner: Qifan Chen 
Gerrit-Reviewer: Aman Sinha 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Sahil Takiar 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 07 Aug 2020 17:04:44 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10034: Add remaining TPC-DS queries to workload.

Tim Armstrong has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16280 )

Change subject: IMPALA-10034: Add remaining TPC-DS queries to workload.
..


Patch Set 4:

Agree with Aman's comments about the planner tests and the changes from the 
official versions...


--
To view, visit http://gerrit.cloudera.org:8080/16280
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id5436689390f149694f14e6da1df624de4f5f7ad
Gerrit-Change-Number: 16280
Gerrit-PatchSet: 4
Gerrit-Owner: Shant Hovsepian 
Gerrit-Reviewer: Aman Sinha 
Gerrit-Reviewer: David Rorke 
Gerrit-Reviewer: Fang-Yu Rao 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Shant Hovsepian 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 07 Aug 2020 16:53:32 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10050: Fixed DCHECK error for backend in terminal state.

2020-08-07 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16303 )

Change subject: IMPALA-10050: Fixed DCHECK error for backend in terminal state.
..


Patch Set 1:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/6828/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/16303
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iba6a72f98c0f9299c22c58830ec5a643335b966a
Gerrit-Change-Number: 16303
Gerrit-PatchSet: 1
Gerrit-Owner: Wenzhe Zhou 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Comment-Date: Fri, 07 Aug 2020 16:52:29 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9744: Treat corrupt table stats as missing to avoid bad plans

2020-08-07 Thread Qifan Chen (Code Review)

Qifan Chen has posted comments on this change. (
http://gerrit.cloudera.org:8080/16098 )

Change subject: IMPALA-9744: Treat corrupt table stats as missing to avoid bad
plans
..

Patch Set 28:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/16098/25/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
File fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java:

http://gerrit.cloudera.org:8080/#/c/16098/25/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java@1199
PS25, Line 1199: table size from thos
> right - I think there can be arguments made for one behavior vs. the other.
Yeah. I sensed the same. On the other hand, we could also think like this.

Tables and partitions form a hierarchy and if the table has a valid RC (as in
this case through ALTER TABLE), then the table does not have missing stats,
regardless of the situations at the partition level.

The current IF basically enforces the following:
1. If table has missing stats, estimate;
2. If table has good stats but some partition has corrupt stats, estimate.

Maybe this is a good comprise such that the intention to set RC at the table
level is preserved?

--
To view, visit http://gerrit.cloudera.org:8080/16098
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9f4c64616ff7c0b6d5a48f2b5331325feeff3576
Gerrit-Change-Number: 16098
Gerrit-PatchSet: 28
Gerrit-Owner: Qifan Chen
Gerrit-Reviewer: Aman Sinha
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Qifan Chen
Gerrit-Reviewer: Sahil Takiar
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Fri, 07 Aug 2020 16:47:12 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-9645 Port LLVM codegen to adapt aarch64

Tim Armstrong has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/15718 )

Change subject: IMPALA-9645 Port LLVM codegen to adapt aarch64
..

IMPALA-9645 Port LLVM codegen to adapt aarch64

On aarch64, the Lowered type  of  struct {bool, int128} is form
{ {i8}, {i128} }. No padding add. This is different with x86-64,
which is { {i8}, {15*i8}, {i128} } with padding add automatically.

And here also add some type conversion between x86 and aarch64 data types.

And also add some aarch64 cpu's feature.

Change-Id: I3f30ee84ea9bf5245da88154632bb69079103d11
Reviewed-on: http://gerrit.cloudera.org:8080/15718
Tested-by: Impala Public Jenkins 
Reviewed-by: Tim Armstrong 
---
M be/src/codegen/codegen-anyval.cc
M be/src/codegen/llvm-codegen.cc
M be/src/exec/text-converter.cc
M be/src/exprs/scalar-fn-call.cc
4 files changed, 175 insertions(+), 11 deletions(-)

Approvals:
  Impala Public Jenkins: Verified
  Tim Armstrong: Looks good to me, approved

--
To view, visit http://gerrit.cloudera.org:8080/15718
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I3f30ee84ea9bf5245da88154632bb69079103d11
Gerrit-Change-Number: 15718
Gerrit-PatchSet: 21
Gerrit-Owner: Anonymous Coward 
Gerrit-Reviewer: Anonymous Coward 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Tamas Mate 
Gerrit-Reviewer: Tim Armstrong

[Impala-ASF-CR] IMPALA-9645 Port LLVM codegen to adapt aarch64

Tim Armstrong has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/15718 )

Change subject: IMPALA-9645 Port LLVM codegen to adapt aarch64
..


Patch Set 20: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/15718
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3f30ee84ea9bf5245da88154632bb69079103d11
Gerrit-Change-Number: 15718
Gerrit-PatchSet: 20
Gerrit-Owner: Anonymous Coward 
Gerrit-Reviewer: Anonymous Coward 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Tamas Mate 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 07 Aug 2020 16:45:04 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10050: Fixed DCHECK error for backend in terminal state.

Wenzhe Zhou has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/16303


Change subject: IMPALA-10050: Fixed DCHECK error for backend in terminal state.
..

IMPALA-10050: Fixed DCHECK error for backend in terminal state.

When backend_exec_state enter terminal state, it could be set as
FINISHED, CANCELLED, or ERROR. If it's ERROR due to executing failure,
QueryState.is_cancelled_ could be 0. We should expect is_cancelled_
as 1 if backend_exec_state is in CANCELLED state.

Testing:
 - Passed exhausive tests.

Change-Id: Iba6a72f98c0f9299c22c58830ec5a643335b966a
---
M be/src/runtime/query-state.cc
1 file changed, 1 insertion(+), 1 deletion(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/03/16303/1
--
To view, visit http://gerrit.cloudera.org:8080/16303
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Iba6a72f98c0f9299c22c58830ec5a643335b966a
Gerrit-Change-Number: 16303
Gerrit-PatchSet: 1
Gerrit-Owner: Wenzhe Zhou 
Gerrit-Reviewer: Thomas Tauber-Marshall

[Impala-ASF-CR] IMPALA-10054: Fix flakiness in test multiple sort run bytes limits

2020-08-07 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16301 )

Change subject: IMPALA-10054: Fix flakiness in 
test_multiple_sort_run_bytes_limits
..


Patch Set 2:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/6827/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/16301
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I84a8b579c943cddba4432cf183f7f002ef8ec6ad
Gerrit-Change-Number: 16301
Gerrit-PatchSet: 2
Gerrit-Owner: Riza Suminto 
Gerrit-Reviewer: Attila Jeges 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Riza Suminto 
Gerrit-Comment-Date: Fri, 07 Aug 2020 16:09:42 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10054: Fix flakiness in test multiple sort run bytes limits

Riza Suminto has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16301 )

Change subject: IMPALA-10054: Fix flakiness in 
test_multiple_sort_run_bytes_limits
..


Patch Set 2:

(1 comment)

Hi Attila, I change the assertion as you suggest, but not quite the same.

http://gerrit.cloudera.org:8080/#/c/16301/1/tests/query_test/test_sort.py
File tests/query_test/test_sort.py:

http://gerrit.cloudera.org:8080/#/c/16301/1/tests/query_test/test_sort.py@90
PS1, Line 90: '\s+\- SpilledRuns: .*', quer
> Also, please use raw strings for regex patterns, e.g.:
I kept the assertion in L92, but remove the one in L91.
The intention is that in case the assertion is fail, we can see the comparison 
between the expected result vs actual result in the stacktrace like this

query_test/test_sort.py:91: in test_multiple_sort_run_bytes_limits
assert "SpilledRuns: " + spilled_runs in m.group()
E   assert ('SpilledRuns: ' + '3') in '\n - SpilledRuns: 2 (2)'
E+  where '\n - SpilledRuns: 2 (2)' = ()
E+where  = <_sre.SRE_Match object at 0x7fe017df5e68>.group

Whereas previously, the actual result is not clearly shown because right hand 
side of the expression is the entire runtime_profile.



--
To view, visit http://gerrit.cloudera.org:8080/16301
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I84a8b579c943cddba4432cf183f7f002ef8ec6ad
Gerrit-Change-Number: 16301
Gerrit-PatchSet: 2
Gerrit-Owner: Riza Suminto 
Gerrit-Reviewer: Attila Jeges 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Riza Suminto 
Gerrit-Comment-Date: Fri, 07 Aug 2020 15:50:02 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10054: Fix flakiness in test multiple sort run bytes limits

Hello Attila Jeges, Csaba Ringhofer, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/16301

to look at the new patch set (#2).

Change subject: IMPALA-10054: Fix flakiness in 
test_multiple_sort_run_bytes_limits
..

IMPALA-10054: Fix flakiness in test_multiple_sort_run_bytes_limits

test_multiple_sort_run_bytes_limits seems to become flaky in
ubuntu-16.04-dockerised-tests. This flakiness may come from accuracy
change in query estimates or mem_limit specified in the test does not
fit anymore. This patch tune the parameter of the first and the second
test case of test_multiple_sort_run_bytes_limits to pass the assertion.
The assertion is also changed a bit to allow easier debugging in case if
test regression occurs again in the future.

Testing:
- Run and pass test_sort.py

Change-Id: I84a8b579c943cddba4432cf183f7f002ef8ec6ad
---
M tests/query_test/test_sort.py
1 file changed, 5 insertions(+), 3 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/16301/2
--
To view, visit http://gerrit.cloudera.org:8080/16301
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I84a8b579c943cddba4432cf183f7f002ef8ec6ad
Gerrit-Change-Number: 16301
Gerrit-PatchSet: 2
Gerrit-Owner: Riza Suminto 
Gerrit-Reviewer: Attila Jeges 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Riza Suminto

[Impala-ASF-CR] IMPALA-9478: Profiles should indicate if custom UDFs are being used

2020-08-07 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16188 )

Change subject: IMPALA-9478: Profiles should indicate if custom UDFs are being 
used
..


Patch Set 7:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/6826/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/16188
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I79122e6cc74fd5a62c76962289a1615fbac2f345
Gerrit-Change-Number: 16188
Gerrit-PatchSet: 7
Gerrit-Owner: Sahil Takiar 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Sahil Takiar 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 07 Aug 2020 15:24:26 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9478: Profiles should indicate if custom UDFs are being used

2020-08-07 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16188 )

Change subject: IMPALA-9478: Profiles should indicate if custom UDFs are being 
used
..


Patch Set 8:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6247/ 
DRY_RUN=false


--
To view, visit http://gerrit.cloudera.org:8080/16188
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I79122e6cc74fd5a62c76962289a1615fbac2f345
Gerrit-Change-Number: 16188
Gerrit-PatchSet: 8
Gerrit-Owner: Sahil Takiar 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Sahil Takiar 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 07 Aug 2020 15:03:00 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9478: Profiles should indicate if custom UDFs are being used

2020-08-07 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16188 )

Change subject: IMPALA-9478: Profiles should indicate if custom UDFs are being 
used
..


Patch Set 8: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/16188
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I79122e6cc74fd5a62c76962289a1615fbac2f345
Gerrit-Change-Number: 16188
Gerrit-PatchSet: 8
Gerrit-Owner: Sahil Takiar 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Sahil Takiar 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 07 Aug 2020 15:02:59 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9478: Profiles should indicate if custom UDFs are being used

Sahil Takiar has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16188 )

Change subject: IMPALA-9478: Profiles should indicate if custom UDFs are being 
used
..


Patch Set 7: Code-Review+2

Pre-commit failure was due to IMPALA-10054. Rebased on master. Carrying +2.


--
To view, visit http://gerrit.cloudera.org:8080/16188
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I79122e6cc74fd5a62c76962289a1615fbac2f345
Gerrit-Change-Number: 16188
Gerrit-PatchSet: 7
Gerrit-Owner: Sahil Takiar 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Sahil Takiar 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 07 Aug 2020 15:02:42 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9478: Profiles should indicate if custom UDFs are being used

Hello Qifan Chen, Tim Armstrong, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/16188

to look at the new patch set (#7).

Change subject: IMPALA-9478: Profiles should indicate if custom UDFs are being 
used
..

IMPALA-9478: Profiles should indicate if custom UDFs are being used

Adds a marker to runtime profiles and explain plans indicating if custom
(e.g. non-built in) user-defined functions are being used. For explain
plans, a SQL-style comment is added after any function call. For runtime
profiles, a new Frontend entry called "User Defined Functions (UDFs)"
lists out all UDFs analyzed during planning.

Take the following example:

  create function hive_lower(string) returns string location
  '/test-warehouse/hive-exec.jar'
  symbol='org.apache.hadoop.hive.ql.udf.UDFLower';
  set explain_level=3;
  explain select * from functional.alltypes order by hive_lower(string_col);
  ...
  01:SORT
order by: default.hive_lower(string_col) /* JAVA UDF */ ASC
materialized: default.hive_lower(string_col) /* JAVA UDF */
  ...

This shows up in the runtime profile as well.

When the above query is actually run, the runtime profile includes the
following entry:

  Frontend
User Defined Functions (UDFs): default.hive_lower

Error messages will also include SQL-style comments about any UDFs used.
For example:

  select aggfn(int_col) over (partition by int_col) from
  functional.alltypesagg

Throws:

  Aggregate function 'default.aggfn(int_col) /* NATIVE UDF */' not
  supported with OVER clause.

Testing:
* Added tests to test_udfs.py
* Ran core tests

Change-Id: I79122e6cc74fd5a62c76962289a1615fbac2f345
---
M fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java
M fe/src/main/java/org/apache/impala/service/FrontendProfile.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeSubqueriesTest.java
M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java
M 
testdata/workloads/functional-planner/queries/PlannerTest/sort-expr-materialization.test
M tests/query_test/test_udfs.py
7 files changed, 88 insertions(+), 21 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/16188/7
--
To view, visit http://gerrit.cloudera.org:8080/16188
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I79122e6cc74fd5a62c76962289a1615fbac2f345
Gerrit-Change-Number: 16188
Gerrit-PatchSet: 7
Gerrit-Owner: Sahil Takiar 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Sahil Takiar 
Gerrit-Reviewer: Tim Armstrong

[Impala-ASF-CR] IMPALA-9859: Full ACID Milestone 4: Part 2 Reading modified tables (complex types)

2020-08-07 Thread Aman Sinha (Code Review)

Aman Sinha has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16228 )

Change subject: IMPALA-9859: Full ACID Milestone 4: Part 2 Reading modified 
tables (complex types)
..


Patch Set 6:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/16228/3/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java
File fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java:

http://gerrit.cloudera.org:8080/#/c/16228/3/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java@1508
PS3, Line 1508:  *
> It will involve only one subplan, as such queries are not affected by this
Yup, I had understood the rationale for case with the single collection in the 
FROM clause...i am fine with doing this in the planner.  I was trying to 
understand how the join case would work...good to know it will have a single 
Subplan.
I don't think I had other major comments..I will do a quick pass today and 
should be ready to +1.  Thanks for your patience.



--
To view, visit http://gerrit.cloudera.org:8080/16228
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8b2c6cd3d87c452c5b96a913b14c90ada78d4c6f
Gerrit-Change-Number: 16228
Gerrit-PatchSet: 6
Gerrit-Owner: Zoltan Borok-Nagy 
Gerrit-Reviewer: Aman Sinha 
Gerrit-Reviewer: Gabor Kaszab 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Comment-Date: Fri, 07 Aug 2020 15:00:12 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-9744: Treat corrupt table stats as missing to avoid bad plans

Sahil Takiar has posted comments on this change. (
http://gerrit.cloudera.org:8080/16098 )

Change subject: IMPALA-9744: Treat corrupt table stats as missing to avoid bad
plans
..

Patch Set 25:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/16098/25/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
File fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java:

http://gerrit.cloudera.org:8080/#/c/16098/25/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java@1199
PS25, Line 1199: hasCorruptTableStats_
> Did some experiment on this. It is an interesting thought with a minor draw
right - I think there can be arguments made for one behavior vs. the other. but
for the sake of this patch, I agree that for now we should just honor the
numRows value for the table. it is possible that a user manually sets these at
the table level. in which case, should the if condition just check if numRows
== -1L, should it still have the hasCorruptTableStats_ condition. put another
way, I'm not sure why this if condition would trigger if the table has corrupt
table stats, but it won't trigger if it just has missing stats. seems like in
this case they should be treated consistently.

--
To view, visit http://gerrit.cloudera.org:8080/16098
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9f4c64616ff7c0b6d5a48f2b5331325feeff3576
Gerrit-Change-Number: 16098
Gerrit-PatchSet: 25
Gerrit-Owner: Qifan Chen
Gerrit-Reviewer: Aman Sinha
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Qifan Chen
Gerrit-Reviewer: Sahil Takiar
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Fri, 07 Aug 2020 14:57:07 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-9744: Treat corrupt table stats as missing to avoid bad plans

2020-08-07 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16098 )

Change subject: IMPALA-9744: Treat corrupt table stats as missing to avoid bad 
plans
..


Patch Set 28:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/6825/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/16098
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9f4c64616ff7c0b6d5a48f2b5331325feeff3576
Gerrit-Change-Number: 16098
Gerrit-PatchSet: 28
Gerrit-Owner: Qifan Chen 
Gerrit-Reviewer: Aman Sinha 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Sahil Takiar 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 07 Aug 2020 14:54:46 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10029: Strip debug symbols from libkudu client and libstdc++ binaries

2020-08-07 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16263 )

Change subject: IMPALA-10029: Strip debug symbols from libkudu_client and 
libstdc++ binaries
..


Patch Set 6:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6246/ 
DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/16263
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I61fdf47041bd96248ecb48ae57dde143de2da294
Gerrit-Change-Number: 16263
Gerrit-PatchSet: 6
Gerrit-Owner: Sahil Takiar 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Sahil Takiar 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 07 Aug 2020 14:40:22 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9744: Treat corrupt table stats as missing to avoid bad plans

2020-08-07 Thread Qifan Chen (Code Review)

Qifan Chen has uploaded a new patch set (#28). (
http://gerrit.cloudera.org:8080/16098 )

Change subject: IMPALA-9744: Treat corrupt table stats as missing to avoid bad
plans
..

IMPALA-9744: Treat corrupt table stats as missing to avoid bad plans

This work addresses the current limitation in computing the total row
count for a Hive table in a scan. The row count can be incorrectly
computed as 0, even though there exists data in the Hive table. This
is the stats corruption at table level. Similar stats corruption
exists for a partition. The row count of a table or a partition
sometime can also be -1 which indicates a missing stats situation.

In the fix, as long as no partition in a Hive table exhibits any
missing or corrupt stats, the total row count for the table is computed
from the row counts in all partitions. Otherwise, Impala looks at
the table level stats particularly the table row count.

In addition, if the table stats is missing or corrupted, Impala
estimates a row count for the table, if feasible. This row count is
the sum of the row count from the partitions with good stats, and
an estimation of the number of rows in the partitions with missing or
corrupt stats. Such estimation also applies when some partition
has corrupt stats.

One way to observe the fix is through the explain of queries scanning
Hive tables with missing or corrupted stats. The cardinality for any
full scan should be a positive value (i.e. the estimated row count),
instead of 'unavailable'. At the beginning of the explain output,
that table is still listed in the WARNING section for potentially
corrupt table statistics.

Testing:
1. Ran unit tests with queries documented in the case against Hive
tables with the following configrations:
a. No stats corruption in any partitions
b. Stats corruption in some partitions
c. Stats corruption in all partitions
2. Added two new tests in test_compute_stats.py:
a. test_corrupted_stats_in_partitioned_Hive_tables
b. test_corrupted_stats_in_unpartitioned_Hive_tables
3. Fixed failures in corrupt-stats.test
4. Ran "core" test

Change-Id: I9f4c64616ff7c0b6d5a48f2b5331325feeff3576
---
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M testdata/workloads/functional-planner/queries/PlannerTest/acid-scans.test
M
testdata/workloads/functional-planner/queries/PlannerTest/bloom-filter-assignment.test
M
testdata/workloads/functional-planner/queries/PlannerTest/fk-pk-join-detection-hdfs-num-rows-est-enabled.test
M
testdata/workloads/functional-planner/queries/PlannerTest/min-max-runtime-filters-hdfs-num-rows-est-enabled.test
M
testdata/workloads/functional-planner/queries/PlannerTest/parquet-filtering-disabled.test
M
testdata/workloads/functional-planner/queries/PlannerTest/parquet-filtering.test
M testdata/workloads/functional-planner/queries/PlannerTest/tablesample.test
M testdata/workloads/functional-planner/queries/PlannerTest/union.test
M testdata/workloads/functional-query/queries/QueryTest/corrupt-stats.test
M testdata/workloads/functional-query/queries/QueryTest/stats-extrapolation.test
M tests/metadata/test_compute_stats.py
M tests/metadata/test_explain.py
13 files changed, 235 insertions(+), 82 deletions(-)

git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/98/16098/28
--
To view, visit http://gerrit.cloudera.org:8080/16098
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9f4c64616ff7c0b6d5a48f2b5331325feeff3576
Gerrit-Change-Number: 16098
Gerrit-PatchSet: 28
Gerrit-Owner: Qifan Chen
Gerrit-Reviewer: Aman Sinha
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Qifan Chen
Gerrit-Reviewer: Sahil Takiar
Gerrit-Reviewer: Tim Armstrong

[Impala-ASF-CR] IMPALA-10018: Implement ds kll rank() function

2020-08-07 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/16283 )

Change subject: IMPALA-10018: Implement ds_kll_rank() function
..

IMPALA-10018: Implement ds_kll_rank() function

ds_kll_rank() receives two parameters: a STRING that represents a
serialized DataSketches KLL sketch and a float to provide a probing
value in the sketch.
Returns a DOUBLE that is the rank of the given probing value in the
range of [0,1]. E.g. a return value of 0.2 means that the probing value
given as parameter is greater than the 20% of all the values in the
sketch. Note, this is an approximate calculation.

Change-Id: I95857886dfbb8c84aeeaf718c0e610012fda4be0
Reviewed-on: http://gerrit.cloudera.org:8080/16283
Reviewed-by: Gabor Kaszab 
Tested-by: Impala Public Jenkins 
---
M be/src/exprs/datasketches-functions-ir.cc
M be/src/exprs/datasketches-functions.h
M common/function-registry/impala_functions.py
M testdata/workloads/functional-query/queries/QueryTest/datasketches-kll.test
4 files changed, 76 insertions(+), 5 deletions(-)

Approvals:
  Gabor Kaszab: Looks good to me, approved
  Impala Public Jenkins: Verified

--
To view, visit http://gerrit.cloudera.org:8080/16283
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I95857886dfbb8c84aeeaf718c0e610012fda4be0
Gerrit-Change-Number: 16283
Gerrit-PatchSet: 5
Gerrit-Owner: Gabor Kaszab 
Gerrit-Reviewer: Adam Tamas 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Gabor Kaszab 
Gerrit-Reviewer: Impala Public Jenkins

[Impala-ASF-CR] IMPALA-10018: Implement ds kll rank() function

2020-08-07 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16283 )

Change subject: IMPALA-10018: Implement ds_kll_rank() function
..


Patch Set 4: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/16283
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I95857886dfbb8c84aeeaf718c0e610012fda4be0
Gerrit-Change-Number: 16283
Gerrit-PatchSet: 4
Gerrit-Owner: Gabor Kaszab 
Gerrit-Reviewer: Adam Tamas 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Gabor Kaszab 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Fri, 07 Aug 2020 12:58:39 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9859: Full ACID Milestone 4: Part 2 Reading modified tables (complex types)

2020-08-07 Thread Zoltan Borok-Nagy (Code Review)

Zoltan Borok-Nagy has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16228 )

Change subject: IMPALA-9859: Full ACID Milestone 4: Part 2 Reading modified 
tables (complex types)
..


Patch Set 3:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/16228/3/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java
File fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java:

http://gerrit.cloudera.org:8080/#/c/16228/3/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java@1508
PS3, Line 1508:*   SELECT item FROM complextypestbl $a$1, $a$1.int_array;
> Thanks Zoltan.  Just to clarify further...the plans I see in our test suite
It will involve only one subplan, as such queries are not affected by this 
rewrite.

To be precise, the above query raises an error for me (because o.item is not 
scalar), but the following query has the following plan:

 [localhost:21000] functional_orc_def> explain SELECT t.id, o.item
from functional_orc_def.complextypestbl t left outer join t.int_array o;
 PLAN-ROOT SINK
 |
 05:EXCHANGE [UNPARTITIONED]
 |
 01:SUBPLAN
 |  row-size=24B cardinality=2.57K
 |
 |--04:NESTED LOOP JOIN [RIGHT OUTER JOIN]
 |  |  row-size=24B cardinality=1
 |  |
 |  |--02:SINGULAR ROW SRC
 |  | row-size=20B cardinality=1
 |  |
 |  03:UNNEST [t.int_array o]
 | row-size=0B cardinality=10
 |
 00:SCAN HDFS [functional_orc_def.complextypestbl t]
HDFS partitions=1/1 files=2 size=4.04KB
row-size=20B cardinality=2.57K

You get the same plan if you run it on a non-transactional table.

However, if the query was like this:

 SELECT item from functional_orc_def.complextypestbl.int_array;

Then you'll get the following plan for a non-transactional table:

 PLAN-ROOT SINK
 |
 01:EXCHANGE [UNPARTITIONED]
 |
 00:SCAN HDFS [functional_parquet.complextypestbl.int_array]
HDFS partitions=1/1 files=2 size=6.92KB
row-size=4B cardinality=44.00K

And the following for a full ACID table:

 PLAN-ROOT SINK
 |
 05:EXCHANGE [UNPARTITIONED]
 |
 01:SUBPLAN
 |  row-size=16B cardinality=25.68K
 |
 |--04:NESTED LOOP JOIN [CROSS JOIN]
 |  |  row-size=16B cardinality=10
 |  |
 |  |--02:SINGULAR ROW SRC
 |  | row-size=12B cardinality=1
 |  |
 |  03:UNNEST [$a$1.int_array int_array]
 | row-size=0B cardinality=10
 |
 00:SCAN HDFS [functional_orc_def.complextypestbl $a$1]
HDFS partitions=1/1 files=2 size=4.04KB
predicates: !empty($a$1.int_array)
row-size=12B cardinality=2.57K

But we cannot really avoid this without making significant changes to the 
backend, because the HDFS SCAN node in the non-transactional plan only has a 
single tuple descriptor for the collection item. To return correct results, 
we'd need to smarten up the BE scanner significantly. It would need to
* automatically read the ACID fields
* open and read all the relevant delete delta files
* only return rows that are not deleted

Instead of doing that, with this rewrite we can just create a plan that does 
everything for us. With the rewrite the scan node will have two tuple 
descriptors, one at the table level, and one for the collection items. Then in 
SingleNodePlanner we'll just add the ACID field slot refs to the table level 
tuple, the rest (the subplan) are added automatically.



--
To view, visit http://gerrit.cloudera.org:8080/16228
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8b2c6cd3d87c452c5b96a913b14c90ada78d4c6f
Gerrit-Change-Number: 16228
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy 
Gerrit-Reviewer: Aman Sinha 
Gerrit-Reviewer: Gabor Kaszab 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Comment-Date: Fri, 07 Aug 2020 12:54:52 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10054: Fix flakiness in test multiple sort run bytes limits

2020-08-07 Thread Attila Jeges (Code Review)

Attila Jeges has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16301 )

Change subject: IMPALA-10054: Fix flakiness in 
test_multiple_sort_run_bytes_limits
..


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/16301/1/tests/query_test/test_sort.py
File tests/query_test/test_sort.py:

http://gerrit.cloudera.org:8080/#/c/16301/1/tests/query_test/test_sort.py@90
PS1, Line 90: '   - SpilledRuns:.*'
> nit: Perhaps you could use a more complete regex pattern here:
Also, please use raw strings for regex patterns, e.g.:
r'\s+\- SpilledRuns: %s' % spilled_runs



--
To view, visit http://gerrit.cloudera.org:8080/16301
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I84a8b579c943cddba4432cf183f7f002ef8ec6ad
Gerrit-Change-Number: 16301
Gerrit-PatchSet: 1
Gerrit-Owner: Riza Suminto 
Gerrit-Reviewer: Attila Jeges 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Riza Suminto 
Gerrit-Comment-Date: Fri, 07 Aug 2020 11:02:29 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10017: Implement ds kll union() function

2020-08-07 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16267 )

Change subject: IMPALA-10017: Implement ds_kll_union() function
..


Patch Set 6:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/6824/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/16267
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I020aea28d36f9b6ef9fb57c08411f2170f5c24bf
Gerrit-Change-Number: 16267
Gerrit-PatchSet: 6
Gerrit-Owner: Gabor Kaszab 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Gabor Kaszab 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Fri, 07 Aug 2020 10:56:08 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10017: Implement ds kll union() function

2020-08-07 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16267 )

Change subject: IMPALA-10017: Implement ds_kll_union() function
..


Patch Set 5:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/6823/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/16267
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I020aea28d36f9b6ef9fb57c08411f2170f5c24bf
Gerrit-Change-Number: 16267
Gerrit-PatchSet: 5
Gerrit-Owner: Gabor Kaszab 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Gabor Kaszab 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Fri, 07 Aug 2020 10:45:40 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10017: Implement ds kll union() function

Gabor Kaszab has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16267 )

Change subject: IMPALA-10017: Implement ds_kll_union() function
..


Patch Set 6:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/16267/3/be/src/exprs/aggregate-functions-ir.cc
File be/src/exprs/aggregate-functions-ir.cc:

http://gerrit.cloudera.org:8080/#/c/16267/3/be/src/exprs/aggregate-functions-ir.cc@1851
PS3, Line 1851:
> Can you also add a similar block for HLL (line 1796)? It is ok to do that i
Done



--
To view, visit http://gerrit.cloudera.org:8080/16267
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I020aea28d36f9b6ef9fb57c08411f2170f5c24bf
Gerrit-Change-Number: 16267
Gerrit-PatchSet: 6
Gerrit-Owner: Gabor Kaszab 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Gabor Kaszab 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Fri, 07 Aug 2020 10:44:16 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10017: Implement ds kll union() function

Hello Csaba Ringhofer, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/16267

to look at the new patch set (#6).

Change subject: IMPALA-10017: Implement ds_kll_union() function
..

IMPALA-10017: Implement ds_kll_union() function

This function receives a set of serialized Apache DataSketches KLL
sketches produced by ds_kll_sketch() and merges them into a single
sketch.

An example usage is to create a sketch for each partition of a table,
write these sketches to a separate table and based on which partition
the user is interested of the relevant sketches can be union-ed
together to get an estimate. E.g.:
  SELECT
  ds_kll_quantile(ds_kll_union(sketch_col), 0.5)
  FROM sketch_tbl
  WHERE partition_col=1 OR partition_col=5;

Testing:
  - Apart from the automated tests I added to this patch I also
tested ds_kll_union() on a bigger dataset to check that
serialization, deserialization and merging steps work well. I
took TPCH25.linelitem, created a number of sketches with grouping
by l_shipdate and called ds_kll_union() on those sketches.

Change-Id: I020aea28d36f9b6ef9fb57c08411f2170f5c24bf
---
M be/src/exprs/aggregate-functions-ir.cc
M be/src/exprs/aggregate-functions.h
M fe/src/main/java/org/apache/impala/catalog/BuiltinsDb.java
M testdata/data/README
A testdata/data/kll_sketches_from_impala.parquet
M testdata/workloads/functional-query/queries/QueryTest/datasketches-kll.test
M tests/query_test/test_datasketches.py
7 files changed, 204 insertions(+), 39 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/67/16267/6
--
To view, visit http://gerrit.cloudera.org:8080/16267
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I020aea28d36f9b6ef9fb57c08411f2170f5c24bf
Gerrit-Change-Number: 16267
Gerrit-PatchSet: 6
Gerrit-Owner: Gabor Kaszab 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Gabor Kaszab 
Gerrit-Reviewer: Impala Public Jenkins

[Impala-ASF-CR] IMPALA-10017: Implement ds kll union() function

Gabor Kaszab has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16267 )

Change subject: IMPALA-10017: Implement ds_kll_union() function
..


Patch Set 5:

PS5 is a rebase with master.


--
To view, visit http://gerrit.cloudera.org:8080/16267
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I020aea28d36f9b6ef9fb57c08411f2170f5c24bf
Gerrit-Change-Number: 16267
Gerrit-PatchSet: 5
Gerrit-Owner: Gabor Kaszab 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Gabor Kaszab 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Fri, 07 Aug 2020 10:24:30 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10017: Implement ds kll union() function

Hello Csaba Ringhofer, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/16267

to look at the new patch set (#5).

Change subject: IMPALA-10017: Implement ds_kll_union() function
..

IMPALA-10017: Implement ds_kll_union() function

This function receives a set of serialized Apache DataSketches KLL
sketches produced by ds_kll_sketch() and merges them into a single
sketch.

An example usage is to create a sketch for each partition of a table,
write these sketches to a separate table and based on which partition
the user is interested of the relevant sketches can be union-ed
together to get an estimate. E.g.:
  SELECT
  ds_kll_quantile(ds_kll_union(sketch_col), 0.5)
  FROM sketch_tbl
  WHERE partition_col=1 OR partition_col=5;

Testing:
  - Apart from the automated tests I added to this patch I also
tested ds_kll_union() on a bigger dataset to check that
serialization, deserialization and merging steps work well. I
took TPCH25.linelitem, created a number of sketches with grouping
by l_shipdate and called ds_kll_union() on those sketches.

Change-Id: I020aea28d36f9b6ef9fb57c08411f2170f5c24bf
---
M be/src/exprs/aggregate-functions-ir.cc
M be/src/exprs/aggregate-functions.h
M fe/src/main/java/org/apache/impala/catalog/BuiltinsDb.java
M testdata/data/README
A testdata/data/kll_sketches_from_impala.parquet
M testdata/workloads/functional-query/queries/QueryTest/datasketches-kll.test
M tests/query_test/test_datasketches.py
7 files changed, 199 insertions(+), 37 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/67/16267/5
--
To view, visit http://gerrit.cloudera.org:8080/16267
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I020aea28d36f9b6ef9fb57c08411f2170f5c24bf
Gerrit-Change-Number: 16267
Gerrit-PatchSet: 5
Gerrit-Owner: Gabor Kaszab 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Gabor Kaszab 
Gerrit-Reviewer: Impala Public Jenkins

[Impala-ASF-CR] IMPALA-10054: Fix flakiness in test multiple sort run bytes limits

2020-08-07 Thread Attila Jeges (Code Review)

Attila Jeges has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16301 )

Change subject: IMPALA-10054: Fix flakiness in 
test_multiple_sort_run_bytes_limits
..


Patch Set 1: Code-Review+1

(1 comment)

http://gerrit.cloudera.org:8080/#/c/16301/1/tests/query_test/test_sort.py
File tests/query_test/test_sort.py:

http://gerrit.cloudera.org:8080/#/c/16301/1/tests/query_test/test_sort.py@90
PS1, Line 90: '   - SpilledRuns:.*'
nit: Perhaps you could use a more complete regex pattern here:

'\s+\- SpilledRuns: %s.*' % spilled_runs

and then you can remove the extra check in L92.

You can also use re.search() instead of re.findall() since you don't need to 
scan the whole runtime profile after the first match.



--
To view, visit http://gerrit.cloudera.org:8080/16301
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I84a8b579c943cddba4432cf183f7f002ef8ec6ad
Gerrit-Change-Number: 16301
Gerrit-PatchSet: 1
Gerrit-Owner: Riza Suminto 
Gerrit-Reviewer: Attila Jeges 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Riza Suminto 
Gerrit-Comment-Date: Fri, 07 Aug 2020 10:02:09 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10018: Implement ds kll rank() function

2020-08-07 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16283 )

Change subject: IMPALA-10018: Implement ds_kll_rank() function
..


Patch Set 4:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/6822/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/16283
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I95857886dfbb8c84aeeaf718c0e610012fda4be0
Gerrit-Change-Number: 16283
Gerrit-PatchSet: 4
Gerrit-Owner: Gabor Kaszab 
Gerrit-Reviewer: Adam Tamas 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Gabor Kaszab 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Fri, 07 Aug 2020 08:09:36 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10018: Implement ds kll rank() function

2020-08-07 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/16283 )

Change subject: IMPALA-10018: Implement ds_kll_rank() function
..


Patch Set 4:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6245/ 
DRY_RUN=false


--
To view, visit http://gerrit.cloudera.org:8080/16283
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I95857886dfbb8c84aeeaf718c0e610012fda4be0
Gerrit-Change-Number: 16283
Gerrit-PatchSet: 4
Gerrit-Owner: Gabor Kaszab 
Gerrit-Reviewer: Adam Tamas 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Gabor Kaszab 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Fri, 07 Aug 2020 07:45:48 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10018: Implement ds kll rank() function