[Impala-ASF-CR] IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor

2019-08-16 Thread Vihang Karajgaonkar (Code Review)
Vihang Karajgaonkar has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13932 )

Change subject: IMPALA-8661 : Add randomized tests to stress 
MetastoreEventsProcessor
..


Patch Set 9: Code-Review+2

carrying forward Bharath's +2 from earlier.


--
To view, visit http://gerrit.cloudera.org:8080/13932
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce
Gerrit-Change-Number: 13932
Gerrit-PatchSet: 9
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Fri, 16 Aug 2019 22:14:52 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor

2019-08-16 Thread Vihang Karajgaonkar (Code Review)
Vihang Karajgaonkar has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/13932 )

Change subject: IMPALA-8661 : Add randomized tests to stress 
MetastoreEventsProcessor
..

IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor

This change adds a new stress test for MetastoreEventsProcessor. This
test randomly executes hive queries to generate a lot of events. The
event processor is invoked at random intervals so that a variable batch
of events is processed everytime. After each batch is processed, the
test checks the status of events processor. By default, on CDH builds
the test is configured to run with 4 concurrent Hive clients and each
of the client runs 50 random Hive queries. These defaults can be
overridden by passing system properties using maven command arguments
"-DnumClients" and "-DnumQueriesPerClients". Additionally, the test
also creates impala clients which keep issuing refresh table commands
on the test databases to make sure that eventProcessor is doing some
real work rather than invalidating/refreshing tables which are
already incomplete.

This test is added as a junit test and uses Hive JDBC to issue the sqls.
This is much faster than the end-to-end python test which issues each
hive query in a separate beeline sessions which re-establishes the
connection every time.

Notes:
1. Ran the test with defaults. It generates about 500 events
and runs for close to 4.5 min. This can be changed to a lower
value if we see significant increased delay in the test job runtimes.
3. On CDP builds the concurrent hive queries run very slow due to
container provisioning time on the minicluster. I have left this as a
TODO to investigate. The test runs in single threaded mode with
increased number of queries when running against Hive-3

Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce
Reviewed-on: http://gerrit.cloudera.org:8080/13932
Tested-by: Impala Public Jenkins 
Reviewed-by: Vihang Karajgaonkar 
---
M 
fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
A 
fe/src/test/java/org/apache/impala/catalog/events/EventsProcessorStressTest.java
A fe/src/test/java/org/apache/impala/testutil/HiveJdbcClientPool.java
M fe/src/test/java/org/apache/impala/testutil/ImpalaJdbcClient.java
M fe/src/test/java/org/apache/impala/testutil/TestUtils.java
A fe/src/test/java/org/apache/impala/util/RandomHiveQueryRunner.java
6 files changed, 1,598 insertions(+), 24 deletions(-)

Approvals:
  Impala Public Jenkins: Verified
  Vihang Karajgaonkar: Looks good to me, approved

--
To view, visit http://gerrit.cloudera.org:8080/13932
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce
Gerrit-Change-Number: 13932
Gerrit-PatchSet: 10
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Vihang Karajgaonkar 


[Impala-ASF-CR] IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor

2019-08-16 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13932 )

Change subject: IMPALA-8661 : Add randomized tests to stress 
MetastoreEventsProcessor
..


Patch Set 9: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/13932
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce
Gerrit-Change-Number: 13932
Gerrit-PatchSet: 9
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Fri, 16 Aug 2019 20:33:39 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor

2019-08-16 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13932 )

Change subject: IMPALA-8661 : Add randomized tests to stress 
MetastoreEventsProcessor
..


Patch Set 9:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4806/ 
DRY_RUN=true


--
To view, visit http://gerrit.cloudera.org:8080/13932
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce
Gerrit-Change-Number: 13932
Gerrit-PatchSet: 9
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Fri, 16 Aug 2019 16:27:16 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor

2019-08-16 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13932 )

Change subject: IMPALA-8661 : Add randomized tests to stress 
MetastoreEventsProcessor
..


Patch Set 9:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/4276/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/13932
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce
Gerrit-Change-Number: 13932
Gerrit-PatchSet: 9
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Fri, 16 Aug 2019 07:09:27 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor

2019-08-16 Thread Vihang Karajgaonkar (Code Review)
Vihang Karajgaonkar has uploaded a new patch set (#9). ( 
http://gerrit.cloudera.org:8080/13932 )

Change subject: IMPALA-8661 : Add randomized tests to stress 
MetastoreEventsProcessor
..

IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor

This change adds a new stress test for MetastoreEventsProcessor. This
test randomly executes hive queries to generate a lot of events. The
event processor is invoked at random intervals so that a variable batch
of events is processed everytime. After each batch is processed, the
test checks the status of events processor. By default, on CDH builds
the test is configured to run with 4 concurrent Hive clients and each
of the client runs 50 random Hive queries. These defaults can be
overridden by passing system properties using maven command arguments
"-DnumClients" and "-DnumQueriesPerClients". Additionally, the test
also creates impala clients which keep issuing refresh table commands
on the test databases to make sure that eventProcessor is doing some
real work rather than invalidating/refreshing tables which are
already incomplete.

This test is added as a junit test and uses Hive JDBC to issue the sqls.
This is much faster than the end-to-end python test which issues each
hive query in a separate beeline sessions which re-establishes the
connection every time.

Notes:
1. Ran the test with defaults. It generates about 500 events
and runs for close to 4.5 min. This can be changed to a lower
value if we see significant increased delay in the test job runtimes.
3. On CDP builds the concurrent hive queries run very slow due to
container provisioning time on the minicluster. I have left this as a
TODO to investigate. The test runs in single threaded mode with
increased number of queries when running against Hive-3

Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce
---
M 
fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
A 
fe/src/test/java/org/apache/impala/catalog/events/EventsProcessorStressTest.java
A fe/src/test/java/org/apache/impala/testutil/HiveJdbcClientPool.java
M fe/src/test/java/org/apache/impala/testutil/ImpalaJdbcClient.java
M fe/src/test/java/org/apache/impala/testutil/TestUtils.java
A fe/src/test/java/org/apache/impala/util/RandomHiveQueryRunner.java
6 files changed, 1,598 insertions(+), 24 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/13932/9
--
To view, visit http://gerrit.cloudera.org:8080/13932
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce
Gerrit-Change-Number: 13932
Gerrit-PatchSet: 9
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Vihang Karajgaonkar 


[Impala-ASF-CR] IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor

2019-08-15 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13932 )

Change subject: IMPALA-8661 : Add randomized tests to stress 
MetastoreEventsProcessor
..


Patch Set 8: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4801/


--
To view, visit http://gerrit.cloudera.org:8080/13932
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce
Gerrit-Change-Number: 13932
Gerrit-PatchSet: 8
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Fri, 16 Aug 2019 00:14:15 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor

2019-08-15 Thread Bharath Vissapragada (Code Review)
Bharath Vissapragada has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13932 )

Change subject: IMPALA-8661 : Add randomized tests to stress 
MetastoreEventsProcessor
..


Patch Set 8: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/13932
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce
Gerrit-Change-Number: 13932
Gerrit-PatchSet: 8
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Thu, 15 Aug 2019 22:07:29 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor

2019-08-15 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13932 )

Change subject: IMPALA-8661 : Add randomized tests to stress 
MetastoreEventsProcessor
..


Patch Set 8:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4801/ 
DRY_RUN=true


--
To view, visit http://gerrit.cloudera.org:8080/13932
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce
Gerrit-Change-Number: 13932
Gerrit-PatchSet: 8
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Thu, 15 Aug 2019 20:07:20 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor

2019-08-15 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13932 )

Change subject: IMPALA-8661 : Add randomized tests to stress 
MetastoreEventsProcessor
..


Patch Set 7:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/4267/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/13932
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce
Gerrit-Change-Number: 13932
Gerrit-PatchSet: 7
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Thu, 15 Aug 2019 19:36:06 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor

2019-08-15 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13932 )

Change subject: IMPALA-8661 : Add randomized tests to stress 
MetastoreEventsProcessor
..


Patch Set 8:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/4268/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/13932
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce
Gerrit-Change-Number: 13932
Gerrit-PatchSet: 8
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Thu, 15 Aug 2019 19:35:39 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor

2019-08-15 Thread Vihang Karajgaonkar (Code Review)
Vihang Karajgaonkar has uploaded a new patch set (#8). ( 
http://gerrit.cloudera.org:8080/13932 )

Change subject: IMPALA-8661 : Add randomized tests to stress 
MetastoreEventsProcessor
..

IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor

This change adds a new stress test for MetastoreEventsProcessor. This
test randomly executes hive queries to generate a lot of events. The
event processor is invoked at random intervals so that a variable batch
of events is processed everytime. After each batch is processed, the
test checks the status of events processor. By default, on CDH builds
the test is configured to run with 4 concurrent Hive clients and each
of the client runs 50 random Hive queries. These defaults can be
overridden by passing system properties using maven command arguments
"-DnumClients" and "-DnumQueriesPerClients". Additionally, the test
also creates impala clients which keep issuing refresh table commands
on the test databases to make sure that eventProcessor is doing some
real work rather than invalidating/refreshing tables which are
already incomplete.

This test is added as a junit test and uses Hive JDBC to issue the sqls.
This is much faster than the end-to-end python test which issues each
hive query in a separate beeline sessions which re-establishes the
connection every time.

Notes:
1. Ran the test with defaults. It generates about 500 events
and runs for close to 4.5 min. This can be changed to a lower
value if we see significant increased delay in the test job runtimes.
3. On CDP builds the concurrent hive queries run very slow due to
container provisioning time on the minicluster. I have left this as a
TODO to investigate. The test runs in single threaded mode with
increased number of queries when running against Hive-3

Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce
---
M 
fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
A 
fe/src/test/java/org/apache/impala/catalog/events/EventsProcessorStressTest.java
A fe/src/test/java/org/apache/impala/testutil/HiveJdbcClientPool.java
M fe/src/test/java/org/apache/impala/testutil/ImpalaJdbcClient.java
M fe/src/test/java/org/apache/impala/testutil/TestUtils.java
A fe/src/test/java/org/apache/impala/util/RandomHiveQueryRunner.java
6 files changed, 1,590 insertions(+), 24 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/13932/8
--
To view, visit http://gerrit.cloudera.org:8080/13932
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce
Gerrit-Change-Number: 13932
Gerrit-PatchSet: 8
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Vihang Karajgaonkar 


[Impala-ASF-CR] IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor

2019-08-15 Thread Vihang Karajgaonkar (Code Review)
Vihang Karajgaonkar has uploaded a new patch set (#7). ( 
http://gerrit.cloudera.org:8080/13932 )

Change subject: IMPALA-8661 : Add randomized tests to stress 
MetastoreEventsProcessor
..

IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor

This change adds a new stress test for MetastoreEventsProcessor. This
test randomly executes hive queries to generate a lot of events. The
event processor is invoked at random intervals so that a variable batch
of events is processed everytime. After each batch is processed, the
test checks the status of events processor. By default, on CDH builds
the test is configured to run with 8 concurrent Hive clients and each
of the client runs 100 random Hive queries. These defaults can be
overridden by passing system properties using maven command arguments
"-DnumClients" and "-DnumQueriesPerClients". Additionally, the test
also creates impala clients which keep issuing refresh table commands
on the test databases to make sure that eventProcessor is doing some
real work rather than invalidating/refreshing tables which are
already incomplete.

This test is added as a junit test and uses Hive JDBC to issue the sqls.
This is much faster than the end-to-end python test which issues each
hive query in a separate beeline sessions which re-establishes the
connection every time.

Notes:
1. Ran the test with defaults. It generates about 2100 events
and runs for close to 15 min. This can be changed to a lower
value if we see significant increased delay in the test job runtimes.
3. On CDP builds the concurrent hive queries run very slow due to
container provisioning time on the minicluster. I have left this as a
TODO to investigate. The test runs in single threaded mode with
increased number of queries when running against Hive-3

Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce
---
M 
fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
A 
fe/src/test/java/org/apache/impala/catalog/events/EventsProcessorStressTest.java
A fe/src/test/java/org/apache/impala/testutil/HiveJdbcClientPool.java
M fe/src/test/java/org/apache/impala/testutil/ImpalaJdbcClient.java
M fe/src/test/java/org/apache/impala/testutil/TestUtils.java
A fe/src/test/java/org/apache/impala/util/RandomHiveQueryRunner.java
6 files changed, 1,590 insertions(+), 24 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/13932/7
--
To view, visit http://gerrit.cloudera.org:8080/13932
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce
Gerrit-Change-Number: 13932
Gerrit-PatchSet: 7
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Vihang Karajgaonkar 


[Impala-ASF-CR] IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor

2019-07-31 Thread Bharath Vissapragada (Code Review)
Bharath Vissapragada has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13932 )

Change subject: IMPALA-8661 : Add randomized tests to stress 
MetastoreEventsProcessor
..


Patch Set 6:

(1 comment)

lgtm pending the test timings.

http://gerrit.cloudera.org:8080/#/c/13932/5//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/13932/5//COMMIT_MSG@35
PS5, Line 35: 1. Ran the test with defaults. It generates about 2100 events
> I think the fe tests are not the long pole in the test execution, right? So
Doesn't this job run everything serially? be tests -> fe-tests->e-e tests

https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/6891/consoleFull

(just picked up random job from verify-dry-run)



--
To view, visit http://gerrit.cloudera.org:8080/13932
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce
Gerrit-Change-Number: 13932
Gerrit-PatchSet: 6
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Thu, 01 Aug 2019 02:24:16 +
Gerrit-HasComments: Yes


[Impala-ASF-CR] IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor

2019-07-30 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13932 )

Change subject: IMPALA-8661 : Add randomized tests to stress 
MetastoreEventsProcessor
..


Patch Set 6: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4695/


--
To view, visit http://gerrit.cloudera.org:8080/13932
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce
Gerrit-Change-Number: 13932
Gerrit-PatchSet: 6
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Wed, 31 Jul 2019 04:50:11 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor

2019-07-30 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13932 )

Change subject: IMPALA-8661 : Add randomized tests to stress 
MetastoreEventsProcessor
..


Patch Set 6:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/4086/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/13932
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce
Gerrit-Change-Number: 13932
Gerrit-PatchSet: 6
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Tue, 30 Jul 2019 22:43:05 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor

2019-07-30 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13932 )

Change subject: IMPALA-8661 : Add randomized tests to stress 
MetastoreEventsProcessor
..


Patch Set 6:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4695/ 
DRY_RUN=true


--
To view, visit http://gerrit.cloudera.org:8080/13932
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce
Gerrit-Change-Number: 13932
Gerrit-PatchSet: 6
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Tue, 30 Jul 2019 22:11:01 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor

2019-07-30 Thread Vihang Karajgaonkar (Code Review)
Vihang Karajgaonkar has uploaded a new patch set (#6). ( 
http://gerrit.cloudera.org:8080/13932 )

Change subject: IMPALA-8661 : Add randomized tests to stress 
MetastoreEventsProcessor
..

IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor

This change adds a new stress test for MetastoreEventsProcessor. This
test randomly executes hive queries to generate a lot of events. The
event processor is invoked at random intervals so that a variable batch
of events is processed everytime. After each batch is processed, the
test checks the status of events processor. By default, on CDH builds
the test is configured to run with 8 concurrent Hive clients and each
of the client runs 100 random Hive queries. These defaults can be
overridden by passing system properties using maven command arguments
"-DnumClients" and "-DnumQueriesPerClients". Additionally, the test
also creates impala clients which keep issuing refresh table commands
on the test databases to make sure that eventProcessor is doing some
real work rather than invalidating/refreshing tables which are
already incomplete.

This test is added as a junit test and uses Hive JDBC to issue the sqls.
This is much faster than the end-to-end python test which issues each
hive query in a separate beeline sessions which re-establishes the
connection every time.

The test already found a bug which is caused when a Hive issues a alter
table add if not exists partition" query and the partition is not really
added since it is preexisting. In such a case the ADD_PARTITION events
is still generated but with a empty list of partitions in the events.
Such events are now ignored.

Notes:
1. Ran the test with defaults. It generates about 2100 events
and runs for close to 15 min. This can be changed to a lower
value if we see significant increased delay in the test job runtimes.
3. On CDP builds the concurrent hive queries run very slow due to
container provisioning time on the minicluster. I have left this as a
TODO to investigate. The test runs in single threaded mode with
increased number of queries when running against Hive-3

Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce
---
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M 
fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
A 
fe/src/test/java/org/apache/impala/catalog/events/EventsProcessorStressTest.java
A fe/src/test/java/org/apache/impala/testutil/HiveJdbcClientPool.java
M fe/src/test/java/org/apache/impala/testutil/ImpalaJdbcClient.java
M fe/src/test/java/org/apache/impala/testutil/TestUtils.java
A fe/src/test/java/org/apache/impala/util/RandomHiveQueryRunner.java
7 files changed, 1,591 insertions(+), 34 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/13932/6
--
To view, visit http://gerrit.cloudera.org:8080/13932
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce
Gerrit-Change-Number: 13932
Gerrit-PatchSet: 6
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Vihang Karajgaonkar 


[Impala-ASF-CR] IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor

2019-07-30 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13932 )

Change subject: IMPALA-8661 : Add randomized tests to stress 
MetastoreEventsProcessor
..


Patch Set 3: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4689/


--
To view, visit http://gerrit.cloudera.org:8080/13932
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce
Gerrit-Change-Number: 13932
Gerrit-PatchSet: 3
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Tue, 30 Jul 2019 12:31:19 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor

2019-07-30 Thread Bharath Vissapragada (Code Review)
Bharath Vissapragada has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13932 )

Change subject: IMPALA-8661 : Add randomized tests to stress 
MetastoreEventsProcessor
..


Patch Set 5:

(14 comments)

Great work, exhaustive coverage. I skimmed through most parts of the patch. 
seems reasonable to me.  I think it is difficult to review the entire patch in 
a single go. We should probably commit it and keep fixing any issues 
incrementally. Since most of the patch is isolated to test framework, any bugs 
(like connection leaks, if any) shouldn't affect the product itself.

http://gerrit.cloudera.org:8080/#/c/13932/5//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/13932/5//COMMIT_MSG@35
PS5, Line 35: 1. Ran the test with defaults. It generates about 2100 events
Should we consider lower defaults for core tests? 15mins seems like a 
considerable increase in test execution time.


http://gerrit.cloudera.org:8080/#/c/13932/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:

http://gerrit.cloudera.org:8080/#/c/13932/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@299
PS5, Line 299:   infoLog("Notification event received");
nit: isn't this too chatty for info?


http://gerrit.cloudera.org:8080/#/c/13932/5/fe/src/test/java/org/apache/impala/catalog/events/EventsProcessorStressTest.java
File 
fe/src/test/java/org/apache/impala/catalog/events/EventsProcessorStressTest.java:

http://gerrit.cloudera.org:8080/#/c/13932/5/fe/src/test/java/org/apache/impala/catalog/events/EventsProcessorStressTest.java@46
PS5, Line 46: refresh
nit: grammar


http://gerrit.cloudera.org:8080/#/c/13932/5/fe/src/test/java/org/apache/impala/catalog/events/EventsProcessorStressTest.java@83
PS5, Line 83: else
do we want to override numClients in 3.x? in which case it should probably be 
something like

if (version < 3 && numClients != null)...


http://gerrit.cloudera.org:8080/#/c/13932/5/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
File 
fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java:

http://gerrit.cloudera.org:8080/#/c/13932/5/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@138
PS5, Line 138: import org.mockito.Mockito;
remove?


http://gerrit.cloudera.org:8080/#/c/13932/5/fe/src/test/java/org/apache/impala/testutil/HiveJdbcClientPool.java
File fe/src/test/java/org/apache/impala/testutil/HiveJdbcClientPool.java:

http://gerrit.cloudera.org:8080/#/c/13932/5/fe/src/test/java/org/apache/impala/testutil/HiveJdbcClientPool.java@28
PS5, Line 28: import java.util.concurrent.BlockingQueue;
nit: duplicate.


http://gerrit.cloudera.org:8080/#/c/13932/5/fe/src/test/java/org/apache/impala/testutil/HiveJdbcClientPool.java@36
PS5, Line 36: public class HiveJdbcClientPool implements Closeable {
class comment.


http://gerrit.cloudera.org:8080/#/c/13932/5/fe/src/test/java/org/apache/impala/testutil/HiveJdbcClientPool.java@71
PS5, Line 71:   if (conn_ == null) {
: throw new RuntimeException("Connection not initialized.");
:   } else if (conn_.isClosed()) {
: throw new RuntimeException("Connection not open.");
:   }
Make them preconditions?


http://gerrit.cloudera.org:8080/#/c/13932/5/fe/src/test/java/org/apache/impala/testutil/HiveJdbcClientPool.java@94
PS5, Line 94: public boolean executeSql(String sql) throws SQLException {
method doc.


http://gerrit.cloudera.org:8080/#/c/13932/5/fe/src/test/java/org/apache/impala/testutil/HiveJdbcClientPool.java@146
PS5, Line 146: closedCount > 0
!freeClients_.isEmpty() ?


http://gerrit.cloudera.org:8080/#/c/13932/5/fe/src/test/java/org/apache/impala/testutil/TestUtils.java
File fe/src/test/java/org/apache/impala/testutil/TestUtils.java:

http://gerrit.cloudera.org:8080/#/c/13932/5/fe/src/test/java/org/apache/impala/testutil/TestUtils.java@42
PS5, Line 42: import org.apache.commons.lang3.RandomStringUtils;
remove?


http://gerrit.cloudera.org:8080/#/c/13932/5/fe/src/test/java/org/apache/impala/testutil/TestUtils.java@77
PS5, Line 77: DEFAULT_CONNECTION_TEMPLATE
nit:HS2_CONNECTION_TEMPLATE?


http://gerrit.cloudera.org:8080/#/c/13932/5/fe/src/test/java/org/apache/impala/util/RandomHiveQueryRunner.java
File fe/src/test/java/org/apache/impala/util/RandomHiveQueryRunner.java:

http://gerrit.cloudera.org:8080/#/c/13932/5/fe/src/test/java/org/apache/impala/util/RandomHiveQueryRunner.java@117
PS5, Line 117: for
nit: remove


http://gerrit.cloudera.org:8080/#/c/13932/5/fe/src/test/java/org/apache/impala/util/RandomHiveQueryRunner.java@345
PS5, Line 345: exists
exist



--
To view, visit http://gerrit.cloudera.org:8080/13932
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF

[Impala-ASF-CR] IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor

2019-07-30 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13932 )

Change subject: IMPALA-8661 : Add randomized tests to stress 
MetastoreEventsProcessor
..


Patch Set 5:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/4075/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/13932
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce
Gerrit-Change-Number: 13932
Gerrit-PatchSet: 5
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Tue, 30 Jul 2019 06:53:45 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor

2019-07-30 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13932 )

Change subject: IMPALA-8661 : Add randomized tests to stress 
MetastoreEventsProcessor
..


Patch Set 4:

Build Failed

https://jenkins.impala.io/job/gerrit-code-review-checks/4074/ : Initial code 
review checks failed. See linked job for details on the failure.


--
To view, visit http://gerrit.cloudera.org:8080/13932
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce
Gerrit-Change-Number: 13932
Gerrit-PatchSet: 4
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Tue, 30 Jul 2019 06:38:55 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor

2019-07-30 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13932 )

Change subject: IMPALA-8661 : Add randomized tests to stress 
MetastoreEventsProcessor
..


Patch Set 5:

(5 comments)

http://gerrit.cloudera.org:8080/#/c/13932/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:

http://gerrit.cloudera.org:8080/#/c/13932/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1343
PS5, Line 1343:   // when multiple partitions are added in HMS they are 
all added as one transaction
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/13932/5/fe/src/test/java/org/apache/impala/testutil/ImpalaJdbcClient.java
File fe/src/test/java/org/apache/impala/testutil/ImpalaJdbcClient.java:

http://gerrit.cloudera.org:8080/#/c/13932/5/fe/src/test/java/org/apache/impala/testutil/ImpalaJdbcClient.java@142
PS5, Line 142: return createClient(TestUtils.HIVE_SERVER2_DRIVER_NAME, 
getNoAuthConnectionStr("binary"));
line too long (94 > 90)


http://gerrit.cloudera.org:8080/#/c/13932/5/fe/src/test/java/org/apache/impala/testutil/ImpalaJdbcClient.java@150
PS5, Line 150: return createClient(TestUtils.HIVE_SERVER2_DRIVER_NAME, 
getNoAuthConnectionStr("http"));
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/13932/5/fe/src/test/java/org/apache/impala/util/RandomHiveQueryRunner.java
File fe/src/test/java/org/apache/impala/util/RandomHiveQueryRunner.java:

http://gerrit.cloudera.org:8080/#/c/13932/5/fe/src/test/java/org/apache/impala/util/RandomHiveQueryRunner.java@838
PS5, Line 838:  query.addDependentQuery(new 
SetConfigStmt("tez.session.am.dag.submit.timeout.secs", "2"));
line too long (99 > 90)


http://gerrit.cloudera.org:8080/#/c/13932/5/fe/src/test/java/org/apache/impala/util/RandomHiveQueryRunner.java@918
PS5, Line 918:  query.addDependentQuery(new 
SetConfigStmt("tez.session.am.dag.submit.timeout.secs", "2"));
line too long (99 > 90)



--
To view, visit http://gerrit.cloudera.org:8080/13932
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce
Gerrit-Change-Number: 13932
Gerrit-PatchSet: 5
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Tue, 30 Jul 2019 06:36:00 +
Gerrit-HasComments: Yes


[Impala-ASF-CR] IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor

2019-07-30 Thread Vihang Karajgaonkar (Code Review)
Hello Bharath Vissapragada, Quanlong Huang, Anurag Mantripragada, Impala Public 
Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/13932

to look at the new patch set (#5).

Change subject: IMPALA-8661 : Add randomized tests to stress 
MetastoreEventsProcessor
..

IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor

This change adds a new stress test for MetastoreEventsProcessor. This
test randomly executes hive queries to generate a lot of events. The
event processor is invoked at random intervals so that a variable batch
of events is processed everytime. After each batch is processed, the
test checks the status of events processor. By default, on CDH builds
the test is configured to run with 8 concurrent Hive clients and each
of the client runs 100 random Hive queries. These defaults can be
overridden by passing system properties using maven command arguments
"-DnumClients" and "-DnumQueriesPerClients". Additionally, the test
also creates impala clients which keep issuing refresh table commands
on the test databases to make sure that eventProcessor is doing some
real work rather than invalidating/refreshing tables which are
already incomplete.

This test is added as a junit test and uses Hive JDBC to issue the sqls.
This is much faster than the end-to-end python test which issues each
hive query in a separate beeline sessions which re-establishes the
connection every time.

The test already found a bug which is caused when a Hive issues a alter
table add if not exists partition" query and the partition is not really
added since it is preexisting. In such a case the ADD_PARTITION events
is still generated but with a empty list of partitions in the events.
Such events are now ignored.

Notes:
1. Ran the test with defaults. It generates about 2100 events
and runs for close to 15 min. This can be changed to a lower
value if we see significant increased delay in the test job runtimes.
3. On CDP builds the concurrent hive queries run very slow due to
container provisioning time on the minicluster. I have left this as a
TODO to investigate. The test runs in single threaded mode with
increased number of queries when running against Hive-3

Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce
---
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M 
fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
A 
fe/src/test/java/org/apache/impala/catalog/events/EventsProcessorStressTest.java
M 
fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
A fe/src/test/java/org/apache/impala/testutil/HiveJdbcClientPool.java
M fe/src/test/java/org/apache/impala/testutil/ImpalaJdbcClient.java
M fe/src/test/java/org/apache/impala/testutil/TestUtils.java
A fe/src/test/java/org/apache/impala/util/RandomHiveQueryRunner.java
8 files changed, 1,578 insertions(+), 34 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/13932/5
-- 
To view, visit http://gerrit.cloudera.org:8080/13932
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce
Gerrit-Change-Number: 13932
Gerrit-PatchSet: 5
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Vihang Karajgaonkar 


[Impala-ASF-CR] IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor

2019-07-30 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13932 )

Change subject: IMPALA-8661 : Add randomized tests to stress 
MetastoreEventsProcessor
..


Patch Set 3:

Build Failed

https://jenkins.impala.io/job/gerrit-code-review-checks/4073/ : Initial code 
review checks failed. See linked job for details on the failure.


--
To view, visit http://gerrit.cloudera.org:8080/13932
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce
Gerrit-Change-Number: 13932
Gerrit-PatchSet: 3
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Tue, 30 Jul 2019 06:22:47 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor

2019-07-30 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13932 )

Change subject: IMPALA-8661 : Add randomized tests to stress 
MetastoreEventsProcessor
..


Patch Set 4:

(5 comments)

http://gerrit.cloudera.org:8080/#/c/13932/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:

http://gerrit.cloudera.org:8080/#/c/13932/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1343
PS4, Line 1343:   // when multiple partitions are added in HMS they are 
all added as one transaction
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/13932/4/fe/src/test/java/org/apache/impala/testutil/ImpalaJdbcClient.java
File fe/src/test/java/org/apache/impala/testutil/ImpalaJdbcClient.java:

http://gerrit.cloudera.org:8080/#/c/13932/4/fe/src/test/java/org/apache/impala/testutil/ImpalaJdbcClient.java@142
PS4, Line 142: return createClient(TestUtils.HIVE_SERVER2_DRIVER_NAME, 
getNoAuthConnectionStr("binary"));
line too long (94 > 90)


http://gerrit.cloudera.org:8080/#/c/13932/4/fe/src/test/java/org/apache/impala/testutil/ImpalaJdbcClient.java@150
PS4, Line 150: return createClient(TestUtils.HIVE_SERVER2_DRIVER_NAME, 
getNoAuthConnectionStr("http"));
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/13932/4/fe/src/test/java/org/apache/impala/util/RandomHiveQueryRunner.java
File fe/src/test/java/org/apache/impala/util/RandomHiveQueryRunner.java:

http://gerrit.cloudera.org:8080/#/c/13932/4/fe/src/test/java/org/apache/impala/util/RandomHiveQueryRunner.java@838
PS4, Line 838:  query.addDependentQuery(new 
SetConfigStmt("tez.session.am.dag.submit.timeout.secs", "2"));
line too long (99 > 90)


http://gerrit.cloudera.org:8080/#/c/13932/4/fe/src/test/java/org/apache/impala/util/RandomHiveQueryRunner.java@918
PS4, Line 918:  query.addDependentQuery(new 
SetConfigStmt("tez.session.am.dag.submit.timeout.secs", "2"));
line too long (99 > 90)



--
To view, visit http://gerrit.cloudera.org:8080/13932
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce
Gerrit-Change-Number: 13932
Gerrit-PatchSet: 4
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Tue, 30 Jul 2019 06:20:59 +
Gerrit-HasComments: Yes


[Impala-ASF-CR] IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor

2019-07-30 Thread Vihang Karajgaonkar (Code Review)
Vihang Karajgaonkar has uploaded a new patch set (#4). ( 
http://gerrit.cloudera.org:8080/13932 )

Change subject: IMPALA-8661 : Add randomized tests to stress 
MetastoreEventsProcessor
..

IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor

This change adds a new stress test for MetastoreEventsProcessor. This
test randomly executes hive queries to generate a lot of events. The
event processor is invoked at random intervals so that a variable batch
of events is processed everytime. After each batch is processed, the
test checks the status of events processor. By default, on CDH builds
the test is configured to run with 8 concurrent Hive clients and each
of the client runs 100 random Hive queries. These defaults can be
overridden by passing system properties using maven command arguments
"-DnumClients" and "-DnumQueriesPerClients". Additionally, the test
also creates impala clients which keep issuing refresh table commands
on the test databases to make sure that eventProcessor is doing some
real work rather than invalidating/refreshing tables which are
already incomplete.

This test is added as a junit test and uses Hive JDBC to issue the sqls.
This is much faster than the end-to-end python test which issues each
hive query in a separate beeline sessions which re-establishes the
connection every time.

The test already found a bug which is caused when a Hive issues a alter
table add if not exists partition" query and the partition is not really
added since it is preexisting. In such a case the ADD_PARTITION events
is still generated but with a empty list of partitions in the events.
Such events are now ignored.

Notes:
1. Ran the test with defaults. It generates about 2100 events
and runs for close to 15 min. This can be changed to a lower
value if we see significant increased delay in the test job runtimes.
3. On CDP builds the concurrent hive queries run very slow due to
container provisioning time on the minicluster. I have left this as a
TODO to investigate. The test runs in single threaded mode with
increased number of queries when running against Hive-3

Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce
---
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M 
fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
A 
fe/src/test/java/org/apache/impala/catalog/events/EventsProcessorStressTest.java
M 
fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
A fe/src/test/java/org/apache/impala/testutil/HiveJdbcClientPool.java
M fe/src/test/java/org/apache/impala/testutil/ImpalaJdbcClient.java
M fe/src/test/java/org/apache/impala/testutil/TestUtils.java
A fe/src/test/java/org/apache/impala/util/RandomHiveQueryRunner.java
8 files changed, 1,544 insertions(+), 34 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/13932/4
--
To view, visit http://gerrit.cloudera.org:8080/13932
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce
Gerrit-Change-Number: 13932
Gerrit-PatchSet: 4
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Vihang Karajgaonkar 


[Impala-ASF-CR] IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor

2019-07-29 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13932 )

Change subject: IMPALA-8661 : Add randomized tests to stress 
MetastoreEventsProcessor
..


Patch Set 3:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4689/ 
DRY_RUN=true


--
To view, visit http://gerrit.cloudera.org:8080/13932
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce
Gerrit-Change-Number: 13932
Gerrit-PatchSet: 3
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Tue, 30 Jul 2019 05:53:26 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor

2019-07-29 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13932 )

Change subject: IMPALA-8661 : Add randomized tests to stress 
MetastoreEventsProcessor
..


Patch Set 3:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/13932/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:

http://gerrit.cloudera.org:8080/#/c/13932/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1343
PS3, Line 1343:   // when multiple partitions are added in HMS they are 
all added as one transaction
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/13932/3/fe/src/test/java/org/apache/impala/testutil/ImpalaJdbcClient.java
File fe/src/test/java/org/apache/impala/testutil/ImpalaJdbcClient.java:

http://gerrit.cloudera.org:8080/#/c/13932/3/fe/src/test/java/org/apache/impala/testutil/ImpalaJdbcClient.java@142
PS3, Line 142: return createClient(TestUtils.HIVE_SERVER2_DRIVER_NAME, 
getNoAuthConnectionStr("binary"));
line too long (94 > 90)


http://gerrit.cloudera.org:8080/#/c/13932/3/fe/src/test/java/org/apache/impala/testutil/ImpalaJdbcClient.java@150
PS3, Line 150: return createClient(TestUtils.HIVE_SERVER2_DRIVER_NAME, 
getNoAuthConnectionStr("http"));
line too long (92 > 90)



--
To view, visit http://gerrit.cloudera.org:8080/13932
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce
Gerrit-Change-Number: 13932
Gerrit-PatchSet: 3
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Tue, 30 Jul 2019 05:42:27 +
Gerrit-HasComments: Yes


[Impala-ASF-CR] IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor

2019-07-29 Thread Vihang Karajgaonkar (Code Review)
Vihang Karajgaonkar has uploaded a new patch set (#3). ( 
http://gerrit.cloudera.org:8080/13932 )

Change subject: IMPALA-8661 : Add randomized tests to stress 
MetastoreEventsProcessor
..

IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor

This change adds a new stress test for MetastoreEventsProcessor. This
test randomly executes hive queries to generate a lot of events. The
event processor is invoked at random intervals so that a variable batch
of events is processed everytime. After each batch is processed, the
test checks the status of events processor. By default, on CDH builds
the test is configured to run with 8 concurrent Hive clients and each
of the client runs 100 random Hive queries. These defaults can be
overridden by passing system properties using maven command arguments
"-DnumClients" and "-DnumQueriesPerClients". Additionally, the test
also creates impala clients which keep issuing refresh table commands
on the test databases to make sure that eventProcessor is doing some
real work rather than invalidating/refreshing tables which are
already incomplete.

This test is added as a junit test and uses Hive JDBC to issue the sqls.
This is much faster than the end-to-end python test which issues each
hive query in a separate beeline sessions which re-establishes the
connection every time.

The test already found a bug which is caused when a Hive issues a alter
table add if not exists partition" query and the partition is not really
added since it is preexisting. In such a case the ADD_PARTITION events
is still generated but with a empty list of partitions in the events.
Such events are now ignored.

Notes:
1. Ran the test with defaults. It generates about 2100 events
and runs for close to 15 min. This can be changed to a lower
value if we see significant increased delay in the test job runtimes.
3. On CDP builds the concurrent hive queries run very slow due to
container provisioning time on the minicluster. I have left this as a
TODO to investigate. The test runs in single threaded mode with
increased number of queries when running against Hive-3

Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce
---
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M 
fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
A 
fe/src/test/java/org/apache/impala/catalog/events/EventsProcessorStressTest.java
M 
fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
A fe/src/test/java/org/apache/impala/catalog/events/RandomHiveQueryRunner.java
A fe/src/test/java/org/apache/impala/testutil/HiveJdbcClientPool.java
M fe/src/test/java/org/apache/impala/testutil/ImpalaJdbcClient.java
M fe/src/test/java/org/apache/impala/testutil/TestUtils.java
8 files changed, 1,516 insertions(+), 34 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/13932/3
--
To view, visit http://gerrit.cloudera.org:8080/13932
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce
Gerrit-Change-Number: 13932
Gerrit-PatchSet: 3
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Vihang Karajgaonkar 


[Impala-ASF-CR] IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor

2019-07-29 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13932 )

Change subject: IMPALA-8661 : Add randomized tests to stress 
MetastoreEventsProcessor
..


Patch Set 2: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4680/


--
To view, visit http://gerrit.cloudera.org:8080/13932
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce
Gerrit-Change-Number: 13932
Gerrit-PatchSet: 2
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Tue, 30 Jul 2019 01:02:26 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor

2019-07-29 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13932 )

Change subject: IMPALA-8661 : Add randomized tests to stress 
MetastoreEventsProcessor
..


Patch Set 2:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4680/ 
DRY_RUN=true


--
To view, visit http://gerrit.cloudera.org:8080/13932
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce
Gerrit-Change-Number: 13932
Gerrit-PatchSet: 2
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Mon, 29 Jul 2019 18:29:33 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor

2019-07-26 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13932 )

Change subject: IMPALA-8661 : Add randomized tests to stress 
MetastoreEventsProcessor
..


Patch Set 2:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/4043/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/13932
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce
Gerrit-Change-Number: 13932
Gerrit-PatchSet: 2
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Sat, 27 Jul 2019 00:07:42 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor

2019-07-26 Thread Vihang Karajgaonkar (Code Review)
Vihang Karajgaonkar has removed Todd Lipcon from this change.  ( 
http://gerrit.cloudera.org:8080/13932 )

Change subject: IMPALA-8661 : Add randomized tests to stress 
MetastoreEventsProcessor
..


Removed reviewer Todd Lipcon.
--
To view, visit http://gerrit.cloudera.org:8080/13932
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: deleteReviewer
Gerrit-Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce
Gerrit-Change-Number: 13932
Gerrit-PatchSet: 2
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Vihang Karajgaonkar 


[Impala-ASF-CR] IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor

2019-07-26 Thread Vihang Karajgaonkar (Code Review)
Vihang Karajgaonkar has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13932 )

Change subject: IMPALA-8661 : Add randomized tests to stress 
MetastoreEventsProcessor
..


Patch Set 2:

> I'm out the next two weeks on PTO, so probably best if someone else
 > reviews this

Thanks, I will remove you from the reviewers list.


--
To view, visit http://gerrit.cloudera.org:8080/13932
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce
Gerrit-Change-Number: 13932
Gerrit-PatchSet: 2
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Todd Lipcon 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Fri, 26 Jul 2019 23:29:54 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor

2019-07-26 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13932 )

Change subject: IMPALA-8661 : Add randomized tests to stress 
MetastoreEventsProcessor
..


Patch Set 2:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/13932/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:

http://gerrit.cloudera.org:8080/#/c/13932/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1345
PS2, Line 1345:   // when multiple partitions are added in HMS they are 
all added as one transaction
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/13932/2/fe/src/test/java/org/apache/impala/testutil/ImpalaJdbcClient.java
File fe/src/test/java/org/apache/impala/testutil/ImpalaJdbcClient.java:

http://gerrit.cloudera.org:8080/#/c/13932/2/fe/src/test/java/org/apache/impala/testutil/ImpalaJdbcClient.java@142
PS2, Line 142: return createClient(TestUtils.HIVE_SERVER2_DRIVER_NAME, 
getNoAuthConnectionStr("binary"));
line too long (94 > 90)


http://gerrit.cloudera.org:8080/#/c/13932/2/fe/src/test/java/org/apache/impala/testutil/ImpalaJdbcClient.java@150
PS2, Line 150: return createClient(TestUtils.HIVE_SERVER2_DRIVER_NAME, 
getNoAuthConnectionStr("http"));
line too long (92 > 90)



--
To view, visit http://gerrit.cloudera.org:8080/13932
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce
Gerrit-Change-Number: 13932
Gerrit-PatchSet: 2
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Todd Lipcon 
Gerrit-Comment-Date: Fri, 26 Jul 2019 23:26:24 +
Gerrit-HasComments: Yes


[Impala-ASF-CR] IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor

2019-07-26 Thread Vihang Karajgaonkar (Code Review)
Vihang Karajgaonkar has uploaded a new patch set (#2). ( 
http://gerrit.cloudera.org:8080/13932 )

Change subject: IMPALA-8661 : Add randomized tests to stress 
MetastoreEventsProcessor
..

IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor

This change adds a new stress test for MetastoreEventsProcessor. This
test randomly executes hive queries to generate a lot of events. The
event processor is invoked at random intervals so that a variable batch
of events is processed everytime. After each batch is processed, the
test checks the status of events processor. By default, on CDH builds
the test is configured to run with 8 concurrent Hive clients and each of
the client runs 100 random Hive queries. These defaults can be
overridden by passing system properties using maven command arguments
"-DnumClients" and "-DnumQueriesPerClients". Additionally, the test also
creates impala clients which keep issuing refresh table commands on the
test databases to make sure that eventProcessor is doing some real work
rather than invalidating/refreshing tables which are already incomplete.

This test is added as a junit test and uses Hive JDBC to issue the sqls.
This is much faster than the end-to-end python test which issues each
hive query in a separate beeline session reestablishing the connection
every time.

The test found a bug which is caused when a Hive issues a alter table
add if not exists partition" query and the partition is not really added
since it is preexisting. In such a case the ADD_PARTITION event is still
generated but with a empty list of partitions in the events. Such events
are now ignored.

Notes:
1. Ran the test with defaults. It generates about 2100 events and runs
for close to 15 min. This can be changed to a lower value if we see
significant increased delay in the test job runtimes.
2. On CDP builds the concurrent hive queries run very slow due to the
limited yarn slots on the minicluster. I have left this as a TODO to
investigate. The test runs in single threaded mode with increased number
of queries when running against Hive-3

Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce
---
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M 
fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
A 
fe/src/test/java/org/apache/impala/catalog/events/EventsProcessorStressTest.java
A fe/src/test/java/org/apache/impala/catalog/events/RandomHiveQueryRunner.java
A fe/src/test/java/org/apache/impala/testutil/HiveJdbcClientPool.java
M fe/src/test/java/org/apache/impala/testutil/ImpalaJdbcClient.java
M fe/src/test/java/org/apache/impala/testutil/TestUtils.java
7 files changed, 1,578 insertions(+), 38 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/13932/2
--
To view, visit http://gerrit.cloudera.org:8080/13932
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce
Gerrit-Change-Number: 13932
Gerrit-PatchSet: 2
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Todd Lipcon 


[Impala-ASF-CR] IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor

2019-07-26 Thread Todd Lipcon (Code Review)
Todd Lipcon has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13932 )

Change subject: IMPALA-8661 : Add randomized tests to stress 
MetastoreEventsProcessor
..


Patch Set 2:

I'm out the next two weeks on PTO, so probably best if someone else reviews this


--
To view, visit http://gerrit.cloudera.org:8080/13932
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce
Gerrit-Change-Number: 13932
Gerrit-PatchSet: 2
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Todd Lipcon 
Gerrit-Comment-Date: Fri, 26 Jul 2019 23:26:16 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor

2019-07-26 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13932 )

Change subject: IMPALA-8661 : Add randomized tests to stress 
MetastoreEventsProcessor
..


Patch Set 1:

Build Failed

https://jenkins.impala.io/job/gerrit-code-review-checks/4039/ : Initial code 
review checks failed. See linked job for details on the failure.


--
To view, visit http://gerrit.cloudera.org:8080/13932
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce
Gerrit-Change-Number: 13932
Gerrit-PatchSet: 1
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Todd Lipcon 
Gerrit-Comment-Date: Fri, 26 Jul 2019 22:25:00 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor

2019-07-26 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13932 )

Change subject: IMPALA-8661 : Add randomized tests to stress 
MetastoreEventsProcessor
..


Patch Set 1:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/13932/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:

http://gerrit.cloudera.org:8080/#/c/13932/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1343
PS1, Line 1343:   // when multiple partitions are added in HMS they are 
all added as one transaction
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/13932/1/fe/src/test/java/org/apache/impala/testutil/ImpalaJdbcClient.java
File fe/src/test/java/org/apache/impala/testutil/ImpalaJdbcClient.java:

http://gerrit.cloudera.org:8080/#/c/13932/1/fe/src/test/java/org/apache/impala/testutil/ImpalaJdbcClient.java@142
PS1, Line 142: return createClient(TestUtils.HIVE_SERVER2_DRIVER_NAME, 
getNoAuthConnectionStr("binary"));
line too long (94 > 90)


http://gerrit.cloudera.org:8080/#/c/13932/1/fe/src/test/java/org/apache/impala/testutil/ImpalaJdbcClient.java@150
PS1, Line 150: return createClient(TestUtils.HIVE_SERVER2_DRIVER_NAME, 
getNoAuthConnectionStr("http"));
line too long (92 > 90)



--
To view, visit http://gerrit.cloudera.org:8080/13932
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce
Gerrit-Change-Number: 13932
Gerrit-PatchSet: 1
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Todd Lipcon 
Gerrit-Comment-Date: Fri, 26 Jul 2019 21:46:01 +
Gerrit-HasComments: Yes


[Impala-ASF-CR] IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor

2019-07-26 Thread Vihang Karajgaonkar (Code Review)
Vihang Karajgaonkar has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/13932


Change subject: IMPALA-8661 : Add randomized tests to stress 
MetastoreEventsProcessor
..

IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor

This change adds a new stress test for MetastoreEventsProcessor. This
test randomly executes hive queries to generate a lot of events. The
event processor is invoked at random intervals so that a variable batch
of events is processed everytime. After each batch is processed, the
test checks the status of events processor. By default, on CDH builds
the test is configured to run with 8 concurrent Hive clients and each
of the client runs 100 random Hive queries. These defaults can be
overridden by passing system properties using maven command arguments
"-DnumClients" and "-DnumQueriesPerClients". Additionally, the test
also creates impala clients which keep issuing refresh table commands
on the test databases to make sure that eventProcessor is doing some
real work rather than invalidating/refreshing tables which are
already incomplete.

This test is added as a junit test and uses Hive JDBC to issue the sqls.
This is much faster than the end-to-end python test which issues each
hive query in a separate beeline sessions which re-establishes the
connection every time.

The test already found a bug which is caused when a Hive issues a alter
table add if not exists partition" query and the partition is not really
added since it is preexisting. In such a case the ADD_PARTITION events
is still generated but with a empty list of partitions in the events.
Such events are now ignored.

Notes:
1. Ran the test with defaults. It generates about 2100 events
and runs for close to 15 min. This can be changed to a lower
value if we see significant increased delay in the test job runtimes.
3. On CDP builds the concurrent hive queries run very slow due to
container provisioning time on the minicluster. I have left this as a
TODO to investigate. The test runs in single threaded mode with
increased number of queries when running against Hive-3

Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce
---
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M 
fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
A 
fe/src/test/java/org/apache/impala/catalog/events/EventsProcessorStressTest.java
M 
fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
A fe/src/test/java/org/apache/impala/catalog/events/RandomHiveQueryRunner.java
A fe/src/test/java/org/apache/impala/testutil/HiveJdbcClientPool.java
M fe/src/test/java/org/apache/impala/testutil/ImpalaJdbcClient.java
M fe/src/test/java/org/apache/impala/testutil/TestUtils.java
8 files changed, 1,515 insertions(+), 34 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/13932/1
--
To view, visit http://gerrit.cloudera.org:8080/13932
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce
Gerrit-Change-Number: 13932
Gerrit-PatchSet: 1
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Todd Lipcon