Vihang Karajgaonkar has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/13932 )
Change subject: IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor ...................................................................... IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor This change adds a new stress test for MetastoreEventsProcessor. This test randomly executes hive queries to generate a lot of events. The event processor is invoked at random intervals so that a variable batch of events is processed everytime. After each batch is processed, the test checks the status of events processor. By default, on CDH builds the test is configured to run with 8 concurrent Hive clients and each of the client runs 100 random Hive queries. These defaults can be overridden by passing system properties using maven command arguments "-DnumClients" and "-DnumQueriesPerClients". Additionally, the test also creates impala clients which keep issuing refresh table commands on the test databases to make sure that eventProcessor is doing some real work rather than invalidating/refreshing tables which are already incomplete. This test is added as a junit test and uses Hive JDBC to issue the sqls. This is much faster than the end-to-end python test which issues each hive query in a separate beeline sessions which re-establishes the connection every time. The test already found a bug which is caused when a Hive issues a alter table add if not exists partition" query and the partition is not really added since it is preexisting. In such a case the ADD_PARTITION events is still generated but with a empty list of partitions in the events. Such events are now ignored. Notes: 1. Ran the test with defaults. It generates about 2100 events and runs for close to 15 min. This can be changed to a lower value if we see significant increased delay in the test job runtimes. 3. On CDP builds the concurrent hive queries run very slow due to container provisioning time on the minicluster. I have left this as a TODO to investigate. The test runs in single threaded mode with increased number of queries when running against Hive-3 Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce --- M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java A fe/src/test/java/org/apache/impala/catalog/events/EventsProcessorStressTest.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java A fe/src/test/java/org/apache/impala/catalog/events/RandomHiveQueryRunner.java A fe/src/test/java/org/apache/impala/testutil/HiveJdbcClientPool.java M fe/src/test/java/org/apache/impala/testutil/ImpalaJdbcClient.java M fe/src/test/java/org/apache/impala/testutil/TestUtils.java 8 files changed, 1,516 insertions(+), 34 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/13932/3 -- To view, visit http://gerrit.cloudera.org:8080/13932 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce Gerrit-Change-Number: 13932 Gerrit-PatchSet: 3 Gerrit-Owner: Vihang Karajgaonkar <vih...@cloudera.com> Gerrit-Reviewer: Anurag Mantripragada <anu...@cloudera.com> Gerrit-Reviewer: Bharath Vissapragada <bhara...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Reviewer: Vihang Karajgaonkar <vih...@cloudera.com>
