zpinto opened a new pull request, #2610: URL: https://github.com/apache/helix/pull/2610
Change TopStateHandoffReportStage to be an Async Stage as it is the slowest stage in the pipeline. ### Description In some cases TopStateHandoffReportStage is taking up to 1/3 of the total pipeline execution time. In order to speed up the pipeline execution time, we can make TopStateHandoffReportStage async since it is simply computing metrics and reporting without adding to ClusterEvent. No following stages depend on it. ### Tests - [x] Updated BaseStageTest to handle Async stages so it can be used in TestTopStateHandoffMetrics ``` ➜ helix git:(top_state_report_async) ✗ mvn test -o -Dtest=TestTopStateHandoffMetrics -pl=helix-core [INFO] Scanning for projects... [INFO] [INFO] --------------------< org.apache.helix:helix-core >--------------------- [INFO] Building Apache Helix :: Core 1.3.1-SNAPSHOT [INFO] from pom.xml [INFO] -------------------------------[ bundle ]------------------------------- [INFO] [INFO] --- enforcer:1.4.1:enforce (enforce-maven-version) @ helix-core --- [INFO] [INFO] --- enforcer:1.4.1:enforce (enforce-java-version) @ helix-core --- [INFO] [INFO] --- enforcer:1.4.1:enforce (enforce-output-timestamp-property) @ helix-core --- [INFO] [INFO] --- jacoco:0.8.6:prepare-agent (default) @ helix-core --- [INFO] argLine set to -javaagent:/Users/zapinto/.m2/repository/org/jacoco/org.jacoco.agent/0.8.6/org.jacoco.agent-0.8.6-runtime.jar=destfile=/Users/zapinto/Documents/git/zpinto/helix/helix-core/target/jacoco.exec [INFO] [INFO] --- remote-resources:1.7.0:process (process-resource-bundles) @ helix-core --- [INFO] Preparing remote bundle org.apache:apache-jar-resource-bundle:1.4 [INFO] Copying 3 resources from 1 bundle. [INFO] [INFO] --- resources:3.2.0:resources (default-resources) @ helix-core --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Using 'UTF-8' encoding to copy filtered properties files. [INFO] Copying 4 resources [INFO] Copying 0 resource [INFO] Copying 3 resources [INFO] The encoding used to copy filtered properties files have not been set. This means that the same encoding will be used to copy filtered properties files as when copying other filtered resources. This might not be what you want! Run your build with --debug to see which files might be affected. Read more at https://maven.apache.org/plugins/maven-resources-plugin/examples/filtering-properties-files.html [INFO] [INFO] --- compiler:3.10.1:compile (default-compile) @ helix-core --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- bundle:5.1.4:manifest (bundle-manifest) @ helix-core --- [WARNING] Manifest org.apache.helix:helix-core:bundle:1.3.1-SNAPSHOT : Unused Import-Package instructions: [org.apache.logging.log4j*, org.apache.logging.slf4j*] [INFO] Writing manifest: /Users/zapinto/Documents/git/zpinto/helix/helix-core/target/classes/META-INF/MANIFEST.MF [INFO] [INFO] --- resources:3.2.0:testResources (default-testResources) @ helix-core --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Using 'UTF-8' encoding to copy filtered properties files. [INFO] Copying 15 resources [INFO] Copying 3 resources [INFO] [INFO] --- compiler:3.10.1:testCompile (default-testCompile) @ helix-core --- [INFO] Changes detected - recompiling the module! [INFO] Compiling 497 source files to /Users/zapinto/Documents/git/zpinto/helix/helix-core/target/test-classes [INFO] /Users/zapinto/Documents/git/zpinto/helix/helix-core/src/test/java/org/apache/helix/integration/messaging/TestMessageThrottle2.java: Some input files use or override a deprecated API. [INFO] /Users/zapinto/Documents/git/zpinto/helix/helix-core/src/test/java/org/apache/helix/integration/messaging/TestMessageThrottle2.java: Recompile with -Xlint:deprecation for details. [INFO] /Users/zapinto/Documents/git/zpinto/helix/helix-core/src/test/java/org/apache/helix/common/ZkTestBase.java: Some input files use unchecked or unsafe operations. [INFO] /Users/zapinto/Documents/git/zpinto/helix/helix-core/src/test/java/org/apache/helix/common/ZkTestBase.java: Recompile with -Xlint:unchecked for details. [INFO] [INFO] --- surefire:3.0.0-M3:test (default-test) @ helix-core --- [INFO] [INFO] ------------------------------------------------------- [INFO] T E S T S [INFO] ------------------------------------------------------- [INFO] Running org.apache.helix.monitoring.mbeans.TestTopStateHandoffMetrics START TestTopStateHandoffMetrics at Thu Aug 31 00:58:03 PDT 2023 START testTopStateSuccessHandoff at Thu Aug 31 00:58:03 PDT 2023 keySplit:[, testCluster-2fa8181e-d720-445f-b41a-4cfefe2c1178, STATEMODELDEFS] pathSplit:[, testCluster-2fa8181e-d720-445f-b41a-4cfefe2c1178, STATEMODELDEFS] keySplit:[, testCluster-2fa8181e-d720-445f-b41a-4cfefe2c1178, STATEMODELDEFS] pathSplit:[, testCluster-2fa8181e-d720-445f-b41a-4cfefe2c1178, STATEMODELDEFS] keySplit:[, testCluster-2fa8181e-d720-445f-b41a-4cfefe2c1178, STATEMODELDEFS] pathSplit:[, testCluster-2fa8181e-d720-445f-b41a-4cfefe2c1178, STATEMODELDEFS] END testTopStateSuccessHandoff at Thu Aug 31 00:58:05 PDT 2023, took: 2437ms. START testTopStateSuccessHandoff at Thu Aug 31 00:58:05 PDT 2023 keySplit:[, testCluster-acac32c1-147a-4723-9761-7c472494a754, STATEMODELDEFS] pathSplit:[, testCluster-acac32c1-147a-4723-9761-7c472494a754, STATEMODELDEFS] keySplit:[, testCluster-acac32c1-147a-4723-9761-7c472494a754, STATEMODELDEFS] pathSplit:[, testCluster-acac32c1-147a-4723-9761-7c472494a754, STATEMODELDEFS] keySplit:[, testCluster-acac32c1-147a-4723-9761-7c472494a754, STATEMODELDEFS] pathSplit:[, testCluster-acac32c1-147a-4723-9761-7c472494a754, STATEMODELDEFS] END testTopStateSuccessHandoff at Thu Aug 31 00:58:07 PDT 2023, took: 1664ms. START testHandoffDurationWithPendingMessage at Thu Aug 31 00:58:07 PDT 2023 keySplit:[, testCluster-8199af37-840d-4b87-8610-af6483529b5c, STATEMODELDEFS] pathSplit:[, testCluster-8199af37-840d-4b87-8610-af6483529b5c, STATEMODELDEFS] keySplit:[, testCluster-8199af37-840d-4b87-8610-af6483529b5c, STATEMODELDEFS] pathSplit:[, testCluster-8199af37-840d-4b87-8610-af6483529b5c, STATEMODELDEFS] END testHandoffDurationWithPendingMessage at Thu Aug 31 00:58:08 PDT 2023, took: 1517ms. START testHandoffDurationWithPendingMessage at Thu Aug 31 00:58:08 PDT 2023 keySplit:[, testCluster-7cb1a436-6a12-4592-88da-059fd157577d, STATEMODELDEFS] pathSplit:[, testCluster-7cb1a436-6a12-4592-88da-059fd157577d, STATEMODELDEFS] keySplit:[, testCluster-7cb1a436-6a12-4592-88da-059fd157577d, STATEMODELDEFS] pathSplit:[, testCluster-7cb1a436-6a12-4592-88da-059fd157577d, STATEMODELDEFS] END testHandoffDurationWithPendingMessage at Thu Aug 31 00:58:10 PDT 2023, took: 1492ms. START testHandoffDurationWithDefaultStartTime at Thu Aug 31 00:58:10 PDT 2023 keySplit:[, testCluster-d4b9cd3a-64f2-4daa-afd8-0f8e508bbb55, STATEMODELDEFS] pathSplit:[, testCluster-d4b9cd3a-64f2-4daa-afd8-0f8e508bbb55, STATEMODELDEFS] keySplit:[, testCluster-d4b9cd3a-64f2-4daa-afd8-0f8e508bbb55, STATEMODELDEFS] pathSplit:[, testCluster-d4b9cd3a-64f2-4daa-afd8-0f8e508bbb55, STATEMODELDEFS] END testHandoffDurationWithDefaultStartTime at Thu Aug 31 00:58:11 PDT 2023, took: 1491ms. START testHandoffDurationWithDefaultStartTime at Thu Aug 31 00:58:11 PDT 2023 keySplit:[, testCluster-7b36170e-b244-4a14-b083-073e8889ad4c, STATEMODELDEFS] pathSplit:[, testCluster-7b36170e-b244-4a14-b083-073e8889ad4c, STATEMODELDEFS] keySplit:[, testCluster-7b36170e-b244-4a14-b083-073e8889ad4c, STATEMODELDEFS] pathSplit:[, testCluster-7b36170e-b244-4a14-b083-073e8889ad4c, STATEMODELDEFS] END testHandoffDurationWithDefaultStartTime at Thu Aug 31 00:58:13 PDT 2023, took: 1483ms. START testTopStateFailedUnrecoveredHandoff at Thu Aug 31 00:58:13 PDT 2023 keySplit:[, testCluster-624ef679-da14-47b2-bf15-fb64721fb7c8, STATEMODELDEFS] pathSplit:[, testCluster-624ef679-da14-47b2-bf15-fb64721fb7c8, STATEMODELDEFS] keySplit:[, testCluster-624ef679-da14-47b2-bf15-fb64721fb7c8, STATEMODELDEFS] pathSplit:[, testCluster-624ef679-da14-47b2-bf15-fb64721fb7c8, STATEMODELDEFS] keySplit:[, testCluster-624ef679-da14-47b2-bf15-fb64721fb7c8, STATEMODELDEFS] pathSplit:[, testCluster-624ef679-da14-47b2-bf15-fb64721fb7c8, STATEMODELDEFS] END testTopStateFailedUnrecoveredHandoff at Thu Aug 31 00:58:14 PDT 2023, took: 1705ms. START testTopStateSuccessfulYetNonGracefulHandoff at Thu Aug 31 00:58:14 PDT 2023 keySplit:[, testCluster-f92f60b1-bcc0-4ee1-9b65-ce2dc5b7a02c, STATEMODELDEFS] pathSplit:[, testCluster-f92f60b1-bcc0-4ee1-9b65-ce2dc5b7a02c, STATEMODELDEFS] keySplit:[, testCluster-f92f60b1-bcc0-4ee1-9b65-ce2dc5b7a02c, STATEMODELDEFS] pathSplit:[, testCluster-f92f60b1-bcc0-4ee1-9b65-ce2dc5b7a02c, STATEMODELDEFS] keySplit:[, testCluster-f92f60b1-bcc0-4ee1-9b65-ce2dc5b7a02c, STATEMODELDEFS] pathSplit:[, testCluster-f92f60b1-bcc0-4ee1-9b65-ce2dc5b7a02c, STATEMODELDEFS] END testTopStateSuccessfulYetNonGracefulHandoff at Thu Aug 31 00:58:16 PDT 2023, took: 1598ms. START testFastTopStateHandoffWithNoMissingTopStateAndOldInstanceCrash at Thu Aug 31 00:58:16 PDT 2023 keySplit:[, testCluster-a24b6e09-533f-488d-9562-58df5b895e30, STATEMODELDEFS] pathSplit:[, testCluster-a24b6e09-533f-488d-9562-58df5b895e30, STATEMODELDEFS] keySplit:[, testCluster-a24b6e09-533f-488d-9562-58df5b895e30, STATEMODELDEFS] pathSplit:[, testCluster-a24b6e09-533f-488d-9562-58df5b895e30, STATEMODELDEFS] END testFastTopStateHandoffWithNoMissingTopStateAndOldInstanceCrash at Thu Aug 31 00:58:18 PDT 2023, took: 1484ms. START testFastTopStateHandoffWithNoMissingTopState at Thu Aug 31 00:58:18 PDT 2023 keySplit:[, testCluster-5f45495e-b483-40c9-8b3e-69790d55ca0b, STATEMODELDEFS] pathSplit:[, testCluster-5f45495e-b483-40c9-8b3e-69790d55ca0b, STATEMODELDEFS] keySplit:[, testCluster-5f45495e-b483-40c9-8b3e-69790d55ca0b, STATEMODELDEFS] pathSplit:[, testCluster-5f45495e-b483-40c9-8b3e-69790d55ca0b, STATEMODELDEFS] END testFastTopStateHandoffWithNoMissingTopState at Thu Aug 31 00:58:19 PDT 2023, took: 1491ms. START testTopStateFailedHandoff at Thu Aug 31 00:58:19 PDT 2023 keySplit:[, testCluster-27693b68-9fe9-4479-8274-f01f2be8263b, STATEMODELDEFS] pathSplit:[, testCluster-27693b68-9fe9-4479-8274-f01f2be8263b, STATEMODELDEFS] keySplit:[, testCluster-27693b68-9fe9-4479-8274-f01f2be8263b, STATEMODELDEFS] pathSplit:[, testCluster-27693b68-9fe9-4479-8274-f01f2be8263b, STATEMODELDEFS] keySplit:[, testCluster-27693b68-9fe9-4479-8274-f01f2be8263b, STATEMODELDEFS] pathSplit:[, testCluster-27693b68-9fe9-4479-8274-f01f2be8263b, STATEMODELDEFS] END testTopStateFailedHandoff at Thu Aug 31 00:58:21 PDT 2023, took: 1721ms. START testTopStateFailedHandoff at Thu Aug 31 00:58:21 PDT 2023 keySplit:[, testCluster-86014c83-701d-45f7-9686-13cc7642780d, STATEMODELDEFS] pathSplit:[, testCluster-86014c83-701d-45f7-9686-13cc7642780d, STATEMODELDEFS] keySplit:[, testCluster-86014c83-701d-45f7-9686-13cc7642780d, STATEMODELDEFS] pathSplit:[, testCluster-86014c83-701d-45f7-9686-13cc7642780d, STATEMODELDEFS] keySplit:[, testCluster-86014c83-701d-45f7-9686-13cc7642780d, STATEMODELDEFS] pathSplit:[, testCluster-86014c83-701d-45f7-9686-13cc7642780d, STATEMODELDEFS] END testTopStateFailedHandoff at Thu Aug 31 00:58:22 PDT 2023, took: 1691ms. END TestTopStateHandoffMetrics at Thu Aug 31 00:58:22 PDT 2023 [INFO] Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.791 s - in org.apache.helix.monitoring.mbeans.TestTopStateHandoffMetrics [INFO] [INFO] Results: [INFO] [INFO] Tests run: 12, Failures: 0, Errors: 0, Skipped: 0 [INFO] [INFO] [INFO] --- jacoco:0.8.6:report (generate-code-coverage-report) @ helix-core --- [INFO] Loading execution data file /Users/zapinto/Documents/git/zpinto/helix/helix-core/target/jacoco.exec [INFO] Analyzed bundle 'Apache Helix :: Core' with 943 classes [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 36.588 s [INFO] Finished at: 2023-08-31T00:58:26-07:00 [INFO] ------------------------------------------------------------------------ ``` ### Changes that Break Backward Compatibility (Optional) NA ### Commits - My commits all reference appropriate Apache Helix GitHub issues in their subject lines. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 1. Subject is limited to 50 characters (not including Jira issue reference) 1. Subject does not end with a period 1. Subject uses the imperative mood ("add", not "adding") 1. Body wraps at 72 characters 1. Body explains "what" and "why", not "how" ### Code Quality - My diff has been formatted using helix-style.xml (helix-style-intellij.xml if IntelliJ IDE is used) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
