See
<https://ci-beam.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow_V2_Java11/292/display/redirect?page=changes>
Changes:
[Etienne Chauchot] [BEAM-12591] Put Spark Structured Streaming runner sources
back to main
[Etienne Chauchot] [BEAM-12629] As spark DataSourceV2 is only available for
spark 2,
[Etienne Chauchot] [BEAM-12627] Deal with spark Encoders braking change between
spark 2 and
[Etienne Chauchot] [BEAM-12591] move SchemaHelpers to correct package
[Etienne Chauchot] [BEAM-8470] Disable wait for termination in a streaming
pipeline because
[Etienne Chauchot] [BEAM-12630] Deal with breaking change in streaming
pipelines start by
[Etienne Chauchot] [BEAM-12629] Make source tests spark version agnostic and
move them back
[Etienne Chauchot] [BEAM-12629] Make a spark 3 source impl
[Etienne Chauchot] [BEAM-12591] Fix checkstyle and spotless
[Etienne Chauchot] [BEAM-12629] Reduce serializable to only needed classes and
Fix schema
[Etienne Chauchot] [BEAM-12591] Add checkstyle exceptions for version specific
classes
[Etienne Chauchot] [BEAM-12629] Fix sources javadocs and improve impl
[Etienne Chauchot] [BEAM-12591] Add spark 3 to structured streaming validates
runner tests
[noreply] [BEAM-6516] Fixes race condition in RabbitMqIO causing duplicate acks
------------------------------------------
[...truncated 2.93 MB...]
Aug 05, 2021 1:23:39 PM org.apache.beam.sdk.testutils.metrics.MetricsReader
getCounterMetric
SEVERE: Failed to get metric result.bytes, from namespace Query2.Results
2021-08-05T13:23:39.005Z 0 samples not enough to calculate steady-state event
rate
==========================================================================================
Run started 2021-08-05T12:16:25.553Z and ran for PT4033.452S
Default configuration:
{"debug":true,"query":null,"sourceType":"DIRECT","generateEventFilePathPrefix":null,"sinkType":"DEVNULL","exportSummaryToBigQuery":false,"pubSubMode":"COMBINED","pubsubMessageSerializationMethod":"CODER","sideInputType":"DIRECT","sideInputRowCount":500,"sideInputNumShards":3,"sideInputUrl":null,"sessionGap":600000,"numEvents":100000,"numEventGenerators":100,"rateShape":"SINE","firstEventRate":10000,"nextEventRate":10000,"rateUnit":"PER_SECOND","ratePeriodSec":600,"preloadSeconds":0,"streamTimeout":240,"isRateLimited":false,"useWallclockEventTime":false,"avgPersonByteSize":200,"avgAuctionByteSize":500,"avgBidByteSize":100,"hotAuctionRatio":2,"hotSellersRatio":4,"hotBiddersRatio":4,"windowSizeSec":10,"windowPeriodSec":5,"watermarkHoldbackSec":0,"numInFlightAuctions":100,"numActivePeople":1000,"coderStrategy":"HAND","cpuDelayMs":0,"diskBusyBytes":0,"auctionSkip":123,"fanout":5,"maxAuctionsWaitingTime":600,"occasionalDelaySec":3,"probDelayedEvent":0.1,"maxLogEvents":100000,"usePubsubPublishTime":false,"outOfOrderGroupSize":1,"numKeyBuckets":20000,"pardoCPUFactor":1.0}
Configurations:
Conf Description
0000 query:PASSTHROUGH; numEvents:10000000
0001 query:CURRENCY_CONVERSION; numEvents:10000000
0002 query:SELECTION; numEvents:10000000
0003 query:LOCAL_ITEM_SUGGESTION; numEvents:10000000
0004 query:AVERAGE_PRICE_FOR_CATEGORY; numEvents:1000000
0005 query:HOT_ITEMS; numEvents:10000000
0006 query:AVERAGE_SELLING_PRICE_BY_SELLER; numEvents:1000000
0007 query:HIGHEST_BID; numEvents:10000000
0008 query:MONITOR_NEW_USERS; numEvents:10000000
0009 query:WINNING_BIDS; numEvents:1000000
0010 query:LOG_TO_SHARDED_FILES; numEvents:10000000
0011 query:USER_SESSIONS; numEvents:10000000
0012 query:PROCESSING_TIME_WINDOWS; numEvents:10000000
0013 query:PORTABILITY_BATCH; numEvents:10000000
0014 query:BOUNDED_SIDE_INPUT_JOIN; numEvents:10000000
0015 query:SESSION_SIDE_INPUT_JOIN; numEvents:10000000
Performance:
Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results
(Baseline)
0000 -1.0 -1.0 -1
*** Cancelling streaming job since it appeared stuck for 60 min. ***
0001 -1.0 -1.0 -1
*** Cancelling streaming job since it appeared stuck for 60 min. ***
0002 -1.0 -1.0 -1
*** Cancelling streaming job since it appeared stuck for 60 min. ***
0003 -1.0 -1.0 -1
*** Cancelling streaming job since it appeared stuck for 60 min. ***
0004 *** not run ***
0005 -1.0 -1.0 -1
*** Cancelling streaming job since it appeared stuck for 60 min. ***
0006 *** not run ***
0007 -1.0 -1.0 -1
*** Cancelling streaming job since it appeared stuck for 60 min. ***
0008 -1.0 -1.0 -1
*** Cancelling streaming job since it appeared stuck for 60 min. ***
0009 *** not run ***
0010 -1.0 -1.0 -1
*** Cancelling streaming job since it appeared stuck for 60 min. ***
0011 *** not run ***
0012 -1.0 -1.0 -1
*** Cancelling streaming job since it appeared stuck for 60 min. ***
0013 *** not run ***
0014 *** not run ***
0015 *** not run ***
==========================================================================================
2021-08-05T13:23:39.559Z CANCELLED Query11 (waiting for shutdown)
Aug 05, 2021 1:23:39 PM org.apache.beam.sdk.testutils.metrics.MetricsReader
getCounterMetric
SEVERE: Failed to get metric event.elements, from namespace Query11.Events
Aug 05, 2021 1:23:39 PM org.apache.beam.sdk.testutils.metrics.MetricsReader
getCounterMetric
SEVERE: Failed to get metric event.bytes, from namespace Query11.Events
Aug 05, 2021 1:23:39 PM org.apache.beam.sdk.testutils.metrics.MetricsReader
getCounterMetric
SEVERE: Failed to get metric result.elements, from namespace Query11.Results
Aug 05, 2021 1:23:39 PM org.apache.beam.sdk.testutils.metrics.MetricsReader
getCounterMetric
SEVERE: Failed to get metric result.bytes, from namespace Query11.Results
2021-08-05T13:23:39.575Z 0 samples not enough to calculate steady-state event
rate
==========================================================================================
Run started 2021-08-05T12:16:25.553Z and ran for PT4034.022S
Default configuration:
{"debug":true,"query":null,"sourceType":"DIRECT","generateEventFilePathPrefix":null,"sinkType":"DEVNULL","exportSummaryToBigQuery":false,"pubSubMode":"COMBINED","pubsubMessageSerializationMethod":"CODER","sideInputType":"DIRECT","sideInputRowCount":500,"sideInputNumShards":3,"sideInputUrl":null,"sessionGap":600000,"numEvents":100000,"numEventGenerators":100,"rateShape":"SINE","firstEventRate":10000,"nextEventRate":10000,"rateUnit":"PER_SECOND","ratePeriodSec":600,"preloadSeconds":0,"streamTimeout":240,"isRateLimited":false,"useWallclockEventTime":false,"avgPersonByteSize":200,"avgAuctionByteSize":500,"avgBidByteSize":100,"hotAuctionRatio":2,"hotSellersRatio":4,"hotBiddersRatio":4,"windowSizeSec":10,"windowPeriodSec":5,"watermarkHoldbackSec":0,"numInFlightAuctions":100,"numActivePeople":1000,"coderStrategy":"HAND","cpuDelayMs":0,"diskBusyBytes":0,"auctionSkip":123,"fanout":5,"maxAuctionsWaitingTime":600,"occasionalDelaySec":3,"probDelayedEvent":0.1,"maxLogEvents":100000,"usePubsubPublishTime":false,"outOfOrderGroupSize":1,"numKeyBuckets":20000,"pardoCPUFactor":1.0}
Configurations:
Conf Description
0000 query:PASSTHROUGH; numEvents:10000000
0001 query:CURRENCY_CONVERSION; numEvents:10000000
0002 query:SELECTION; numEvents:10000000
0003 query:LOCAL_ITEM_SUGGESTION; numEvents:10000000
0004 query:AVERAGE_PRICE_FOR_CATEGORY; numEvents:1000000
0005 query:HOT_ITEMS; numEvents:10000000
0006 query:AVERAGE_SELLING_PRICE_BY_SELLER; numEvents:1000000
0007 query:HIGHEST_BID; numEvents:10000000
0008 query:MONITOR_NEW_USERS; numEvents:10000000
0009 query:WINNING_BIDS; numEvents:1000000
0010 query:LOG_TO_SHARDED_FILES; numEvents:10000000
0011 query:USER_SESSIONS; numEvents:10000000
0012 query:PROCESSING_TIME_WINDOWS; numEvents:10000000
0013 query:PORTABILITY_BATCH; numEvents:10000000
0014 query:BOUNDED_SIDE_INPUT_JOIN; numEvents:10000000
0015 query:SESSION_SIDE_INPUT_JOIN; numEvents:10000000
Performance:
Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results
(Baseline)
0000 -1.0 -1.0 -1
*** Cancelling streaming job since it appeared stuck for 60 min. ***
0001 -1.0 -1.0 -1
*** Cancelling streaming job since it appeared stuck for 60 min. ***
0002 -1.0 -1.0 -1
*** Cancelling streaming job since it appeared stuck for 60 min. ***
0003 -1.0 -1.0 -1
*** Cancelling streaming job since it appeared stuck for 60 min. ***
0004 *** not run ***
0005 -1.0 -1.0 -1
*** Cancelling streaming job since it appeared stuck for 60 min. ***
0006 *** not run ***
0007 -1.0 -1.0 -1
*** Cancelling streaming job since it appeared stuck for 60 min. ***
0008 -1.0 -1.0 -1
*** Cancelling streaming job since it appeared stuck for 60 min. ***
0009 *** not run ***
0010 -1.0 -1.0 -1
*** Cancelling streaming job since it appeared stuck for 60 min. ***
0011 -1.0 -1.0 -1
*** Cancelling streaming job since it appeared stuck for 60 min. ***
0012 -1.0 -1.0 -1
*** Cancelling streaming job since it appeared stuck for 60 min. ***
0013 *** not run ***
0014 *** not run ***
0015 *** not run ***
==========================================================================================
==========================================================================================
Run started 2021-08-05T12:16:25.553Z and ran for PT4034.226S
Default configuration:
{"debug":true,"query":null,"sourceType":"DIRECT","generateEventFilePathPrefix":null,"sinkType":"DEVNULL","exportSummaryToBigQuery":false,"pubSubMode":"COMBINED","pubsubMessageSerializationMethod":"CODER","sideInputType":"DIRECT","sideInputRowCount":500,"sideInputNumShards":3,"sideInputUrl":null,"sessionGap":600000,"numEvents":100000,"numEventGenerators":100,"rateShape":"SINE","firstEventRate":10000,"nextEventRate":10000,"rateUnit":"PER_SECOND","ratePeriodSec":600,"preloadSeconds":0,"streamTimeout":240,"isRateLimited":false,"useWallclockEventTime":false,"avgPersonByteSize":200,"avgAuctionByteSize":500,"avgBidByteSize":100,"hotAuctionRatio":2,"hotSellersRatio":4,"hotBiddersRatio":4,"windowSizeSec":10,"windowPeriodSec":5,"watermarkHoldbackSec":0,"numInFlightAuctions":100,"numActivePeople":1000,"coderStrategy":"HAND","cpuDelayMs":0,"diskBusyBytes":0,"auctionSkip":123,"fanout":5,"maxAuctionsWaitingTime":600,"occasionalDelaySec":3,"probDelayedEvent":0.1,"maxLogEvents":100000,"usePubsubPublishTime":false,"outOfOrderGroupSize":1,"numKeyBuckets":20000,"pardoCPUFactor":1.0}
Configurations:
Conf Description
0000 query:PASSTHROUGH; numEvents:10000000
0001 query:CURRENCY_CONVERSION; numEvents:10000000
0002 query:SELECTION; numEvents:10000000
0003 query:LOCAL_ITEM_SUGGESTION; numEvents:10000000
0004 query:AVERAGE_PRICE_FOR_CATEGORY; numEvents:1000000
0005 query:HOT_ITEMS; numEvents:10000000
0006 query:AVERAGE_SELLING_PRICE_BY_SELLER; numEvents:1000000
0007 query:HIGHEST_BID; numEvents:10000000
0008 query:MONITOR_NEW_USERS; numEvents:10000000
0009 query:WINNING_BIDS; numEvents:1000000
0010 query:LOG_TO_SHARDED_FILES; numEvents:10000000
0011 query:USER_SESSIONS; numEvents:10000000
0012 query:PROCESSING_TIME_WINDOWS; numEvents:10000000
0013 query:PORTABILITY_BATCH; numEvents:10000000
0014 query:BOUNDED_SIDE_INPUT_JOIN; numEvents:10000000
0015 query:SESSION_SIDE_INPUT_JOIN; numEvents:10000000
Performance:
Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results
(Baseline)
0000 -1.0 -1.0 -1
*** Cancelling streaming job since it appeared stuck for 60 min. ***
0001 -1.0 -1.0 -1
*** Cancelling streaming job since it appeared stuck for 60 min. ***
0002 -1.0 -1.0 -1
*** Cancelling streaming job since it appeared stuck for 60 min. ***
0003 -1.0 -1.0 -1
*** Cancelling streaming job since it appeared stuck for 60 min. ***
0004 *** not run ***
0005 -1.0 -1.0 -1
*** Cancelling streaming job since it appeared stuck for 60 min. ***
0006 *** not run ***
0007 -1.0 -1.0 -1
*** Cancelling streaming job since it appeared stuck for 60 min. ***
0008 -1.0 -1.0 -1
*** Cancelling streaming job since it appeared stuck for 60 min. ***
0009 *** not run ***
0010 -1.0 -1.0 -1
*** Cancelling streaming job since it appeared stuck for 60 min. ***
0011 -1.0 -1.0 -1
*** Cancelling streaming job since it appeared stuck for 60 min. ***
0012 -1.0 -1.0 -1
*** Cancelling streaming job since it appeared stuck for 60 min. ***
0013 *** not run ***
0014 *** not run ***
0015 *** not run ***
==========================================================================================
Exception in thread "main" java.lang.RuntimeException: Execution was not
successful
at org.apache.beam.sdk.nexmark.Main.runAll(Main.java:186)
at org.apache.beam.sdk.nexmark.Main.main(Main.java:495)
> Task :sdks:java:testing:nexmark:run FAILED
> Task :runners:google-cloud-dataflow-java:cleanUpDockerImages
Untagged: us.gcr.io/apache-beam-testing/java-postcommit-it/java:20210805121548
Untagged:
us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:9d6bdde69ed5e2fd48d4f0c51abfa270a858fe9e07bd7ca4f0840cef2336f6be
Deleted: sha256:98c002e80d368121039a061198281df1f849eac5e294484e07d30481da8d3461
Deleted: sha256:4ee759bf9d4c0204b67b395bf809c2d5a5582a427961081635bed5e2afb477ce
Deleted: sha256:b9d63bcaf7379509326d0817f2690b43cda253555be95d9cc54cf8fc903c98c2
Deleted: sha256:5c6202739936c3c6d36e1ed563026420926601e15c3d9e8666034b727e629cf5
Deleted: sha256:cbcf33f62465aa43db4fa995cc3a45daa2b11f5d018c9022a4da84ccc27f7bed
Deleted: sha256:33e874304e98d013ca11024b5bfe4b873a78d88460e8417d03eb41eada0da139
Deleted: sha256:9da2569e17c1b73571258702ca84f961721e7e027987493c0335b65fc2cfada2
Deleted: sha256:ac03e0ea7ffc2e452ece88475429497881d8f2e8c63b498183e7254ffc8378fd
Deleted: sha256:3c4826585b26914d4b820d539560baacbc95c1516c73aa8f60fc71b7dc30008a
Deleted: sha256:c9643f1acb3765a16872eaa7fa0199181f442dd2ba9a61e60c59baf46e30f6bd
Deleted: sha256:228be821cf6c25bdbecb7aa5aab3117b5bcc2a1ab7516b9298bf9302fdf9acdf
Deleted: sha256:baa464827def8591c986799ff4bc22f2b1536a8d521dfec66bfd3802b45726d3
Deleted: sha256:6eeb38c2a651ceb19baf757d4af4d1ee43a0db79a16225708f7962bb3cace338
Deleted: sha256:209d5ccdc916890836f1ec7a45e45d5df2670660e41638c61728164811f154bc
Deleted: sha256:f0474ea45723af1e533a888ca4dbd438a128167039eafa90da768c20637a6cae
Deleted: sha256:651e07aef829cb3f094fb32ab997ed6ce8fbbbd5d78e79c9cd55a1a96a143322
Deleted: sha256:02d1f1725265a35cbfdacd7a8f1c13806a280b5aad1018e51a813f239e87ef4e
Deleted: sha256:e6d5cbde83da2a14db079228fcc8a2c022951c436f24fd1afa430fbf2ef80ff5
Deleted: sha256:0d0924fd189163d1821b9972b030da54b44d6e7394d8a3109c8a10083a54d42e
Deleted: sha256:bded3026f1d09239ebf8c4b7ee7bea66885d552f1e5b196a4c974cfe2b962c8b
Deleted: sha256:72fa10c6530e4327da8782a433cb7d710604f3b2ac3e5b94b9e14ac1dbd96685
Deleted: sha256:4745c017c0d72fcb0a7cef1c31d2191b681216fd31fc79c3112f141ed10f11ac
Deleted: sha256:3fb83bf50aa14d56c74f1f199fcc952d42dd0c99f58d7f62301317ffcb21ba7a
Deleted: sha256:ab857ed106bc07753907d626510c53fed68655114c34ea1e648faac24ae64c58
Deleted: sha256:89b02deeb3b348a84c413d6902ac6ad9fb62a7738f13bf4a290cd4a760d04d22
Deleted: sha256:df2e8f1b87ad3b000c53148ef1b343abb244e41210326d94437ae7db02f86af3
Deleted: sha256:4308b8e416c73c80ff3458791ce221691057eb87cc1e77f4923fc12a6954ea66
Deleted: sha256:78ba36c6a7864a46c83992a32b34f3a8b8b628d7a1b95d93e859f1f80f58c110
Tag: [us.gcr.io/apache-beam-testing/java-postcommit-it/java:20210805121548]
- referencing digest:
[us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:9d6bdde69ed5e2fd48d4f0c51abfa270a858fe9e07bd7ca4f0840cef2336f6be]
Deleted [[us.gcr.io/apache-beam-testing/java-postcommit-it/java:20210805121548]
(referencing
[us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:9d6bdde69ed5e2fd48d4f0c51abfa270a858fe9e07bd7ca4f0840cef2336f6be])].
Removing untagged image
us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:9d6bdde69ed5e2fd48d4f0c51abfa270a858fe9e07bd7ca4f0840cef2336f6be
Digests:
-
us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:9d6bdde69ed5e2fd48d4f0c51abfa270a858fe9e07bd7ca4f0840cef2336f6be
Deleted
[us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:9d6bdde69ed5e2fd48d4f0c51abfa270a858fe9e07bd7ca4f0840cef2336f6be].
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:java:testing:nexmark:run'.
> Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with
> non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See
https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 1h 7m 56s
129 actionable tasks: 15 executed, 114 up-to-date
Publishing build scan...
https://gradle.com/s/bnowkpbxquero
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
