See
<https://ci-beam.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/6218/display/redirect?page=changes>
Changes:
[Robin Qiu] Propagate BigQuery streaming insert throttled time to Dataflow
worker in
[Kamil Wasilewski] [BEAM-10524] JSON decoder for ReadFromBigQuery supports
repeatable
[noreply] Merge pull request #12415 from [BEAM-10603] Add the RecordingManager
and
------------------------------------------
[...truncated 970.38 KB...]
2020-08-25T00:12:53.100Z 3 samples not enough to calculate steady-state event
rate
Aug 25, 2020 12:12:53 AM org.apache.beam.sdk.testutils.metrics.MetricsReader
getCounterMetric
SEVERE: Failed to get metric fatal, from namespace BoundedSideInputJoin
2020-08-25T00:12:53.100Z no activity
Aug 25, 2020 12:12:53 AM org.apache.beam.sdk.testutils.metrics.MetricsReader
getCounterMetric
SEVERE: Failed to get metric fatal, from namespace Query11
2020-08-25T00:12:53.161Z new perf
{"snapshots":null,"runtimeSec":207.625,"numEvents":10000000,"eventsPerSec":48163.7567730283,"eventBytesPerSec":6182130.750150512,"numResults":1839657,"resultsPerSec":8860.479229379891,"resultBytesPerSec":141767.66767007826,"startupDelaySec":-149.523,"processingDelaySec":31.617,"shutdownDelaySec":21.377,"timeDilation":4.816134858518964,"errors":null,"jobId":null}
Aug 25, 2020 12:12:53 AM org.apache.beam.sdk.testutils.metrics.MetricsReader
getCounterMetric
SEVERE: Failed to get metric fatal, from namespace Query5
2020-08-25T00:12:53.166Z no activity
2020-08-25T00:13:08.172Z DONE BoundedSideInputJoin
2020-08-25T00:13:08.204Z DONE Query5
2020-08-25T00:13:08.218Z 3 samples not enough to calculate steady-state event
rate
Aug 25, 2020 12:13:08 AM org.apache.beam.sdk.testutils.metrics.MetricsReader
getCounterMetric
SEVERE: Failed to get metric fatal, from namespace BoundedSideInputJoin
2020-08-25T00:13:08.219Z streaming query appears to have finished waiting for
completion.
==========================================================================================
Run started 2020-08-25T00:07:08.545Z and ran for PT359.674S
Default configuration:
{"debug":true,"query":null,"sourceType":"DIRECT","sinkType":"DEVNULL","exportSummaryToBigQuery":false,"pubSubMode":"COMBINED","pubsubMessageSerializationMethod":"CODER","sideInputType":"DIRECT","sideInputRowCount":500,"sideInputNumShards":3,"sideInputUrl":null,"sessionGap":{"standardDays":0,"standardHours":0,"standardMinutes":10,"standardSeconds":600,"millis":600000},"numEvents":100000,"numEventGenerators":100,"rateShape":"SINE","firstEventRate":10000,"nextEventRate":10000,"rateUnit":"PER_SECOND","ratePeriodSec":600,"preloadSeconds":0,"streamTimeout":240,"isRateLimited":false,"useWallclockEventTime":false,"avgPersonByteSize":200,"avgAuctionByteSize":500,"avgBidByteSize":100,"hotAuctionRatio":2,"hotSellersRatio":4,"hotBiddersRatio":4,"windowSizeSec":10,"windowPeriodSec":5,"watermarkHoldbackSec":0,"numInFlightAuctions":100,"numActivePeople":1000,"coderStrategy":"HAND","cpuDelayMs":0,"diskBusyBytes":0,"auctionSkip":123,"fanout":5,"maxAuctionsWaitingTime":600,"occasionalDelaySec":3,"probDelayedEvent":0.1,"maxLogEvents":100000,"usePubsubPublishTime":false,"outOfOrderGroupSize":1}
Configurations:
Conf Description
0000 query:PASSTHROUGH; exportSummaryToBigQuery:true; numEvents:10000000
0001 query:CURRENCY_CONVERSION; exportSummaryToBigQuery:true;
numEvents:10000000
0002 query:SELECTION; exportSummaryToBigQuery:true; numEvents:10000000
0003 query:LOCAL_ITEM_SUGGESTION; exportSummaryToBigQuery:true;
numEvents:10000000
0004 query:AVERAGE_PRICE_FOR_CATEGORY; exportSummaryToBigQuery:true;
numEvents:1000000
0005 query:HOT_ITEMS; exportSummaryToBigQuery:true; numEvents:10000000
0006 query:AVERAGE_SELLING_PRICE_BY_SELLER; exportSummaryToBigQuery:true;
numEvents:1000000
0007 query:HIGHEST_BID; exportSummaryToBigQuery:true; numEvents:10000000
0008 query:MONITOR_NEW_USERS; exportSummaryToBigQuery:true;
numEvents:10000000
0009 query:WINNING_BIDS; exportSummaryToBigQuery:true; numEvents:1000000
0010 query:LOG_TO_SHARDED_FILES; exportSummaryToBigQuery:true;
numEvents:10000000
0011 query:USER_SESSIONS; exportSummaryToBigQuery:true; numEvents:10000000
0012 query:PROCESSING_TIME_WINDOWS; exportSummaryToBigQuery:true;
numEvents:10000000
0013 query:BOUNDED_SIDE_INPUT_JOIN; exportSummaryToBigQuery:true;
numEvents:10000000
0014 query:SESSION_SIDE_INPUT_JOIN; exportSummaryToBigQuery:true;
numEvents:10000000
Performance:
Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results
(Baseline)
0000 23.6 424142.2 10000000
0001 19.1 522821.1 9200000
0002 12.8 781250.0 83527
0003 42.6 234780.4 59435
0004 60.2 16611.6 140
0005 *** not run ***
0006 35.6 28056.0 14085
0007 -1.0 -1.0 -1
*** Job was unexpectedly updated ***
0008 52.2 191581.9 92833
0009 42.7 23392.4 32548
0010 96.0 104189.5 100
0011 *** not run ***
0012 106.4 93947.9 609995
0013 41.3 242001.8 9200000
0014 150.1 66638.7 9200000
==========================================================================================
2020-08-25T00:13:08.232Z RUNNING Query11
2020-08-25T00:13:08.262Z 3 samples not enough to calculate steady-state event
rate
Aug 25, 2020 12:13:08 AM org.apache.beam.sdk.testutils.metrics.MetricsReader
getCounterMetric
SEVERE: Failed to get metric fatal, from namespace Query5
2020-08-25T00:13:08.263Z streaming query appears to have finished waiting for
completion.
==========================================================================================
Run started 2020-08-25T00:07:08.545Z and ran for PT359.718S
Default configuration:
{"debug":true,"query":null,"sourceType":"DIRECT","sinkType":"DEVNULL","exportSummaryToBigQuery":false,"pubSubMode":"COMBINED","pubsubMessageSerializationMethod":"CODER","sideInputType":"DIRECT","sideInputRowCount":500,"sideInputNumShards":3,"sideInputUrl":null,"sessionGap":{"standardDays":0,"standardHours":0,"standardMinutes":10,"standardSeconds":600,"millis":600000},"numEvents":100000,"numEventGenerators":100,"rateShape":"SINE","firstEventRate":10000,"nextEventRate":10000,"rateUnit":"PER_SECOND","ratePeriodSec":600,"preloadSeconds":0,"streamTimeout":240,"isRateLimited":false,"useWallclockEventTime":false,"avgPersonByteSize":200,"avgAuctionByteSize":500,"avgBidByteSize":100,"hotAuctionRatio":2,"hotSellersRatio":4,"hotBiddersRatio":4,"windowSizeSec":10,"windowPeriodSec":5,"watermarkHoldbackSec":0,"numInFlightAuctions":100,"numActivePeople":1000,"coderStrategy":"HAND","cpuDelayMs":0,"diskBusyBytes":0,"auctionSkip":123,"fanout":5,"maxAuctionsWaitingTime":600,"occasionalDelaySec":3,"probDelayedEvent":0.1,"maxLogEvents":100000,"usePubsubPublishTime":false,"outOfOrderGroupSize":1}
Configurations:
Conf Description
0000 query:PASSTHROUGH; exportSummaryToBigQuery:true; numEvents:10000000
0001 query:CURRENCY_CONVERSION; exportSummaryToBigQuery:true;
numEvents:10000000
0002 query:SELECTION; exportSummaryToBigQuery:true; numEvents:10000000
0003 query:LOCAL_ITEM_SUGGESTION; exportSummaryToBigQuery:true;
numEvents:10000000
0004 query:AVERAGE_PRICE_FOR_CATEGORY; exportSummaryToBigQuery:true;
numEvents:1000000
0005 query:HOT_ITEMS; exportSummaryToBigQuery:true; numEvents:10000000
0006 query:AVERAGE_SELLING_PRICE_BY_SELLER; exportSummaryToBigQuery:true;
numEvents:1000000
0007 query:HIGHEST_BID; exportSummaryToBigQuery:true; numEvents:10000000
0008 query:MONITOR_NEW_USERS; exportSummaryToBigQuery:true;
numEvents:10000000
0009 query:WINNING_BIDS; exportSummaryToBigQuery:true; numEvents:1000000
0010 query:LOG_TO_SHARDED_FILES; exportSummaryToBigQuery:true;
numEvents:10000000
0011 query:USER_SESSIONS; exportSummaryToBigQuery:true; numEvents:10000000
0012 query:PROCESSING_TIME_WINDOWS; exportSummaryToBigQuery:true;
numEvents:10000000
0013 query:BOUNDED_SIDE_INPUT_JOIN; exportSummaryToBigQuery:true;
numEvents:10000000
0014 query:SESSION_SIDE_INPUT_JOIN; exportSummaryToBigQuery:true;
numEvents:10000000
Performance:
Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results
(Baseline)
0000 23.6 424142.2 10000000
0001 19.1 522821.1 9200000
0002 12.8 781250.0 83527
0003 42.6 234780.4 59435
0004 60.2 16611.6 140
0005 151.6 65949.6 4567
0006 35.6 28056.0 14085
0007 -1.0 -1.0 -1
*** Job was unexpectedly updated ***
0008 52.2 191581.9 92833
0009 42.7 23392.4 32548
0010 96.0 104189.5 100
0011 *** not run ***
0012 106.4 93947.9 609995
0013 41.3 242001.8 9200000
0014 150.1 66638.7 9200000
==========================================================================================
2020-08-25T00:13:09.046Z 6 samples not enough to calculate steady-state event
rate
Aug 25, 2020 12:13:09 AM org.apache.beam.sdk.testutils.metrics.MetricsReader
getCounterMetric
SEVERE: Failed to get metric fatal, from namespace Query11
2020-08-25T00:13:09.113Z no activity
2020-08-25T00:13:24.170Z RUNNING Query11
2020-08-25T00:13:25.209Z 6 samples not enough to calculate steady-state event
rate
Aug 25, 2020 12:13:25 AM org.apache.beam.sdk.testutils.metrics.MetricsReader
getCounterMetric
SEVERE: Failed to get metric fatal, from namespace Query11
2020-08-25T00:13:25.286Z no activity
2020-08-25T00:13:40.322Z RUNNING Query11
2020-08-25T00:13:41.215Z 6 samples not enough to calculate steady-state event
rate
Aug 25, 2020 12:13:41 AM org.apache.beam.sdk.testutils.metrics.MetricsReader
getCounterMetric
SEVERE: Failed to get metric fatal, from namespace Query11
2020-08-25T00:13:41.298Z no activity
2020-08-25T00:13:56.357Z DONE Query11
2020-08-25T00:13:56.405Z 6 samples not enough to calculate steady-state event
rate
Aug 25, 2020 12:13:56 AM org.apache.beam.sdk.testutils.metrics.MetricsReader
getCounterMetric
SEVERE: Failed to get metric fatal, from namespace Query11
2020-08-25T00:13:56.406Z streaming query appears to have finished waiting for
completion.
==========================================================================================
Run started 2020-08-25T00:07:08.545Z and ran for PT407.862S
Default configuration:
{"debug":true,"query":null,"sourceType":"DIRECT","sinkType":"DEVNULL","exportSummaryToBigQuery":false,"pubSubMode":"COMBINED","pubsubMessageSerializationMethod":"CODER","sideInputType":"DIRECT","sideInputRowCount":500,"sideInputNumShards":3,"sideInputUrl":null,"sessionGap":{"standardDays":0,"standardHours":0,"standardMinutes":10,"standardSeconds":600,"millis":600000},"numEvents":100000,"numEventGenerators":100,"rateShape":"SINE","firstEventRate":10000,"nextEventRate":10000,"rateUnit":"PER_SECOND","ratePeriodSec":600,"preloadSeconds":0,"streamTimeout":240,"isRateLimited":false,"useWallclockEventTime":false,"avgPersonByteSize":200,"avgAuctionByteSize":500,"avgBidByteSize":100,"hotAuctionRatio":2,"hotSellersRatio":4,"hotBiddersRatio":4,"windowSizeSec":10,"windowPeriodSec":5,"watermarkHoldbackSec":0,"numInFlightAuctions":100,"numActivePeople":1000,"coderStrategy":"HAND","cpuDelayMs":0,"diskBusyBytes":0,"auctionSkip":123,"fanout":5,"maxAuctionsWaitingTime":600,"occasionalDelaySec":3,"probDelayedEvent":0.1,"maxLogEvents":100000,"usePubsubPublishTime":false,"outOfOrderGroupSize":1}
Configurations:
Conf Description
0000 query:PASSTHROUGH; exportSummaryToBigQuery:true; numEvents:10000000
0001 query:CURRENCY_CONVERSION; exportSummaryToBigQuery:true;
numEvents:10000000
0002 query:SELECTION; exportSummaryToBigQuery:true; numEvents:10000000
0003 query:LOCAL_ITEM_SUGGESTION; exportSummaryToBigQuery:true;
numEvents:10000000
0004 query:AVERAGE_PRICE_FOR_CATEGORY; exportSummaryToBigQuery:true;
numEvents:1000000
0005 query:HOT_ITEMS; exportSummaryToBigQuery:true; numEvents:10000000
0006 query:AVERAGE_SELLING_PRICE_BY_SELLER; exportSummaryToBigQuery:true;
numEvents:1000000
0007 query:HIGHEST_BID; exportSummaryToBigQuery:true; numEvents:10000000
0008 query:MONITOR_NEW_USERS; exportSummaryToBigQuery:true;
numEvents:10000000
0009 query:WINNING_BIDS; exportSummaryToBigQuery:true; numEvents:1000000
0010 query:LOG_TO_SHARDED_FILES; exportSummaryToBigQuery:true;
numEvents:10000000
0011 query:USER_SESSIONS; exportSummaryToBigQuery:true; numEvents:10000000
0012 query:PROCESSING_TIME_WINDOWS; exportSummaryToBigQuery:true;
numEvents:10000000
0013 query:BOUNDED_SIDE_INPUT_JOIN; exportSummaryToBigQuery:true;
numEvents:10000000
0014 query:SESSION_SIDE_INPUT_JOIN; exportSummaryToBigQuery:true;
numEvents:10000000
Performance:
Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results
(Baseline)
0000 23.6 424142.2 10000000
0001 19.1 522821.1 9200000
0002 12.8 781250.0 83527
0003 42.6 234780.4 59435
0004 60.2 16611.6 140
0005 151.6 65949.6 4567
0006 35.6 28056.0 14085
0007 -1.0 -1.0 -1
*** Job was unexpectedly updated ***
0008 52.2 191581.9 92833
0009 42.7 23392.4 32548
0010 96.0 104189.5 100
0011 207.6 48163.8 1839657
0012 106.4 93947.9 609995
0013 41.3 242001.8 9200000
0014 150.1 66638.7 9200000
==========================================================================================
==========================================================================================
Run started 2020-08-25T00:07:08.545Z and ran for PT411.750S
Default configuration:
{"debug":true,"query":null,"sourceType":"DIRECT","sinkType":"DEVNULL","exportSummaryToBigQuery":false,"pubSubMode":"COMBINED","pubsubMessageSerializationMethod":"CODER","sideInputType":"DIRECT","sideInputRowCount":500,"sideInputNumShards":3,"sideInputUrl":null,"sessionGap":{"standardDays":0,"standardHours":0,"standardMinutes":10,"standardSeconds":600,"millis":600000},"numEvents":100000,"numEventGenerators":100,"rateShape":"SINE","firstEventRate":10000,"nextEventRate":10000,"rateUnit":"PER_SECOND","ratePeriodSec":600,"preloadSeconds":0,"streamTimeout":240,"isRateLimited":false,"useWallclockEventTime":false,"avgPersonByteSize":200,"avgAuctionByteSize":500,"avgBidByteSize":100,"hotAuctionRatio":2,"hotSellersRatio":4,"hotBiddersRatio":4,"windowSizeSec":10,"windowPeriodSec":5,"watermarkHoldbackSec":0,"numInFlightAuctions":100,"numActivePeople":1000,"coderStrategy":"HAND","cpuDelayMs":0,"diskBusyBytes":0,"auctionSkip":123,"fanout":5,"maxAuctionsWaitingTime":600,"occasionalDelaySec":3,"probDelayedEvent":0.1,"maxLogEvents":100000,"usePubsubPublishTime":false,"outOfOrderGroupSize":1}
Configurations:
Conf Description
0000 query:PASSTHROUGH; exportSummaryToBigQuery:true; numEvents:10000000
0001 query:CURRENCY_CONVERSION; exportSummaryToBigQuery:true;
numEvents:10000000
0002 query:SELECTION; exportSummaryToBigQuery:true; numEvents:10000000
0003 query:LOCAL_ITEM_SUGGESTION; exportSummaryToBigQuery:true;
numEvents:10000000
0004 query:AVERAGE_PRICE_FOR_CATEGORY; exportSummaryToBigQuery:true;
numEvents:1000000
0005 query:HOT_ITEMS; exportSummaryToBigQuery:true; numEvents:10000000
0006 query:AVERAGE_SELLING_PRICE_BY_SELLER; exportSummaryToBigQuery:true;
numEvents:1000000
0007 query:HIGHEST_BID; exportSummaryToBigQuery:true; numEvents:10000000
0008 query:MONITOR_NEW_USERS; exportSummaryToBigQuery:true;
numEvents:10000000
0009 query:WINNING_BIDS; exportSummaryToBigQuery:true; numEvents:1000000
0010 query:LOG_TO_SHARDED_FILES; exportSummaryToBigQuery:true;
numEvents:10000000
0011 query:USER_SESSIONS; exportSummaryToBigQuery:true; numEvents:10000000
0012 query:PROCESSING_TIME_WINDOWS; exportSummaryToBigQuery:true;
numEvents:10000000
0013 query:BOUNDED_SIDE_INPUT_JOIN; exportSummaryToBigQuery:true;
numEvents:10000000
0014 query:SESSION_SIDE_INPUT_JOIN; exportSummaryToBigQuery:true;
numEvents:10000000
Performance:
Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results
(Baseline)
0000 23.6 424142.2 10000000
0001 19.1 522821.1 9200000
0002 12.8 781250.0 83527
0003 42.6 234780.4 59435
0004 60.2 16611.6 140
0005 151.6 65949.6 4567
0006 35.6 28056.0 14085
0007 -1.0 -1.0 -1
*** Job was unexpectedly updated ***
0008 52.2 191581.9 92833
0009 42.7 23392.4 32548
0010 96.0 104189.5 100
0011 207.6 48163.8 1839657
0012 106.4 93947.9 609995
0013 41.3 242001.8 9200000
0014 150.1 66638.7 9200000
==========================================================================================
Exception in thread "main" java.lang.RuntimeException: Execution was not
successful
at org.apache.beam.sdk.nexmark.Main.runAll(Main.java:182)
at org.apache.beam.sdk.nexmark.Main.main(Main.java:490)
> Task :sdks:java:testing:nexmark:run FAILED
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:java:testing:nexmark:run'.
> Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with
> non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See
https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 7m 2s
89 actionable tasks: 2 executed, 87 up-to-date
Publishing build scan...
https://gradle.com/s/2c2dcb5gr6g4o
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
