See
<https://ci-beam.apache.org/job/beam_PostCommit_Java_Nexmark_Direct/7251/display/redirect?page=changes>
Changes:
[MATTHEW.Ouyang] [BEAM-12059] include literal T in DATETIME format
[suztomo] protobuf-java to be in-line with libraries-bom 16.3.0
[suztomo] Removing unused variable google_auth_version
[noreply] Merge pull request #14388 from [BEAM-7372] remove codes for py2
[noreply] Merge pull request #14365 from [BEAM-10884] - Adding tests to
------------------------------------------
[...truncated 220.58 KB...]
0010 query:LOG_TO_SHARDED_FILES; exportSummaryToBigQuery:true
0011 query:USER_SESSIONS; exportSummaryToBigQuery:true
0012 query:PROCESSING_TIME_WINDOWS; exportSummaryToBigQuery:true
0013 query:PORTABILITY_BATCH; exportSummaryToBigQuery:true
0014 query:BOUNDED_SIDE_INPUT_JOIN; exportSummaryToBigQuery:true
0015 query:SESSION_SIDE_INPUT_JOIN; exportSummaryToBigQuery:true
Performance:
Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results
(Baseline)
0000 4.9 20487.6 100000
0001 3.5 28473.8 92000
0002 1.6 62774.6 351
0003 3.0 32970.7 580
0004 1.5 6825.9 40
0005 7.5 13399.4 12
0006 1.0 10245.9 401
0007 8.0 12479.7 1
0008 2.5 39463.3 6000
0009 0.7 13966.5 298
0010 14.8 6757.7 1
0011 *** not run ***
0012 *** not run ***
0013 *** not run ***
0014 *** not run ***
0015 *** not run ***
==========================================================================================
2021-04-07T06:14:15.644Z Generating 100000 events in streaming mode
2021-04-07T06:14:19.388Z Waiting for main pipeline to 'finish'
2021-04-07T06:14:19.388Z DONE Query11
2021-04-07T06:14:19.388Z 1 samples not enough to calculate steady-state event
rate
Apr 07, 2021 6:14:19 AM org.apache.beam.sdk.testutils.metrics.MetricsReader
getCounterMetric
SEVERE: Failed to get metric fatal, from namespace Query11
2021-04-07T06:14:19.389Z Running query:PROCESSING_TIME_WINDOWS;
exportSummaryToBigQuery:true
==========================================================================================
Run started 2021-04-07T06:13:20.106Z and ran for PT59.283S
Default configuration:
{"debug":true,"query":null,"sourceType":"DIRECT","generateEventFilePathPrefix":null,"sinkType":"DEVNULL","exportSummaryToBigQuery":false,"pubSubMode":"COMBINED","pubsubMessageSerializationMethod":"CODER","sideInputType":"DIRECT","sideInputRowCount":500,"sideInputNumShards":3,"sideInputUrl":null,"sessionGap":600000,"numEvents":100000,"numEventGenerators":100,"rateShape":"SINE","firstEventRate":10000,"nextEventRate":10000,"rateUnit":"PER_SECOND","ratePeriodSec":600,"preloadSeconds":0,"streamTimeout":240,"isRateLimited":false,"useWallclockEventTime":false,"avgPersonByteSize":200,"avgAuctionByteSize":500,"avgBidByteSize":100,"hotAuctionRatio":2,"hotSellersRatio":4,"hotBiddersRatio":4,"windowSizeSec":10,"windowPeriodSec":5,"watermarkHoldbackSec":0,"numInFlightAuctions":100,"numActivePeople":1000,"coderStrategy":"HAND","cpuDelayMs":0,"diskBusyBytes":0,"auctionSkip":123,"fanout":5,"maxAuctionsWaitingTime":600,"occasionalDelaySec":3,"probDelayedEvent":0.1,"maxLogEvents":100000,"usePubsubPublishTime":false,"outOfOrderGroupSize":1,"numKeyBuckets":20000,"pardoCPUFactor":1.0}
Configurations:
Conf Description
0000 query:PASSTHROUGH; exportSummaryToBigQuery:true
0001 query:CURRENCY_CONVERSION; exportSummaryToBigQuery:true
0002 query:SELECTION; exportSummaryToBigQuery:true
0003 query:LOCAL_ITEM_SUGGESTION; exportSummaryToBigQuery:true
0004 query:AVERAGE_PRICE_FOR_CATEGORY; exportSummaryToBigQuery:true;
numEvents:10000
0005 query:HOT_ITEMS; exportSummaryToBigQuery:true
0006 query:AVERAGE_SELLING_PRICE_BY_SELLER; exportSummaryToBigQuery:true;
numEvents:10000
0007 query:HIGHEST_BID; exportSummaryToBigQuery:true
0008 query:MONITOR_NEW_USERS; exportSummaryToBigQuery:true
0009 query:WINNING_BIDS; exportSummaryToBigQuery:true; numEvents:10000
0010 query:LOG_TO_SHARDED_FILES; exportSummaryToBigQuery:true
0011 query:USER_SESSIONS; exportSummaryToBigQuery:true
0012 query:PROCESSING_TIME_WINDOWS; exportSummaryToBigQuery:true
0013 query:PORTABILITY_BATCH; exportSummaryToBigQuery:true
0014 query:BOUNDED_SIDE_INPUT_JOIN; exportSummaryToBigQuery:true
0015 query:SESSION_SIDE_INPUT_JOIN; exportSummaryToBigQuery:true
Performance:
Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results
(Baseline)
0000 4.9 20487.6 100000
0001 3.5 28473.8 92000
0002 1.6 62774.6 351
0003 3.0 32970.7 580
0004 1.5 6825.9 40
0005 7.5 13399.4 12
0006 1.0 10245.9 401
0007 8.0 12479.7 1
0008 2.5 39463.3 6000
0009 0.7 13966.5 298
0010 14.8 6757.7 1
0011 3.3 30193.2 1919
0012 *** not run ***
0013 *** not run ***
0014 *** not run ***
0015 *** not run ***
==========================================================================================
2021-04-07T06:14:19.391Z Generating 100000 events in streaming mode
2021-04-07T06:14:22.025Z Waiting for main pipeline to 'finish'
2021-04-07T06:14:22.025Z DONE Query12
2021-04-07T06:14:22.026Z 1 samples not enough to calculate steady-state event
rate
Apr 07, 2021 6:14:22 AM org.apache.beam.sdk.testutils.metrics.MetricsReader
getCounterMetric
SEVERE: Failed to get metric fatal, from namespace Query12
2021-04-07T06:14:22.026Z Running query:PORTABILITY_BATCH;
exportSummaryToBigQuery:true
==========================================================================================
Run started 2021-04-07T06:13:20.106Z and ran for PT61.920S
Default configuration:
{"debug":true,"query":null,"sourceType":"DIRECT","generateEventFilePathPrefix":null,"sinkType":"DEVNULL","exportSummaryToBigQuery":false,"pubSubMode":"COMBINED","pubsubMessageSerializationMethod":"CODER","sideInputType":"DIRECT","sideInputRowCount":500,"sideInputNumShards":3,"sideInputUrl":null,"sessionGap":600000,"numEvents":100000,"numEventGenerators":100,"rateShape":"SINE","firstEventRate":10000,"nextEventRate":10000,"rateUnit":"PER_SECOND","ratePeriodSec":600,"preloadSeconds":0,"streamTimeout":240,"isRateLimited":false,"useWallclockEventTime":false,"avgPersonByteSize":200,"avgAuctionByteSize":500,"avgBidByteSize":100,"hotAuctionRatio":2,"hotSellersRatio":4,"hotBiddersRatio":4,"windowSizeSec":10,"windowPeriodSec":5,"watermarkHoldbackSec":0,"numInFlightAuctions":100,"numActivePeople":1000,"coderStrategy":"HAND","cpuDelayMs":0,"diskBusyBytes":0,"auctionSkip":123,"fanout":5,"maxAuctionsWaitingTime":600,"occasionalDelaySec":3,"probDelayedEvent":0.1,"maxLogEvents":100000,"usePubsubPublishTime":false,"outOfOrderGroupSize":1,"numKeyBuckets":20000,"pardoCPUFactor":1.0}
Configurations:
Conf Description
0000 query:PASSTHROUGH; exportSummaryToBigQuery:true
0001 query:CURRENCY_CONVERSION; exportSummaryToBigQuery:true
0002 query:SELECTION; exportSummaryToBigQuery:true
0003 query:LOCAL_ITEM_SUGGESTION; exportSummaryToBigQuery:true
0004 query:AVERAGE_PRICE_FOR_CATEGORY; exportSummaryToBigQuery:true;
numEvents:10000
0005 query:HOT_ITEMS; exportSummaryToBigQuery:true
0006 query:AVERAGE_SELLING_PRICE_BY_SELLER; exportSummaryToBigQuery:true;
numEvents:10000
0007 query:HIGHEST_BID; exportSummaryToBigQuery:true
0008 query:MONITOR_NEW_USERS; exportSummaryToBigQuery:true
0009 query:WINNING_BIDS; exportSummaryToBigQuery:true; numEvents:10000
0010 query:LOG_TO_SHARDED_FILES; exportSummaryToBigQuery:true
0011 query:USER_SESSIONS; exportSummaryToBigQuery:true
0012 query:PROCESSING_TIME_WINDOWS; exportSummaryToBigQuery:true
0013 query:PORTABILITY_BATCH; exportSummaryToBigQuery:true
0014 query:BOUNDED_SIDE_INPUT_JOIN; exportSummaryToBigQuery:true
0015 query:SESSION_SIDE_INPUT_JOIN; exportSummaryToBigQuery:true
Performance:
Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results
(Baseline)
0000 4.9 20487.6 100000
0001 3.5 28473.8 92000
0002 1.6 62774.6 351
0003 3.0 32970.7 580
0004 1.5 6825.9 40
0005 7.5 13399.4 12
0006 1.0 10245.9 401
0007 8.0 12479.7 1
0008 2.5 39463.3 6000
0009 0.7 13966.5 298
0010 14.8 6757.7 1
0011 3.3 30193.2 1919
0012 2.2 44903.5 1919
0013 *** not run ***
0014 *** not run ***
0015 *** not run ***
==========================================================================================
2021-04-07T06:14:22.028Z Generating 100000 events in streaming mode
2021-04-07T06:14:22.040Z Running query:BOUNDED_SIDE_INPUT_JOIN;
exportSummaryToBigQuery:true
==========================================================================================
Run started 2021-04-07T06:13:20.106Z and ran for PT61.934S
Default configuration:
{"debug":true,"query":null,"sourceType":"DIRECT","generateEventFilePathPrefix":null,"sinkType":"DEVNULL","exportSummaryToBigQuery":false,"pubSubMode":"COMBINED","pubsubMessageSerializationMethod":"CODER","sideInputType":"DIRECT","sideInputRowCount":500,"sideInputNumShards":3,"sideInputUrl":null,"sessionGap":600000,"numEvents":100000,"numEventGenerators":100,"rateShape":"SINE","firstEventRate":10000,"nextEventRate":10000,"rateUnit":"PER_SECOND","ratePeriodSec":600,"preloadSeconds":0,"streamTimeout":240,"isRateLimited":false,"useWallclockEventTime":false,"avgPersonByteSize":200,"avgAuctionByteSize":500,"avgBidByteSize":100,"hotAuctionRatio":2,"hotSellersRatio":4,"hotBiddersRatio":4,"windowSizeSec":10,"windowPeriodSec":5,"watermarkHoldbackSec":0,"numInFlightAuctions":100,"numActivePeople":1000,"coderStrategy":"HAND","cpuDelayMs":0,"diskBusyBytes":0,"auctionSkip":123,"fanout":5,"maxAuctionsWaitingTime":600,"occasionalDelaySec":3,"probDelayedEvent":0.1,"maxLogEvents":100000,"usePubsubPublishTime":false,"outOfOrderGroupSize":1,"numKeyBuckets":20000,"pardoCPUFactor":1.0}
Configurations:
Conf Description
0000 query:PASSTHROUGH; exportSummaryToBigQuery:true
0001 query:CURRENCY_CONVERSION; exportSummaryToBigQuery:true
0002 query:SELECTION; exportSummaryToBigQuery:true
0003 query:LOCAL_ITEM_SUGGESTION; exportSummaryToBigQuery:true
0004 query:AVERAGE_PRICE_FOR_CATEGORY; exportSummaryToBigQuery:true;
numEvents:10000
0005 query:HOT_ITEMS; exportSummaryToBigQuery:true
0006 query:AVERAGE_SELLING_PRICE_BY_SELLER; exportSummaryToBigQuery:true;
numEvents:10000
0007 query:HIGHEST_BID; exportSummaryToBigQuery:true
0008 query:MONITOR_NEW_USERS; exportSummaryToBigQuery:true
0009 query:WINNING_BIDS; exportSummaryToBigQuery:true; numEvents:10000
0010 query:LOG_TO_SHARDED_FILES; exportSummaryToBigQuery:true
0011 query:USER_SESSIONS; exportSummaryToBigQuery:true
0012 query:PROCESSING_TIME_WINDOWS; exportSummaryToBigQuery:true
0013 query:PORTABILITY_BATCH; exportSummaryToBigQuery:true
0014 query:BOUNDED_SIDE_INPUT_JOIN; exportSummaryToBigQuery:true
0015 query:SESSION_SIDE_INPUT_JOIN; exportSummaryToBigQuery:true
Performance:
Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results
(Baseline)
Exception in thread "main" 0000 4.9 20487.6
100000
0001 3.5 28473.8 92000
0002 1.6 62774.6 351
0003 3.0 32970.7 580
0004 1.5 6825.9 40
0005 7.5 13399.4 12
0006 1.0 10245.9 401
0007 8.0 12479.7 1
0008 2.5 39463.3 6000
java.lang.RuntimeException: java.lang.IllegalStateException: GroupByKey cannot
be applied to non-bounded PCollection in the GlobalWindow without a trigger.
Use a Window.into or Window.triggering transform prior to GroupByKey.
at org.apache.beam.sdk.nexmark.Main.runAll(Main.java:138)
at org.apache.beam.sdk.nexmark.Main.main(Main.java:494)
0009 0.7 13966.5 298
0010 14.8 6757.7 1
0011 3.3 30193.2 1919
0012 2.2 44903.5 1919
0013 *** not run ***
0014 *** not run ***
0015 *** not run ***
==========================================================================================
Caused by: java.lang.IllegalStateException: GroupByKey cannot be applied to
non-bounded PCollection in the GlobalWindow without a trigger. Use a
Window.into or Window.triggering transform prior to GroupByKey.
at
org.apache.beam.sdk.transforms.GroupByKey.applicableTo(GroupByKey.java:155)
at org.apache.beam.sdk.transforms.GroupByKey.expand(GroupByKey.java:212)
at org.apache.beam.sdk.transforms.GroupByKey.expand(GroupByKey.java:109)
at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:547)
at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:481)
at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:357)
at org.apache.beam.sdk.nexmark.queries.Query13.expand(Query13.java:60)
2021-04-07T06:14:22.043Z Generating 100000 events in streaming mode
at org.apache.beam.sdk.nexmark.queries.Query13.expand(Query13.java:44)
at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:547)
at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:481)
at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:357)
at
org.apache.beam.sdk.nexmark.queries.NexmarkQuery.expand(NexmarkQuery.java:89)
at
org.apache.beam.sdk.nexmark.queries.NexmarkQuery.expand(NexmarkQuery.java:33)
at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:547)
at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:481)
at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:357)
at
org.apache.beam.sdk.nexmark.NexmarkLauncher.run(NexmarkLauncher.java:1219)
at org.apache.beam.sdk.nexmark.Main$Run.call(Main.java:100)
at org.apache.beam.sdk.nexmark.Main$Run.call(Main.java:89)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2021-04-07T06:14:25.015Z Waiting for main pipeline to 'finish'
2021-04-07T06:14:25.015Z DONE BoundedSideInputJoin
2021-04-07T06:14:25.015Z 1 samples not enough to calculate steady-state event
rate
Apr 07, 2021 6:14:25 AM org.apache.beam.sdk.testutils.metrics.MetricsReader
getCounterMetric
SEVERE: Failed to get metric fatal, from namespace BoundedSideInputJoin
2021-04-07T06:14:25.016Z Running query:SESSION_SIDE_INPUT_JOIN;
exportSummaryToBigQuery:true
2021-04-07T06:14:25.017Z Generating 100000 events in streaming mode
2021-04-07T06:14:29.309Z Waiting for main pipeline to 'finish'
2021-04-07T06:14:29.309Z DONE SessionSideInputJoin
2021-04-07T06:14:29.310Z 1 samples not enough to calculate steady-state event
rate
Apr 07, 2021 6:14:29 AM org.apache.beam.sdk.testutils.metrics.MetricsReader
getCounterMetric
SEVERE: Failed to get metric fatal, from namespace SessionSideInputJoin
> Task :sdks:java:testing:nexmark:run FAILED
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:java:testing:nexmark:run'.
> Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with
> non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See
https://docs.gradle.org/6.8/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 1m 16s
100 actionable tasks: 2 executed, 98 up-to-date
Publishing build scan...
https://gradle.com/s/7clwteekk4zi4
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
