See
<https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Direct/3/display/redirect?page=changes>
Changes:
[jiangkai] introduce optimized logic plan
[jiangkai] update
[jiangkai] fix unit tests failed
[jiangkai] address comments
[jiangkai] separate rules into sets
[jiangkai] explain plan unit tests
[jiangkai] address conflicts and comments
[jiangkai] remove volcano plan function
[jiangkai] ignore explain unit test
------------------------------------------
[...truncated 221.25 KB...]
, 2015-07-15T00:00:09.999Z shard-00015-of-00025 0 ON_TIME null
, 2015-07-15T00:00:09.999Z shard-00018-of-00025 0 ON_TIME null
, 2015-07-15T00:00:09.999Z shard-00014-of-00025 0 ON_TIME null
, 2015-07-15T00:00:09.999Z shard-00012-of-00025 0 ON_TIME null
, 2015-07-15T00:00:09.999Z shard-00017-of-00025 0 ON_TIME null
, 2015-07-15T00:00:09.999Z shard-00020-of-00025 0 ON_TIME null
, 2015-07-15T00:00:09.999Z shard-00005-of-00025 0 ON_TIME null
, 2015-07-15T00:00:09.999Z shard-00009-of-00025 0 ON_TIME null
]}
Jun 19, 2018 5:37:11 PM
org.apache.beam.sdk.util.MutationDetectors$CodedValueMutationDetector
verifyUnmodifiedThrowingCheckedExceptions
WARNING: Coder of type class org.apache.beam.sdk.coders.KvCoder has a
#structuralValue method which does not return true when the encoding of the
elements is equal. Element KV{null, [2015-07-15T00:00:09.999Z
shard-00022-of-00025 0 ON_TIME null
, 2015-07-15T00:00:09.999Z shard-00019-of-00025 0 ON_TIME null
, 2015-07-15T00:00:09.999Z shard-00010-of-00025 0 ON_TIME null
, 2015-07-15T00:00:09.999Z shard-00006-of-00025 0 ON_TIME null
, 2015-07-15T00:00:09.999Z shard-00016-of-00025 0 ON_TIME null
, 2015-07-15T00:00:09.999Z shard-00004-of-00025 0 ON_TIME null
, 2015-07-15T00:00:09.999Z shard-00008-of-00025 0 ON_TIME null
, 2015-07-15T00:00:09.999Z shard-00023-of-00025 0 ON_TIME null
, 2015-07-15T00:00:09.999Z shard-00007-of-00025 0 ON_TIME null
, 2015-07-15T00:00:09.999Z shard-00013-of-00025 0 ON_TIME null
, 2015-07-15T00:00:09.999Z shard-00003-of-00025 0 ON_TIME null
, 2015-07-15T00:00:09.999Z shard-00024-of-00025 0 ON_TIME null
, 2015-07-15T00:00:09.999Z shard-00021-of-00025 0 ON_TIME null
, 2015-07-15T00:00:09.999Z shard-00000-of-00025 0 ON_TIME null
, 2015-07-15T00:00:09.999Z shard-00001-of-00025 0 ON_TIME null
, 2015-07-15T00:00:09.999Z shard-00002-of-00025 0 ON_TIME null
, 2015-07-15T00:00:09.999Z shard-00011-of-00025 0 ON_TIME null
, 2015-07-15T00:00:09.999Z shard-00015-of-00025 0 ON_TIME null
, 2015-07-15T00:00:09.999Z shard-00018-of-00025 0 ON_TIME null
, 2015-07-15T00:00:09.999Z shard-00014-of-00025 0 ON_TIME null
, 2015-07-15T00:00:09.999Z shard-00012-of-00025 0 ON_TIME null
, 2015-07-15T00:00:09.999Z shard-00017-of-00025 0 ON_TIME null
, 2015-07-15T00:00:09.999Z shard-00020-of-00025 0 ON_TIME null
, 2015-07-15T00:00:09.999Z shard-00005-of-00025 0 ON_TIME null
, 2015-07-15T00:00:09.999Z shard-00009-of-00025 0 ON_TIME null
]}
2018-06-19T17:37:11.689Z Waiting for main pipeline to 'finish'
2018-06-19T17:37:11.689Z DONE Query10
==========================================================================================
Run started 2018-06-19T17:36:30.155Z and ran for PT41.534S
Default configuration:
{"debug":true,"query":0,"sourceType":"DIRECT","sinkType":"DEVNULL","exportSummaryToBigQuery":false,"pubSubMode":"COMBINED","numEvents":100000,"numEventGenerators":100,"rateShape":"SINE","firstEventRate":10000,"nextEventRate":10000,"rateUnit":"PER_SECOND","ratePeriodSec":600,"preloadSeconds":0,"streamTimeout":240,"isRateLimited":false,"useWallclockEventTime":false,"avgPersonByteSize":200,"avgAuctionByteSize":500,"avgBidByteSize":100,"hotAuctionRatio":2,"hotSellersRatio":4,"hotBiddersRatio":4,"windowSizeSec":10,"windowPeriodSec":5,"watermarkHoldbackSec":0,"numInFlightAuctions":100,"numActivePeople":1000,"coderStrategy":"HAND","cpuDelayMs":0,"diskBusyBytes":0,"auctionSkip":123,"fanout":5,"maxAuctionsWaitingTime":600,"occasionalDelaySec":3,"probDelayedEvent":0.1,"maxLogEvents":100000,"usePubsubPublishTime":false,"outOfOrderGroupSize":1}
Configurations:
Conf Description
0000 query:0; exportSummaryToBigQuery:true
0001 query:1; exportSummaryToBigQuery:true
0002 query:2; exportSummaryToBigQuery:true
0003 query:3; exportSummaryToBigQuery:true
0004 query:4; exportSummaryToBigQuery:true; numEvents:10000
0005 query:5; exportSummaryToBigQuery:true
0006 query:6; exportSummaryToBigQuery:true; numEvents:10000
0007 query:7; exportSummaryToBigQuery:true
0008 query:8; exportSummaryToBigQuery:true
0009 query:9; exportSummaryToBigQuery:true; numEvents:10000
0010 query:10; exportSummaryToBigQuery:true
0011 query:11; exportSummaryToBigQuery:true
0012 query:12; exportSummaryToBigQuery:true
Performance:
Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results
(Baseline)
0000 4.5 22207.4 100000
0001 2.8 36350.4 92000
0002 1.0 105152.5 351
0003 2.3 43668.1 454
0004 1.3 7473.8 40
0005 4.1 24685.3 12
0006 0.7 13459.0 401
0007 9.6 10457.0 1
0008 1.2 85324.2 6000
0009 0.6 15772.9 298
0010 10.2 9771.4 1
0011 *** not run ***
0012 *** not run ***
==========================================================================================
2018-06-19T17:37:11.691Z Running query:11; exportSummaryToBigQuery:true
2018-06-19T17:37:11.692Z Generating 100000 events in batch mode
2018-06-19T17:37:14.403Z Waiting for main pipeline to 'finish'
2018-06-19T17:37:14.403Z DONE Query11
==========================================================================================
Run started 2018-06-19T17:36:30.155Z and ran for PT44.249S
Default configuration:
{"debug":true,"query":0,"sourceType":"DIRECT","sinkType":"DEVNULL","exportSummaryToBigQuery":false,"pubSubMode":"COMBINED","numEvents":100000,"numEventGenerators":100,"rateShape":"SINE","firstEventRate":10000,"nextEventRate":10000,"rateUnit":"PER_SECOND","ratePeriodSec":600,"preloadSeconds":0,"streamTimeout":240,"isRateLimited":false,"useWallclockEventTime":false,"avgPersonByteSize":200,"avgAuctionByteSize":500,"avgBidByteSize":100,"hotAuctionRatio":2,"hotSellersRatio":4,"hotBiddersRatio":4,"windowSizeSec":10,"windowPeriodSec":5,"watermarkHoldbackSec":0,"numInFlightAuctions":100,"numActivePeople":1000,"coderStrategy":"HAND","cpuDelayMs":0,"diskBusyBytes":0,"auctionSkip":123,"fanout":5,"maxAuctionsWaitingTime":600,"occasionalDelaySec":3,"probDelayedEvent":0.1,"maxLogEvents":100000,"usePubsubPublishTime":false,"outOfOrderGroupSize":1}
Configurations:
Conf Description
0000 query:0; exportSummaryToBigQuery:true
0001 query:1; exportSummaryToBigQuery:true
0002 query:2; exportSummaryToBigQuery:true
0003 query:3; exportSummaryToBigQuery:true
0004 query:4; exportSummaryToBigQuery:true; numEvents:10000
0005 query:5; exportSummaryToBigQuery:true
0006 query:6; exportSummaryToBigQuery:true; numEvents:10000
0007 query:7; exportSummaryToBigQuery:true
0008 query:8; exportSummaryToBigQuery:true
0009 query:9; exportSummaryToBigQuery:true; numEvents:10000
0010 query:10; exportSummaryToBigQuery:true
0011 query:11; exportSummaryToBigQuery:true
0012 query:12; exportSummaryToBigQuery:true
Performance:
Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results
(Baseline)
0000 4.5 22207.4 100000
0001 2.8 36350.4 92000
0002 1.0 105152.5 351
0003 2.3 43668.1 454
0004 1.3 7473.8 40
0005 4.1 24685.3 12
0006 0.7 13459.0 401
0007 9.6 10457.0 1
0008 1.2 85324.2 6000
0009 0.6 15772.9 298
0010 10.2 9771.4 1
0011 2.5 39666.8 1919
0012 *** not run ***
==========================================================================================
2018-06-19T17:37:14.405Z Running query:12; exportSummaryToBigQuery:true
2018-06-19T17:37:14.406Z Generating 100000 events in batch mode
2018-06-19T17:37:15.937Z Waiting for main pipeline to 'finish'
2018-06-19T17:37:15.937Z DONE Query12
==========================================================================================
Run started 2018-06-19T17:36:30.155Z and ran for PT45.782S
Default configuration:
{"debug":true,"query":0,"sourceType":"DIRECT","sinkType":"DEVNULL","exportSummaryToBigQuery":false,"pubSubMode":"COMBINED","numEvents":100000,"numEventGenerators":100,"rateShape":"SINE","firstEventRate":10000,"nextEventRate":10000,"rateUnit":"PER_SECOND","ratePeriodSec":600,"preloadSeconds":0,"streamTimeout":240,"isRateLimited":false,"useWallclockEventTime":false,"avgPersonByteSize":200,"avgAuctionByteSize":500,"avgBidByteSize":100,"hotAuctionRatio":2,"hotSellersRatio":4,"hotBiddersRatio":4,"windowSizeSec":10,"windowPeriodSec":5,"watermarkHoldbackSec":0,"numInFlightAuctions":100,"numActivePeople":1000,"coderStrategy":"HAND","cpuDelayMs":0,"diskBusyBytes":0,"auctionSkip":123,"fanout":5,"maxAuctionsWaitingTime":600,"occasionalDelaySec":3,"probDelayedEvent":0.1,"maxLogEvents":100000,"usePubsubPublishTime":false,"outOfOrderGroupSize":1}
Configurations:
Conf Description
0000 query:0; exportSummaryToBigQuery:true
0001 query:1; exportSummaryToBigQuery:true
0002 query:2; exportSummaryToBigQuery:true
0003 query:3; exportSummaryToBigQuery:true
0004 query:4; exportSummaryToBigQuery:true; numEvents:10000
0005 query:5; exportSummaryToBigQuery:true
0006 query:6; exportSummaryToBigQuery:true; numEvents:10000
0007 query:7; exportSummaryToBigQuery:true
0008 query:8; exportSummaryToBigQuery:true
0009 query:9; exportSummaryToBigQuery:true; numEvents:10000
0010 query:10; exportSummaryToBigQuery:true
0011 query:11; exportSummaryToBigQuery:true
0012 query:12; exportSummaryToBigQuery:true
Performance:
Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results
(Baseline)
0000 4.5 22207.4 100000
0001 2.8 36350.4 92000
0002 1.0 105152.5 351
0003 2.3 43668.1 454
0004 1.3 7473.8 40
0005 4.1 24685.3 12
0006 0.7 13459.0 401
0007 9.6 10457.0 1
0008 1.2 85324.2 6000
0009 0.6 15772.9 298
0010 10.2 9771.4 1
0011 2.5 39666.8 1919
0012 1.4 71225.1 1919
==========================================================================================
==========================================================================================
Exception in thread "main" Run started 2018-06-19T17:36:30.155Z and ran for
PT46.124S
Default configuration:
{"debug":true,"query":0,"sourceType":"DIRECT","sinkType":"DEVNULL","exportSummaryToBigQuery":false,"pubSubMode":"COMBINED","numEvents":100000,"numEventGenerators":100,"rateShape":"SINE","firstEventRate":10000,"nextEventRate":10000,"rateUnit":"PER_SECOND","ratePeriodSec":600,"preloadSeconds":0,"streamTimeout":240,"isRateLimited":false,"useWallclockEventTime":false,"avgPersonByteSize":200,"avgAuctionByteSize":500,"avgBidByteSize":100,"hotAuctionRatio":2,"hotSellersRatio":4,"hotBiddersRatio":4,"windowSizeSec":10,"windowPeriodSec":5,"watermarkHoldbackSec":0,"numInFlightAuctions":100,"numActivePeople":1000,"coderStrategy":"HAND","cpuDelayMs":0,"diskBusyBytes":0,"auctionSkip":123,"fanout":5,"maxAuctionsWaitingTime":600,"occasionalDelaySec":3,"probDelayedEvent":0.1,"maxLogEvents":100000,"usePubsubPublishTime":false,"outOfOrderGroupSize":1}
Configurations:
Conf Description
0000 query:0; exportSummaryToBigQuery:true
0001 query:1; exportSummaryToBigQuery:true
0002 query:2; exportSummaryToBigQuery:true
0003 query:3; exportSummaryToBigQuery:true
0004 query:4; exportSummaryToBigQuery:true; numEvents:10000
0005 query:5; exportSummaryToBigQuery:true
0006 query:6; exportSummaryToBigQuery:true; numEvents:10000
0007 query:7; exportSummaryToBigQuery:true
java.lang.IllegalArgumentException: BigQueryIO.Write needs a GCS temp location
to store temp files.
0008 query:8; exportSummaryToBigQuery:true
0009 query:9; exportSummaryToBigQuery:true; numEvents:10000
at
com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)
0010 query:10; exportSummaryToBigQuery:true
at
org.apache.beam.sdk.io.gcp.bigquery.BatchLoads.validate(BatchLoads.java:194)
0011 query:11; exportSummaryToBigQuery:true
0012 query:12; exportSummaryToBigQuery:true
at
org.apache.beam.sdk.Pipeline$ValidateVisitor.enterCompositeTransform(Pipeline.java:640)
Performance:
Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results
(Baseline)
0000 4.5 22207.4 100000
0001 2.8 36350.4 92000
0002 1.0 105152.5 351
0003 2.3 43668.1 454
at
org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:656)
0004 1.3 7473.8 40
0005 4.1 24685.3 12
at
org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:660)
0006 0.7 13459.0 401
0007 9.6 10457.0 1
at
org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:660)
0008 1.2 85324.2 6000
0009 0.6 15772.9 298
at
org.apache.beam.sdk.runners.TransformHierarchy$Node.access$600(TransformHierarchy.java:311)
0010 10.2 9771.4 1
at
org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:245)
0011 2.5 39666.8 1919
at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:458)
0012 1.4 71225.1 1919
at org.apache.beam.sdk.Pipeline.validate(Pipeline.java:575)
==========================================================================================
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:310)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:297)
at org.apache.beam.sdk.nexmark.Main.savePerfsToBigQuery(Main.java:182)
at org.apache.beam.sdk.nexmark.Main.runAll(Main.java:102)
at org.apache.beam.sdk.nexmark.Main.main(Main.java:395)
> Task :beam-sdks-java-nexmark:run FAILED
:beam-sdks-java-nexmark:run (Thread[Task worker for ':' Thread 8,5,main])
completed. Took 47.954 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':beam-sdks-java-nexmark:run'.
> Process 'command '/usr/local/asfpackages/java/jdk1.8.0_152/bin/java''
> finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to
get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 5.0.
See
https://docs.gradle.org/4.8/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 3m 23s
59 actionable tasks: 55 executed, 4 from cache
Publishing build scan...
https://gradle.com/s/6ppz2mwefbntw
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Not sending mail to unregistered user [email protected]
