See
<https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Direct/190/display/redirect?page=changes>
Changes:
[yifanzou] [BEAM-4963] update org.apache.httpcomponents
[yifanzou] [BEAM-4906] upgrade org.apache.derby
[apilloud] [BEAM-4761] Run Nexmark in thread pool
[aaltay] Automate "Run validation tests" step (#6081)
------------------------------------------
[...truncated 673.03 KB...]
0000 24.6 4065.0 92000
0001 20.4 4895.2 92000
0002 14.1 7077.6 351
0003 *** not run ***
0004 *** not run ***
0005 *** not run ***
0006 *** not run ***
0007 *** not run ***
0008 *** not run ***
0009 *** not run ***
0010 *** not run ***
0011 *** not run ***
0012 *** not run ***
==========================================================================================
2018-08-04T00:14:36.864Z Generating 100000 events in batch mode
Aug 04, 2018 12:14:36 AM
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: SQL:
SELECT `P`.`name`, `P`.`city`, `P`.`state`, `A`.`id`
FROM `beam`.`Auction` AS `A`
INNER JOIN `beam`.`Person` AS `P` ON `A`.`seller` = `P`.`id`
WHERE `A`.`category` = 10 AND (`P`.`state` = 'OR' OR `P`.`state` = 'ID' OR
`P`.`state` = 'CA')
Aug 04, 2018 12:14:36 AM
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: SQLPlan>
LogicalProject(name=[$11], city=[$14], state=[$15], id=[$0])
LogicalFilter(condition=[AND(=($8, 10), OR(=($15, 'OR'), =($15, 'ID'), =($15,
'CA')))])
LogicalJoin(condition=[=($7, $10)], joinType=[inner])
BeamIOSourceRel(table=[[beam, Auction]])
BeamIOSourceRel(table=[[beam, Person]])
Aug 04, 2018 12:14:37 AM
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: BEAMPlan>
BeamCalcRel(expr#0..17=[{inputs}], name=[$t11], city=[$t14], state=[$t15],
id=[$t0])
BeamJoinRel(condition=[=($7, $10)], joinType=[inner])
BeamCalcRel(expr#0..9=[{inputs}], expr#10=[10], expr#11=[=($t8, $t10)],
proj#0..9=[{exprs}], $condition=[$t11])
BeamIOSourceRel(table=[[beam, Auction]])
BeamCalcRel(expr#0..7=[{inputs}], expr#8=['OR'], expr#9=[=($t5, $t8)],
expr#10=['ID'], expr#11=[=($t5, $t10)], expr#12=['CA'], expr#13=[=($t5, $t12)],
expr#14=[OR($t9, $t11, $t13)], proj#0..7=[{exprs}], $condition=[$t14])
BeamIOSourceRel(table=[[beam, Person]])
2018-08-04T00:14:51.883Z Waiting for main pipeline to 'finish'
2018-08-04T00:14:51.883Z DONE SqlQuery3
2018-08-04T00:14:51.884Z Running query:4; exportSummaryToBigQuery:true;
numEvents:10000
2018-08-04T00:14:51.884Z skipping since configuration is not implemented
2018-08-04T00:14:51.884Z Running query:5; exportSummaryToBigQuery:true
==========================================================================================
Run started 2018-08-04T00:13:31.860Z and ran for PT80.024S
Default configuration:
{"debug":true,"query":0,"sourceType":"DIRECT","sinkType":"DEVNULL","exportSummaryToBigQuery":false,"pubSubMode":"COMBINED","numEvents":100000,"numEventGenerators":100,"rateShape":"SINE","firstEventRate":10000,"nextEventRate":10000,"rateUnit":"PER_SECOND","ratePeriodSec":600,"preloadSeconds":0,"streamTimeout":240,"isRateLimited":false,"useWallclockEventTime":false,"avgPersonByteSize":200,"avgAuctionByteSize":500,"avgBidByteSize":100,"hotAuctionRatio":2,"hotSellersRatio":4,"hotBiddersRatio":4,"windowSizeSec":10,"windowPeriodSec":5,"watermarkHoldbackSec":0,"numInFlightAuctions":100,"numActivePeople":1000,"coderStrategy":"HAND","cpuDelayMs":0,"diskBusyBytes":0,"auctionSkip":123,"fanout":5,"maxAuctionsWaitingTime":600,"occasionalDelaySec":3,"probDelayedEvent":0.1,"maxLogEvents":100000,"usePubsubPublishTime":false,"outOfOrderGroupSize":1}
Configurations:
Conf Description
0000 query:0; exportSummaryToBigQuery:true
0001 query:1; exportSummaryToBigQuery:true
0002 query:2; exportSummaryToBigQuery:true
0003 query:3; exportSummaryToBigQuery:true
0004 query:4; exportSummaryToBigQuery:true; numEvents:10000
0005 query:5; exportSummaryToBigQuery:true
0006 query:6; exportSummaryToBigQuery:true; numEvents:10000
0007 query:7; exportSummaryToBigQuery:true
0008 query:8; exportSummaryToBigQuery:true
0009 query:9; exportSummaryToBigQuery:true; numEvents:10000
0010 query:10; exportSummaryToBigQuery:true
0011 query:11; exportSummaryToBigQuery:true
0012 query:12; exportSummaryToBigQuery:true
Performance:
Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results
(Baseline)
0000 24.6 4065.0 92000
0001 20.4 4895.2 92000
0002 14.1 7077.6 351
0003 14.5 6917.1 580
0004 *** not run ***
0005 *** not run ***
0006 *** not run ***
0007 *** not run ***
0008 *** not run ***
0009 *** not run ***
0010 *** not run ***
0011 *** not run ***
0012 *** not run ***
==========================================================================================
==========================================================================================
Run started 2018-08-04T00:13:31.860Z and ran for PT80.025S
Default configuration:
{"debug":true,"query":0,"sourceType":"DIRECT","sinkType":"DEVNULL","exportSummaryToBigQuery":false,"pubSubMode":"COMBINED","numEvents":100000,"numEventGenerators":100,"rateShape":"SINE","firstEventRate":10000,"nextEventRate":10000,"rateUnit":"PER_SECOND","ratePeriodSec":600,"preloadSeconds":0,"streamTimeout":240,"isRateLimited":false,"useWallclockEventTime":false,"avgPersonByteSize":200,"avgAuctionByteSize":500,"avgBidByteSize":100,"hotAuctionRatio":2,"hotSellersRatio":4,"hotBiddersRatio":4,"windowSizeSec":10,"windowPeriodSec":5,"watermarkHoldbackSec":0,"numInFlightAuctions":100,"numActivePeople":1000,"coderStrategy":"HAND","cpuDelayMs":0,"diskBusyBytes":0,"auctionSkip":123,"fanout":5,"maxAuctionsWaitingTime":600,"occasionalDelaySec":3,"probDelayedEvent":0.1,"maxLogEvents":100000,"usePubsubPublishTime":false,"outOfOrderGroupSize":1}
Configurations:
Conf Description
0000 query:0; exportSummaryToBigQuery:true
0001 query:1; exportSummaryToBigQuery:true
0002 query:2; exportSummaryToBigQuery:true
0003 query:3; exportSummaryToBigQuery:true
0004 query:4; exportSummaryToBigQuery:true; numEvents:10000
0005 query:5; exportSummaryToBigQuery:true
0006 query:6; exportSummaryToBigQuery:true; numEvents:10000
0007 query:7; exportSummaryToBigQuery:true
0008 query:8; exportSummaryToBigQuery:true
0009 query:9; exportSummaryToBigQuery:true; numEvents:10000
0010 query:10; exportSummaryToBigQuery:true
0011 query:11; exportSummaryToBigQuery:true
0012 query:12; exportSummaryToBigQuery:true
Performance:
Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results
(Baseline)
0000 24.6 4065.0 92000
Exception in thread "main" java.lang.NullPointerException
0001 20.4 4895.2 92000
0002 14.1 7077.6 351
0003 14.5 6917.1 580
at org.apache.beam.sdk.nexmark.Main.runAll(Main.java:150)
0004 *** not run ***
at org.apache.beam.sdk.nexmark.Main.main(Main.java:475)
0005 *** not run ***
0006 *** not run ***
0007 *** not run ***
0008 *** not run ***
0009 *** not run ***
0010 *** not run ***
0011 *** not run ***
0012 *** not run ***
==========================================================================================
2018-08-04T00:14:51.888Z Generating 100000 events in batch mode
Aug 04, 2018 12:14:51 AM
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: SQL:
SELECT `AuctionBids`.`auction`, `AuctionBids`.`num`
FROM (SELECT `B1`.`auction`, COUNT(*) AS `num`, HOP_START(`B1`.`dateTime`,
INTERVAL '5' SECOND, INTERVAL '10' SECOND) AS `starttime`
FROM `beam`.`Bid` AS `B1`
GROUP BY `B1`.`auction`, HOP(`B1`.`dateTime`, INTERVAL '5' SECOND, INTERVAL
'10' SECOND)) AS `AuctionBids`
INNER JOIN (SELECT MAX(`CountBids`.`num`) AS `maxnum`, `CountBids`.`starttime`
FROM (SELECT COUNT(*) AS `num`, HOP_START(`B2`.`dateTime`, INTERVAL '5' SECOND,
INTERVAL '10' SECOND) AS `starttime`
FROM `beam`.`Bid` AS `B2`
GROUP BY `B2`.`auction`, HOP(`B2`.`dateTime`, INTERVAL '5' SECOND, INTERVAL
'10' SECOND)) AS `CountBids`
GROUP BY `CountBids`.`starttime`) AS `MaxBids` ON `AuctionBids`.`starttime` =
`MaxBids`.`starttime` AND `AuctionBids`.`num` >= `MaxBids`.`maxnum`
Aug 04, 2018 12:14:51 AM
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: SQLPlan>
LogicalProject(auction=[$0], num=[$1])
LogicalJoin(condition=[AND(=($2, $4), >=($1, $3))], joinType=[inner])
LogicalProject(auction=[$0], num=[$2], starttime=[$1])
LogicalAggregate(group=[{0, 1}], num=[COUNT()])
LogicalProject(auction=[$0], $f1=[HOP($3, 5000, 10000)])
BeamIOSourceRel(table=[[beam, Bid]])
LogicalProject(maxnum=[$1], starttime=[$0])
LogicalAggregate(group=[{0}], maxnum=[MAX($1)])
LogicalProject(starttime=[$1], num=[$0])
LogicalProject(num=[$2], starttime=[$1])
LogicalAggregate(group=[{0, 1}], num=[COUNT()])
LogicalProject(auction=[$0], $f1=[HOP($3, 5000, 10000)])
BeamIOSourceRel(table=[[beam, Bid]])
Aug 04, 2018 12:14:52 AM
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: BEAMPlan>
BeamCalcRel(expr#0..4=[{inputs}], proj#0..1=[{exprs}])
BeamJoinRel(condition=[AND(=($2, $4), >=($1, $3))], joinType=[inner])
BeamCalcRel(expr#0..2=[{inputs}], auction=[$t0], num=[$t2], starttime=[$t1])
BeamAggregationRel(group=[{0, 1}], num=[COUNT()])
BeamCalcRel(expr#0..4=[{inputs}], expr#5=[5000], expr#6=[10000],
expr#7=[HOP($t3, $t5, $t6)], auction=[$t0], $f1=[$t7])
BeamIOSourceRel(table=[[beam, Bid]])
BeamCalcRel(expr#0..1=[{inputs}], maxnum=[$t1], starttime=[$t0])
BeamAggregationRel(group=[{1}], maxnum=[MAX($0)])
BeamCalcRel(expr#0..2=[{inputs}], num=[$t2], starttime=[$t1])
BeamAggregationRel(group=[{0, 1}], num=[COUNT()])
BeamCalcRel(expr#0..4=[{inputs}], expr#5=[5000], expr#6=[10000],
expr#7=[HOP($t3, $t5, $t6)], auction=[$t0], $f1=[$t7])
BeamIOSourceRel(table=[[beam, Bid]])
2018-08-04T00:15:18.822Z Waiting for main pipeline to 'finish'
2018-08-04T00:15:18.822Z DONE SqlQuery5
2018-08-04T00:15:18.823Z Running query:6; exportSummaryToBigQuery:true;
numEvents:10000
2018-08-04T00:15:18.823Z skipping since configuration is not implemented
2018-08-04T00:15:18.823Z Running query:7; exportSummaryToBigQuery:true
2018-08-04T00:15:18.827Z Generating 100000 events in batch mode
Aug 04, 2018 12:15:18 AM
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: SQL:
SELECT `B`.`auction`, `B`.`price`, `B`.`bidder`, `B`.`dateTime`, `B`.`extra`
FROM (SELECT `B`.`auction`, `B`.`price`, `B`.`bidder`, `B`.`dateTime`,
`B`.`extra`, TUMBLE_START(`B`.`dateTime`, INTERVAL '10' SECOND) AS `starttime`
FROM `beam`.`Bid` AS `B`
GROUP BY `B`.`auction`, `B`.`price`, `B`.`bidder`, `B`.`dateTime`, `B`.`extra`,
TUMBLE(`B`.`dateTime`, INTERVAL '10' SECOND)) AS `B`
INNER JOIN (SELECT MAX(`B1`.`price`) AS `maxprice`,
TUMBLE_START(`B1`.`dateTime`, INTERVAL '10' SECOND) AS `starttime`
FROM `beam`.`Bid` AS `B1`
GROUP BY TUMBLE(`B1`.`dateTime`, INTERVAL '10' SECOND)) AS `B1` ON
`B`.`starttime` = `B1`.`starttime` AND `B`.`price` = `B1`.`maxprice`
Aug 04, 2018 12:15:18 AM
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: SQLPlan>
LogicalProject(auction=[$0], price=[$1], bidder=[$2], dateTime=[$3], extra=[$4])
LogicalJoin(condition=[AND(=($5, $7), =($1, $6))], joinType=[inner])
LogicalProject(auction=[$0], price=[$1], bidder=[$2], dateTime=[$3],
extra=[$4], starttime=[$5])
LogicalAggregate(group=[{0, 1, 2, 3, 4, 5}])
LogicalProject(auction=[$0], price=[$2], bidder=[$1], dateTime=[$3],
extra=[$4], $f5=[TUMBLE($3, 10000)])
BeamIOSourceRel(table=[[beam, Bid]])
LogicalProject(maxprice=[$1], starttime=[$0])
LogicalAggregate(group=[{0}], maxprice=[MAX($1)])
LogicalProject($f0=[TUMBLE($3, 10000)], price=[$2])
BeamIOSourceRel(table=[[beam, Bid]])
Aug 04, 2018 12:15:18 AM
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: BEAMPlan>
BeamCalcRel(expr#0..7=[{inputs}], proj#0..4=[{exprs}])
BeamJoinRel(condition=[AND(=($5, $7), =($1, $6))], joinType=[inner])
BeamCalcRel(expr#0..5=[{inputs}], proj#0..5=[{exprs}])
BeamAggregationRel(group=[{0, 1, 2, 3, 4, 5}])
BeamCalcRel(expr#0..4=[{inputs}], expr#5=[10000], expr#6=[TUMBLE($t3,
$t5)], auction=[$t0], price=[$t2], bidder=[$t1], dateTime=[$t3], extra=[$t4],
$f5=[$t6])
BeamIOSourceRel(table=[[beam, Bid]])
BeamCalcRel(expr#0..1=[{inputs}], maxprice=[$t1], starttime=[$t0])
BeamAggregationRel(group=[{0}], maxprice=[MAX($1)])
BeamCalcRel(expr#0..4=[{inputs}], expr#5=[10000], expr#6=[TUMBLE($t3,
$t5)], $f0=[$t6], price=[$t2])
BeamIOSourceRel(table=[[beam, Bid]])
2018-08-04T00:16:08.047Z Waiting for main pipeline to 'finish'
2018-08-04T00:16:08.047Z DONE SqlQuery7
2018-08-04T00:16:08.048Z Running query:8; exportSummaryToBigQuery:true
2018-08-04T00:16:08.048Z skipping since configuration is not implemented
2018-08-04T00:16:08.048Z Running query:9; exportSummaryToBigQuery:true;
numEvents:10000
2018-08-04T00:16:08.048Z skipping since configuration is not implemented
2018-08-04T00:16:08.048Z Running query:10; exportSummaryToBigQuery:true
2018-08-04T00:16:08.048Z skipping since configuration is not implemented
2018-08-04T00:16:08.048Z Running query:11; exportSummaryToBigQuery:true
2018-08-04T00:16:08.049Z skipping since configuration is not implemented
2018-08-04T00:16:08.049Z Running query:12; exportSummaryToBigQuery:true
2018-08-04T00:16:08.049Z skipping since configuration is not implemented
> Task :beam-sdks-java-nexmark:run FAILED
:beam-sdks-java-nexmark:run (Thread[Task worker for ':' Thread 7,5,main])
completed. Took 2 mins 37.573 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':beam-sdks-java-nexmark:run'.
> Process 'command '/usr/local/asfpackages/java/jdk1.8.0_152/bin/java''
> finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to
get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 5.0.
See
https://docs.gradle.org/4.8/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 2m 51s
62 actionable tasks: 2 executed, 60 up-to-date
Publishing build scan...
https://gradle.com/s/toi7t5qbijzxg
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
