See
<https://ci-beam.apache.org/job/beam_PostCommit_Java_Nexmark_Spark/7019/display/redirect?page=changes>
Changes:
[heejong] [BEAM-12390] clickhouse test fails when test resource can't be read by
[Ismaël Mejía] [BEAM-12241] Update vendored bytebuddy to version 1.11.0
[noreply] [BEAM-10144] update pipeline options snippets (#14738)
[Kyle Weaver] Update Dataflow Python containers.
[Kyle Weaver] [BEAM-12394] Include UDF provider jar in Hadoop variant test.
[Kyle Weaver] [BEAM-11738] Fail JavaUdfLoaderTest instead of ignoring if system
------------------------------------------
[...truncated 107.59 KB...]
0004 query:AVERAGE_PRICE_FOR_CATEGORY; exportSummaryToBigQuery:true;
numEvents:10000; streamTimeout:60
0005 query:HOT_ITEMS; exportSummaryToBigQuery:true; streamTimeout:60
0006 query:AVERAGE_SELLING_PRICE_BY_SELLER; exportSummaryToBigQuery:true;
numEvents:10000; streamTimeout:60
0007 query:HIGHEST_BID; exportSummaryToBigQuery:true; streamTimeout:60
0008 query:MONITOR_NEW_USERS; exportSummaryToBigQuery:true; streamTimeout:60
0009 query:WINNING_BIDS; exportSummaryToBigQuery:true; numEvents:10000;
streamTimeout:60
0010 query:LOG_TO_SHARDED_FILES; exportSummaryToBigQuery:true;
streamTimeout:60
0011 query:USER_SESSIONS; exportSummaryToBigQuery:true; streamTimeout:60
0012 query:PROCESSING_TIME_WINDOWS; exportSummaryToBigQuery:true;
streamTimeout:60
0013 query:PORTABILITY_BATCH; exportSummaryToBigQuery:true; streamTimeout:60
0014 query:BOUNDED_SIDE_INPUT_JOIN; exportSummaryToBigQuery:true;
streamTimeout:60
0015 query:SESSION_SIDE_INPUT_JOIN; exportSummaryToBigQuery:true;
streamTimeout:60
Performance:
Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results
(Baseline)
0000 1.4 72463.8 100000
0001 0.4 222717.1 92000
0002 0.3 362318.8 351
0003 1.8 54466.2 580
0004 1.0 10111.2 40
0005 1.5 68352.7 12
0006 0.7 15337.4 103
0007 1.4 73964.5 1
0008 1.3 78926.6 6000
0009 0.5 18832.4 298
0010 0.7 139082.1 1
0011 1.3 79113.9 1919
0012 0.9 113250.3 1919
0013 0.7 144717.8 100000
0014 *** not run ***
0015 *** not run ***
==========================================================================================
2021-05-25T00:37:04.693Z Generating 100000 events in batch mode
2021-05-25T00:37:05.247Z Waiting for main pipeline to 'finish'
2021-05-25T00:37:05.248Z DONE BoundedSideInputJoin
2021-05-25T00:37:05.249Z Running query:SESSION_SIDE_INPUT_JOIN;
exportSummaryToBigQuery:true; streamTimeout:60
==========================================================================================
Run started 2021-05-25T00:36:43.544Z and ran for PT21.705S
Default configuration:
{"debug":true,"query":null,"sourceType":"DIRECT","generateEventFilePathPrefix":null,"sinkType":"DEVNULL","exportSummaryToBigQuery":false,"pubSubMode":"COMBINED","pubsubMessageSerializationMethod":"CODER","sideInputType":"DIRECT","sideInputRowCount":500,"sideInputNumShards":3,"sideInputUrl":null,"sessionGap":600000,"numEvents":100000,"numEventGenerators":100,"rateShape":"SINE","firstEventRate":10000,"nextEventRate":10000,"rateUnit":"PER_SECOND","ratePeriodSec":600,"preloadSeconds":0,"streamTimeout":240,"isRateLimited":false,"useWallclockEventTime":false,"avgPersonByteSize":200,"avgAuctionByteSize":500,"avgBidByteSize":100,"hotAuctionRatio":2,"hotSellersRatio":4,"hotBiddersRatio":4,"windowSizeSec":10,"windowPeriodSec":5,"watermarkHoldbackSec":0,"numInFlightAuctions":100,"numActivePeople":1000,"coderStrategy":"HAND","cpuDelayMs":0,"diskBusyBytes":0,"auctionSkip":123,"fanout":5,"maxAuctionsWaitingTime":600,"occasionalDelaySec":3,"probDelayedEvent":0.1,"maxLogEvents":100000,"usePubsubPublishTime":false,"outOfOrderGroupSize":1,"numKeyBuckets":20000,"pardoCPUFactor":1.0}
Configurations:
Conf Description
0000 query:PASSTHROUGH; exportSummaryToBigQuery:true; streamTimeout:60
0001 query:CURRENCY_CONVERSION; exportSummaryToBigQuery:true;
streamTimeout:60
0002 query:SELECTION; exportSummaryToBigQuery:true; streamTimeout:60
0003 query:LOCAL_ITEM_SUGGESTION; exportSummaryToBigQuery:true;
streamTimeout:60
0004 query:AVERAGE_PRICE_FOR_CATEGORY; exportSummaryToBigQuery:true;
numEvents:10000; streamTimeout:60
0005 query:HOT_ITEMS; exportSummaryToBigQuery:true; streamTimeout:60
0006 query:AVERAGE_SELLING_PRICE_BY_SELLER; exportSummaryToBigQuery:true;
numEvents:10000; streamTimeout:60
0007 query:HIGHEST_BID; exportSummaryToBigQuery:true; streamTimeout:60
0008 query:MONITOR_NEW_USERS; exportSummaryToBigQuery:true; streamTimeout:60
0009 query:WINNING_BIDS; exportSummaryToBigQuery:true; numEvents:10000;
streamTimeout:60
0010 query:LOG_TO_SHARDED_FILES; exportSummaryToBigQuery:true;
streamTimeout:60
0011 query:USER_SESSIONS; exportSummaryToBigQuery:true; streamTimeout:60
0012 query:PROCESSING_TIME_WINDOWS; exportSummaryToBigQuery:true;
streamTimeout:60
0013 query:PORTABILITY_BATCH; exportSummaryToBigQuery:true; streamTimeout:60
0014 query:BOUNDED_SIDE_INPUT_JOIN; exportSummaryToBigQuery:true;
streamTimeout:60
0015 query:SESSION_SIDE_INPUT_JOIN; exportSummaryToBigQuery:true;
streamTimeout:60
Performance:
Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results
(Baseline)
0000 1.4 72463.8 100000
0001 0.4 222717.1 92000
0002 0.3 362318.8 351
0003 1.8 54466.2 580
0004 1.0 10111.2 40
0005 1.5 68352.7 12
0006 0.7 15337.4 103
0007 1.4 73964.5 1
0008 1.3 78926.6 6000
0009 0.5 18832.4 298
0010 0.7 139082.1 1
0011 1.3 79113.9 1919
0012 0.9 113250.3 1919
0013 0.7 144717.8 100000
0014 0.3 315457.4 92000
0015 *** not run ***
==========================================================================================
2021-05-25T00:37:05.251Z Generating 100000 events in batch mode
2021-05-25T00:37:06.313Z Waiting for main pipeline to 'finish'
2021-05-25T00:37:06.313Z DONE SessionSideInputJoin
==========================================================================================
Run started 2021-05-25T00:36:43.544Z and ran for PT22.770S
Default configuration:
{"debug":true,"query":null,"sourceType":"DIRECT","generateEventFilePathPrefix":null,"sinkType":"DEVNULL","exportSummaryToBigQuery":false,"pubSubMode":"COMBINED","pubsubMessageSerializationMethod":"CODER","sideInputType":"DIRECT","sideInputRowCount":500,"sideInputNumShards":3,"sideInputUrl":null,"sessionGap":600000,"numEvents":100000,"numEventGenerators":100,"rateShape":"SINE","firstEventRate":10000,"nextEventRate":10000,"rateUnit":"PER_SECOND","ratePeriodSec":600,"preloadSeconds":0,"streamTimeout":240,"isRateLimited":false,"useWallclockEventTime":false,"avgPersonByteSize":200,"avgAuctionByteSize":500,"avgBidByteSize":100,"hotAuctionRatio":2,"hotSellersRatio":4,"hotBiddersRatio":4,"windowSizeSec":10,"windowPeriodSec":5,"watermarkHoldbackSec":0,"numInFlightAuctions":100,"numActivePeople":1000,"coderStrategy":"HAND","cpuDelayMs":0,"diskBusyBytes":0,"auctionSkip":123,"fanout":5,"maxAuctionsWaitingTime":600,"occasionalDelaySec":3,"probDelayedEvent":0.1,"maxLogEvents":100000,"usePubsubPublishTime":false,"outOfOrderGroupSize":1,"numKeyBuckets":20000,"pardoCPUFactor":1.0}
Configurations:
Conf Description
0000 query:PASSTHROUGH; exportSummaryToBigQuery:true; streamTimeout:60
0001 query:CURRENCY_CONVERSION; exportSummaryToBigQuery:true;
streamTimeout:60
0002 query:SELECTION; exportSummaryToBigQuery:true; streamTimeout:60
0003 query:LOCAL_ITEM_SUGGESTION; exportSummaryToBigQuery:true;
streamTimeout:60
0004 query:AVERAGE_PRICE_FOR_CATEGORY; exportSummaryToBigQuery:true;
numEvents:10000; streamTimeout:60
0005 query:HOT_ITEMS; exportSummaryToBigQuery:true; streamTimeout:60
0006 query:AVERAGE_SELLING_PRICE_BY_SELLER; exportSummaryToBigQuery:true;
numEvents:10000; streamTimeout:60
0007 query:HIGHEST_BID; exportSummaryToBigQuery:true; streamTimeout:60
0008 query:MONITOR_NEW_USERS; exportSummaryToBigQuery:true; streamTimeout:60
0009 query:WINNING_BIDS; exportSummaryToBigQuery:true; numEvents:10000;
streamTimeout:60
0010 query:LOG_TO_SHARDED_FILES; exportSummaryToBigQuery:true;
streamTimeout:60
0011 query:USER_SESSIONS; exportSummaryToBigQuery:true; streamTimeout:60
0012 query:PROCESSING_TIME_WINDOWS; exportSummaryToBigQuery:true;
streamTimeout:60
0013 query:PORTABILITY_BATCH; exportSummaryToBigQuery:true; streamTimeout:60
0014 query:BOUNDED_SIDE_INPUT_JOIN; exportSummaryToBigQuery:true;
streamTimeout:60
0015 query:SESSION_SIDE_INPUT_JOIN; exportSummaryToBigQuery:true;
streamTimeout:60
Performance:
Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results
(Baseline)
0000 1.4 72463.8 100000
0001 0.4 222717.1 92000
0002 0.3 362318.8 351
0003 1.8 54466.2 580
0004 1.0 10111.2 40
0005 1.5 68352.7 12
0006 0.7 15337.4 103
0007 1.4 73964.5 1
0008 1.3 78926.6 6000
0009 0.5 18832.4 298
0010 0.7 139082.1 1
0011 1.3 79113.9 1919
0012 0.9 113250.3 1919
0013 0.7 144717.8 100000
0014 0.3 315457.4 92000
0015 0.8 124069.5 92000
==========================================================================================
==========================================================================================
Run started 2021-05-25T00:36:43.544Z and ran for PT128.746S
Default configuration:
{"debug":true,"query":null,"sourceType":"DIRECT","generateEventFilePathPrefix":null,"sinkType":"DEVNULL","exportSummaryToBigQuery":false,"pubSubMode":"COMBINED","pubsubMessageSerializationMethod":"CODER","sideInputType":"DIRECT","sideInputRowCount":500,"sideInputNumShards":3,"sideInputUrl":null,"sessionGap":600000,"numEvents":100000,"numEventGenerators":100,"rateShape":"SINE","firstEventRate":10000,"nextEventRate":10000,"rateUnit":"PER_SECOND","ratePeriodSec":600,"preloadSeconds":0,"streamTimeout":240,"isRateLimited":false,"useWallclockEventTime":false,"avgPersonByteSize":200,"avgAuctionByteSize":500,"avgBidByteSize":100,"hotAuctionRatio":2,"hotSellersRatio":4,"hotBiddersRatio":4,"windowSizeSec":10,"windowPeriodSec":5,"watermarkHoldbackSec":0,"numInFlightAuctions":100,"numActivePeople":1000,"coderStrategy":"HAND","cpuDelayMs":0,"diskBusyBytes":0,"auctionSkip":123,"fanout":5,"maxAuctionsWaitingTime":600,"occasionalDelaySec":3,"probDelayedEvent":0.1,"maxLogEvents":100000,"usePubsubPublishTime":false,"outOfOrderGroupSize":1,"numKeyBuckets":20000,"pardoCPUFactor":1.0}
Configurations:
Conf Description
0000 query:PASSTHROUGH; exportSummaryToBigQuery:true; streamTimeout:60
0001 query:CURRENCY_CONVERSION; exportSummaryToBigQuery:true;
streamTimeout:60
Exception in thread "main" 0002 query:SELECTION;
exportSummaryToBigQuery:true; streamTimeout:60
0003 query:LOCAL_ITEM_SUGGESTION; exportSummaryToBigQuery:true;
streamTimeout:60
0004 query:AVERAGE_PRICE_FOR_CATEGORY; exportSummaryToBigQuery:true;
numEvents:10000; streamTimeout:60
0005 query:HOT_ITEMS; exportSummaryToBigQuery:true; streamTimeout:60
0006 query:AVERAGE_SELLING_PRICE_BY_SELLER; exportSummaryToBigQuery:true;
numEvents:10000; streamTimeout:60
0007 query:HIGHEST_BID; exportSummaryToBigQuery:true; streamTimeout:60
0008 query:MONITOR_NEW_USERS; exportSummaryToBigQuery:true; streamTimeout:60
0009 query:WINNING_BIDS; exportSummaryToBigQuery:true; numEvents:10000;
streamTimeout:60
0010 query:LOG_TO_SHARDED_FILES; exportSummaryToBigQuery:true;
streamTimeout:60
0011 query:USER_SESSIONS; exportSummaryToBigQuery:true; streamTimeout:60
0012 query:PROCESSING_TIME_WINDOWS; exportSummaryToBigQuery:true;
streamTimeout:60
com.google.cloud.bigquery.BigQueryException: Read timed out
0013 query:PORTABILITY_BATCH; exportSummaryToBigQuery:true; streamTimeout:60
0014 query:BOUNDED_SIDE_INPUT_JOIN; exportSummaryToBigQuery:true;
streamTimeout:60
0015 query:SESSION_SIDE_INPUT_JOIN; exportSummaryToBigQuery:true;
streamTimeout:60
at
com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.translate(HttpBigQueryRpc.java:115)
Performance:
Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results
(Baseline)
at
com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.getTable(HttpBigQueryRpc.java:286)
0000 1.4 72463.8 100000
0001 0.4 222717.1 92000
0002 0.3 362318.8 351
0003 1.8 54466.2 580
at com.google.cloud.bigquery.BigQueryImpl$17.call(BigQueryImpl.java:717)
0004 1.0 10111.2 40
0005 1.5 68352.7 12
0006 0.7 15337.4 103
0007 1.4 73964.5 1
0008 1.3 78926.6 6000
0009 0.5 18832.4 298
at com.google.cloud.bigquery.BigQueryImpl$17.call(BigQueryImpl.java:714)
0010 0.7 139082.1 1
0011 1.3 79113.9 1919
0012 0.9 113250.3 1919
0013 0.7 144717.8 100000
0014 0.3 315457.4 92000
0015 0.8 124069.5 92000
==========================================================================================
at
com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105)
at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
at
com.google.cloud.bigquery.BigQueryImpl.getTable(BigQueryImpl.java:713)
at
org.apache.beam.sdk.testutils.publishing.BigQueryClient.createTableIfNotExists(BigQueryClient.java:145)
at
org.apache.beam.sdk.testutils.publishing.BigQueryClient.insertRow(BigQueryClient.java:88)
at
org.apache.beam.sdk.testutils.publishing.BigQueryResultsPublisher.publish(BigQueryResultsPublisher.java:47)
at org.apache.beam.sdk.nexmark.Main.savePerfsToBigQuery(Main.java:203)
at org.apache.beam.sdk.nexmark.Main.runAll(Main.java:164)
at org.apache.beam.sdk.nexmark.Main.main(Main.java:495)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at
sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:457)
at
sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:68)
at
sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1332)
at sun.security.ssl.SSLSocketImpl.access$300(SSLSocketImpl.java:73)
at
sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:948)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:735)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678)
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1593)
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
at
java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
at
sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:352)
at
com.google.api.client.http.javanet.NetHttpResponse.<init>(NetHttpResponse.java:36)
at
com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:149)
at
com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84)
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012)
at
com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:514)
at
com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:455)
at
com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:565)
at
com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.getTable(HttpBigQueryRpc.java:284)
... 12 more
21/05/25 00:38:52 DEBUG org.apache.hadoop.util.ShutdownHookManager: Completed
shutdown in 0.007 seconds; Timeouts: 0
21/05/25 00:38:52 DEBUG org.apache.hadoop.util.ShutdownHookManager:
ShutdownHookManger completed shutdown.
> Task :sdks:java:testing:nexmark:run FAILED
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:java:testing:nexmark:run'.
> Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with
> non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See
https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 3m 27s
108 actionable tasks: 72 executed, 36 from cache
Publishing build scan...
https://gradle.com/s/f723ljnkyh74u
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
