See
<https://ci-beam.apache.org/job/beam_PostCommit_Java_Nexmark_Flink/6686/display/redirect?page=changes>
Changes:
[Kyle Weaver] [BEAM-11579] Specify flink_job_server_jar instead of using
default.
------------------------------------------
[...truncated 27.96 MB...]
"PTRANSFORM":
"SessionSideInputJoin/SessionSideInputJoin.Snoop/ParMultiDo(Anonymous)"
}
}, {
"urn": "beam:metric:user:sum_int64:v1",
"type": "beam:metrics:sum_int64:v1",
"payload": "4M4F",
"labels": {
"NAMESPACE": "SessionSideInputJoin",
"NAME": "bids",
"PTRANSFORM":
"SessionSideInputJoin/SessionSideInputJoin.Snoop/ParMultiDo(Anonymous)"
}
}, {
"urn": "beam:metric:user:sum_int64:v1",
"type": "beam:metrics:sum_int64:v1",
"payload": "4M4F",
"labels": {
"NAMESPACE": "SessionSideInputJoin",
"NAME": "records",
"PTRANSFORM": "SessionSideInputJoin.Format/ParMultiDo(Anonymous)"
}
}]
}
}
Jan 08, 2021 6:10:11 PM org.apache.flink.runtime.taskexecutor.TaskExecutor
closeJobManagerConnection
INFO: Close JobManager connection for job f944c25020aa7105c877d52c515004ef.
Jan 08, 2021 6:10:11 PM org.apache.flink.runtime.taskexecutor.TaskExecutor
closeJobManagerConnection
INFO: Close JobManager connection for job f944c25020aa7105c877d52c515004ef.
==========================================================================================
Run started 2021-01-08T18:09:49.546Z and ran for PT22.160S
Default configuration:
{"debug":true,"query":null,"sourceType":"DIRECT","generateEventFilePathPrefix":null,"sinkType":"DEVNULL","exportSummaryToBigQuery":false,"pubSubMode":"COMBINED","pubsubMessageSerializationMethod":"CODER","sideInputType":"DIRECT","sideInputRowCount":500,"sideInputNumShards":3,"sideInputUrl":null,"sessionGap":{"standardDays":0,"standardHours":0,"standardMinutes":10,"standardSeconds":600,"millis":600000},"numEvents":100000,"numEventGenerators":100,"rateShape":"SINE","firstEventRate":10000,"nextEventRate":10000,"rateUnit":"PER_SECOND","ratePeriodSec":600,"preloadSeconds":0,"streamTimeout":240,"isRateLimited":false,"useWallclockEventTime":false,"avgPersonByteSize":200,"avgAuctionByteSize":500,"avgBidByteSize":100,"hotAuctionRatio":2,"hotSellersRatio":4,"hotBiddersRatio":4,"windowSizeSec":10,"windowPeriodSec":5,"watermarkHoldbackSec":0,"numInFlightAuctions":100,"numActivePeople":1000,"coderStrategy":"HAND","cpuDelayMs":0,"diskBusyBytes":0,"auctionSkip":123,"fanout":5,"maxAuctionsWaitingTime":600,"occasionalDelaySec":3,"probDelayedEvent":0.1,"maxLogEvents":100000,"usePubsubPublishTime":false,"outOfOrderGroupSize":1}
Configurations:
Conf Description
0000 query:PASSTHROUGH; exportSummaryToBigQuery:true; streamTimeout:60
0001 query:CURRENCY_CONVERSION; exportSummaryToBigQuery:true;
streamTimeout:60
0002 query:SELECTION; exportSummaryToBigQuery:true; streamTimeout:60
0003 query:LOCAL_ITEM_SUGGESTION; exportSummaryToBigQuery:true;
streamTimeout:60
0004 query:AVERAGE_PRICE_FOR_CATEGORY; exportSummaryToBigQuery:true;
numEvents:10000; streamTimeout:60
0005 query:HOT_ITEMS; exportSummaryToBigQuery:true; streamTimeout:60
0006 query:AVERAGE_SELLING_PRICE_BY_SELLER; exportSummaryToBigQuery:true;
numEvents:10000; streamTimeout:60
0007 query:HIGHEST_BID; exportSummaryToBigQuery:true; streamTimeout:60
0008 query:MONITOR_NEW_USERS; exportSummaryToBigQuery:true; streamTimeout:60
0009 query:WINNING_BIDS; exportSummaryToBigQuery:true; numEvents:10000;
streamTimeout:60
0010 query:LOG_TO_SHARDED_FILES; exportSummaryToBigQuery:true;
streamTimeout:60
0011 query:USER_SESSIONS; exportSummaryToBigQuery:true; streamTimeout:60
0012 query:PROCESSING_TIME_WINDOWS; exportSummaryToBigQuery:true;
streamTimeout:60
0013 query:BOUNDED_SIDE_INPUT_JOIN; exportSummaryToBigQuery:true;
streamTimeout:60
0014 query:SESSION_SIDE_INPUT_JOIN; exportSummaryToBigQuery:true;
streamTimeout:60
Performance:
Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results
(Baseline)
0000 1.1 91074.7 100000
0001 0.2 628930.8 92000
0002 0.1 1136363.6 351
0003 1.0 97276.3 580
0004 0.8 12165.5 40
0005 0.8 122399.0 12
0006 0.6 16949.2 103
0007 0.5 182481.8 1
0008 0.5 200000.0 6000
0009 0.3 37735.8 298
0010 0.5 203666.0 1
0011 0.3 354609.9 1919
0012 0.2 540540.5 1919
0013 0.4 271739.1 92000
0014 0.5 201207.2 92000
==========================================================================================
Jan 08, 2021 6:10:11 PM org.apache.flink.runtime.taskexecutor.JobLeaderService
stop
INFO: Stop job leader service.
Jan 08, 2021 6:10:11 PM
org.apache.flink.runtime.state.TaskExecutorLocalStateStoresManager shutdown
INFO: Shutting down TaskExecutorLocalStateStoresManager.
Jan 08, 2021 6:10:11 PM org.apache.flink.runtime.io.disk.FileChannelManagerImpl
lambda$getFileCloser$0
INFO: FileChannelManager removed spill file directory
/tmp/flink-io-2946f6cb-1d8f-4d79-a60c-cb64bd03d8a2
Jan 08, 2021 6:10:11 PM
org.apache.flink.runtime.io.network.NettyShuffleEnvironment close
INFO: Shutting down the network environment and its components.
Jan 08, 2021 6:10:11 PM org.apache.flink.runtime.io.disk.FileChannelManagerImpl
lambda$getFileCloser$0
INFO: FileChannelManager removed spill file directory
/tmp/flink-netty-shuffle-67f08ebf-f5cc-4985-907c-e51f48245888
Jan 08, 2021 6:10:11 PM org.apache.flink.runtime.taskexecutor.KvStateService
shutdown
INFO: Shutting down the kvState service and its components.
Jan 08, 2021 6:10:11 PM org.apache.flink.runtime.taskexecutor.JobLeaderService
stop
INFO: Stop job leader service.
Jan 08, 2021 6:10:11 PM org.apache.flink.runtime.filecache.FileCache shutdown
INFO: removed file cache directory
/tmp/flink-dist-cache-ca815097-36bc-4fc4-b59f-bf4d3507f64f
Jan 08, 2021 6:10:11 PM org.apache.flink.runtime.taskexecutor.TaskExecutor
handleOnStopException
INFO: Stopped TaskExecutor akka://flink/user/taskmanager_28.
Jan 08, 2021 6:10:11 PM org.apache.flink.runtime.webmonitor.WebMonitorEndpoint
lambda$shutDownInternal$5
INFO: Removing cache directory /tmp/flink-web-ui
Jan 08, 2021 6:10:11 PM org.apache.flink.runtime.rest.RestServerEndpoint
lambda$closeAsync$1
INFO: Shut down complete.
Jan 08, 2021 6:10:11 PM
org.apache.flink.runtime.resourcemanager.ResourceManager deregisterApplication
INFO: Shut down cluster because application is in CANCELED, diagnostics
DispatcherResourceManagerComponent has been closed..
Jan 08, 2021 6:10:11 PM
org.apache.flink.runtime.entrypoint.component.DispatcherResourceManagerComponent
closeAsyncInternal
INFO: Closing components.
Jan 08, 2021 6:10:11 PM
org.apache.flink.runtime.dispatcher.runner.AbstractDispatcherLeaderProcess
closeInternal
INFO: Stopping SessionDispatcherLeaderProcess.
Jan 08, 2021 6:10:11 PM org.apache.flink.runtime.dispatcher.Dispatcher onStop
INFO: Stopping dispatcher akka://flink/user/dispatcher.
Jan 08, 2021 6:10:11 PM
org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl close
INFO: Closing the SlotManager.
Jan 08, 2021 6:10:11 PM org.apache.flink.runtime.dispatcher.Dispatcher
terminateJobManagerRunners
INFO: Stopping all currently running jobs of dispatcher
akka://flink/user/dispatcher.
Jan 08, 2021 6:10:11 PM
org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl suspend
INFO: Suspending the SlotManager.
Jan 08, 2021 6:10:11 PM
org.apache.flink.runtime.rest.handler.legacy.backpressure.BackPressureRequestCoordinator
shutDown
INFO: Shutting down back pressure request coordinator.
Jan 08, 2021 6:10:11 PM org.apache.flink.runtime.dispatcher.Dispatcher
lambda$onStop$1
INFO: Stopped dispatcher akka://flink/user/dispatcher.
Jan 08, 2021 6:10:11 PM org.apache.flink.runtime.rpc.akka.AkkaRpcService
stopService
INFO: Stopping Akka RPC service.
Jan 08, 2021 6:10:11 PM
akka.event.slf4j.Slf4jLogger$$anonfun$receive$1$$anonfun$applyOrElse$3
apply$mcV$sp
INFO: Shutting down remote daemon.
Jan 08, 2021 6:10:11 PM
akka.event.slf4j.Slf4jLogger$$anonfun$receive$1$$anonfun$applyOrElse$3
apply$mcV$sp
INFO: Remote daemon shut down; proceeding with flushing remote transports.
Jan 08, 2021 6:10:11 PM
akka.event.slf4j.Slf4jLogger$$anonfun$receive$1$$anonfun$applyOrElse$3
apply$mcV$sp
INFO: Remoting shut down.
Jan 08, 2021 6:10:11 PM org.apache.flink.runtime.rpc.akka.AkkaRpcService
stopService
INFO: Stopping Akka RPC service.
Jan 08, 2021 6:10:11 PM org.apache.flink.runtime.rpc.akka.AkkaRpcService
lambda$stopService$5
INFO: Stopped Akka RPC service.
Jan 08, 2021 6:10:11 PM org.apache.flink.runtime.blob.AbstractBlobCache close
INFO: Shutting down BLOB cache
Jan 08, 2021 6:10:11 PM org.apache.flink.runtime.blob.AbstractBlobCache close
INFO: Shutting down BLOB cache
Jan 08, 2021 6:10:11 PM org.apache.flink.runtime.blob.BlobServer close
INFO: Stopped BLOB server at 0.0.0.0:41589
Jan 08, 2021 6:10:11 PM org.apache.flink.runtime.rpc.akka.AkkaRpcService
lambda$stopService$5
INFO: Stopped Akka RPC service.
==========================================================================================
Run started 2021-01-08T18:09:49.546Z and ran for PT33.894S
Default configuration:
{"debug":true,"query":null,"sourceType":"DIRECT","generateEventFilePathPrefix":null,"sinkType":"DEVNULL","exportSummaryToBigQuery":false,"pubSubMode":"COMBINED","pubsubMessageSerializationMethod":"CODER","sideInputType":"DIRECT","sideInputRowCount":500,"sideInputNumShards":3,"sideInputUrl":null,"sessionGap":{"standardDays":0,"standardHours":0,"standardMinutes":10,"standardSeconds":600,"millis":600000},"numEvents":100000,"numEventGenerators":100,"rateShape":"SINE","firstEventRate":10000,"nextEventRate":10000,"rateUnit":"PER_SECOND","ratePeriodSec":600,"preloadSeconds":0,"streamTimeout":240,"isRateLimited":false,"useWallclockEventTime":false,"avgPersonByteSize":200,"avgAuctionByteSize":500,"avgBidByteSize":100,"hotAuctionRatio":2,"hotSellersRatio":4,"hotBiddersRatio":4,"windowSizeSec":10,"windowPeriodSec":5,"watermarkHoldbackSec":0,"numInFlightAuctions":100,"numActivePeople":1000,"coderStrategy":"HAND","cpuDelayMs":0,"diskBusyBytes":0,"auctionSkip":123,"fanout":5,"maxAuctionsWaitingTime":600,"occasionalDelaySec":3,"probDelayedEvent":0.1,"maxLogEvents":100000,"usePubsubPublishTime":false,"outOfOrderGroupSize":1}
Configurations:
Conf Description
0000 query:PASSTHROUGH; exportSummaryToBigQuery:true; streamTimeout:60
0001 query:CURRENCY_CONVERSION; exportSummaryToBigQuery:true;
streamTimeout:60
0002 query:SELECTION; exportSummaryToBigQuery:true; streamTimeout:60
0003 query:LOCAL_ITEM_SUGGESTION; exportSummaryToBigQuery:true;
streamTimeout:60
0004 query:AVERAGE_PRICE_FOR_CATEGORY; exportSummaryToBigQuery:true;
numEvents:10000; streamTimeout:60
0005 query:HOT_ITEMS; exportSummaryToBigQuery:true; streamTimeout:60
0006 query:AVERAGE_SELLING_PRICE_BY_SELLER; exportSummaryToBigQuery:true;
numEvents:10000; streamTimeout:60
0007 query:HIGHEST_BID; exportSummaryToBigQuery:true; streamTimeout:60
Exception in thread "main" 0008 query:MONITOR_NEW_USERS;
exportSummaryToBigQuery:true; streamTimeout:60
0009 query:WINNING_BIDS; exportSummaryToBigQuery:true; numEvents:10000;
streamTimeout:60
0010 query:LOG_TO_SHARDED_FILES; exportSummaryToBigQuery:true;
streamTimeout:60
0011 query:USER_SESSIONS; exportSummaryToBigQuery:true; streamTimeout:60
0012 query:PROCESSING_TIME_WINDOWS; exportSummaryToBigQuery:true;
streamTimeout:60
0013 query:BOUNDED_SIDE_INPUT_JOIN; exportSummaryToBigQuery:true;
streamTimeout:60
0014 query:SESSION_SIDE_INPUT_JOIN; exportSummaryToBigQuery:true;
streamTimeout:60
Performance:
Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results
(Baseline)
0000 1.1 91074.7 100000
0001 0.2 628930.8 92000
0002 0.1 1136363.6 351
0003 1.0 97276.3 580
0004 0.8 12165.5 40
0005 0.8 122399.0 12
0006 0.6 16949.2 103
0007 0.5 182481.8 1
0008 0.5 200000.0 6000
0009 0.3 37735.8 298
0010 0.5 203666.0 1
0011 0.3 354609.9 1919
0012 0.2 540540.5 1919
0013 0.4 271739.1 92000
0014 0.5 201207.2 92000
com.google.cloud.bigquery.BigQueryException: 502 Bad Gateway
POST
https://www.googleapis.com/bigquery/v2/projects/apache-beam-testing/datasets/nexmark/tables/nexmark_5_FlinkRunner_batch/insertAll?prettyPrint=false
<!DOCTYPE html>
==========================================================================================
<html lang=en>
<meta charset=utf-8>
<meta name=viewport content="initial-scale=1, minimum-scale=1,
width=device-width">
<title>Error 502 (Server Error)!!1</title>
<style>
*{margin:0;padding:0}html,code{font:15px/22px
arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7%
auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* >
body{background:url(//www.google.com/images/errors/robot.png) 100% 5px
no-repeat;padding-right:205px}p{margin:11px 0
22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media
screen and
(max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png)
no-repeat;margin-left:-5px}@media only screen and
(min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png)
no-repeat 0% 0%/100%
100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png)
0}}@media only screen and
(-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png)
no-repeat;-webkit-background-size:100%
100%}}#logo{display:inline-block;height:54px;width:150px}
</style>
<a href=//www.google.com/><span id=logo aria-label=Google></span></a>
<p><b>502.</b> <ins>That’s an error.</ins>
<p>The server encountered a temporary error and could not complete your
request.<p>Please try again in 30 seconds. <ins>That’s all we know.</ins>
at
com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.translate(HttpBigQueryRpc.java:115)
at
com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.insertAll(HttpBigQueryRpc.java:491)
at
com.google.cloud.bigquery.BigQueryImpl.insertAll(BigQueryImpl.java:1032)
at
org.apache.beam.sdk.testutils.publishing.BigQueryClient.insertAll(BigQueryClient.java:123)
at
org.apache.beam.sdk.testutils.publishing.BigQueryClient.insertRow(BigQueryClient.java:98)
at
org.apache.beam.sdk.testutils.publishing.BigQueryClient.insertRow(BigQueryClient.java:89)
at
org.apache.beam.sdk.testutils.publishing.BigQueryResultsPublisher.publish(BigQueryResultsPublisher.java:47)
at org.apache.beam.sdk.nexmark.Main.savePerfsToBigQuery(Main.java:203)
at org.apache.beam.sdk.nexmark.Main.runAll(Main.java:164)
at org.apache.beam.sdk.nexmark.Main.main(Main.java:494)
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException:
502 Bad Gateway
POST
https://www.googleapis.com/bigquery/v2/projects/apache-beam-testing/datasets/nexmark/tables/nexmark_5_FlinkRunner_batch/insertAll?prettyPrint=false
<!DOCTYPE html>
<html lang=en>
<meta charset=utf-8>
<meta name=viewport content="initial-scale=1, minimum-scale=1,
width=device-width">
<title>Error 502 (Server Error)!!1</title>
<style>
*{margin:0;padding:0}html,code{font:15px/22px
arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7%
auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* >
body{background:url(//www.google.com/images/errors/robot.png) 100% 5px
no-repeat;padding-right:205px}p{margin:11px 0
22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media
screen and
(max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png)
no-repeat;margin-left:-5px}@media only screen and
(min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png)
no-repeat 0% 0%/100%
100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png)
0}}@media only screen and
(-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png)
no-repeat;-webkit-background-size:100%
100%}}#logo{display:inline-block;height:54px;width:150px}
</style>
<a href=//www.google.com/><span id=logo aria-label=Google></span></a>
<p><b>502.</b> <ins>That’s an error.</ins>
<p>The server encountered a temporary error and could not complete your
request.<p>Please try again in 30 seconds. <ins>That’s all we know.</ins>
at
com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:149)
at
com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:112)
at
com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:39)
at
com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:443)
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1108)
at
com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:541)
at
com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:474)
at
com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:591)
at
com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.insertAll(HttpBigQueryRpc.java:489)
... 8 more
> Task :sdks:java:testing:nexmark:run FAILED
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:java:testing:nexmark:run'.
> Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with
> non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See
https://docs.gradle.org/6.7.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 1m 3s
121 actionable tasks: 80 executed, 41 from cache
Publishing build scan...
https://gradle.com/s/qex5tbakgifqi
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
