See
<https://ci-beam.apache.org/job/beam_PostCommit_Java_Nexmark_Flink/6687/display/redirect?page=changes>
Changes:
[Pablo Estrada] Adding website update for release 2.27.0
[Pablo Estrada] Add 2.27.0 blog post, and update CHANGES.md
[heejong] [BEAM-11508] Set sdkHarnessContainerImages in workerpool configuration
[heejong] add utility function to check runner v2, set single-core flag for
python
[noreply] [BEAM-11327] Update JulHandler with Charset.defaultCharset() (#13701)
[Pablo Estrada] Fixing script for Java8/Java11 docker container push
[noreply] Update the blog date for 2.27
------------------------------------------
[...truncated 40.83 MB...]
"NAMESPACE": "source",
"NAME": "elements_read",
"PTRANSFORM":
"NexmarkUtils.GenerateSideInputData/GenerateSequence/Read(BoundedCountingSource)/ParDo(BoundedSourceAsSDFWrapper)/ParMultiDo(BoundedSourceAsSDFWrapper)/ProcessKeyedElements/SplittableParDoViaKeyedWorkItems.ProcessElements"
}
}, {
"urn": "beam:metric:user:sum_int64:v1",
"type": "beam:metrics:sum_int64:v1",
"payload": "4M4F",
"labels": {
"NAMESPACE": "SessionSideInputJoin",
"NAME": "records",
"PTRANSFORM": "SessionSideInputJoin.Format/ParMultiDo(Anonymous)"
}
}]
}
}
Jan 09, 2021 12:12:15 AM
org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl freeSlotInternal
INFO: Free slot TaskSlot(index:7, state:ACTIVE, resource profile:
ResourceProfile{managedMemory=8.000mb (8388608 bytes), networkMemory=4.000mb
(4194304 bytes)}, allocationId: 41c1f2bcb0a8b4e357ab4826db109716, jobId:
82e8345a4540749cb9b63e40ffe6895b).
2021-01-09T00:12:15.643Z Waiting for main pipeline to 'finish'
2021-01-09T00:12:15.643Z DONE SessionSideInputJoin
2021-01-09T00:12:15.644Z 1 samples not enough to calculate steady-state event
rate
Jan 09, 2021 12:12:15 AM org.apache.beam.sdk.testutils.metrics.MetricsReader
getCounterMetric
SEVERE: Failed to get metric fatal, from namespace SessionSideInputJoin
==========================================================================================
Run started 2021-01-09T00:10:44.489Z and ran for PT91.155S
Default configuration:
{"debug":true,"query":null,"sourceType":"DIRECT","generateEventFilePathPrefix":null,"sinkType":"DEVNULL","exportSummaryToBigQuery":false,"pubSubMode":"COMBINED","pubsubMessageSerializationMethod":"CODER","sideInputType":"DIRECT","sideInputRowCount":500,"sideInputNumShards":3,"sideInputUrl":null,"sessionGap":{"standardDays":0,"standardHours":0,"standardMinutes":10,"standardSeconds":600,"millis":600000},"numEvents":100000,"numEventGenerators":100,"rateShape":"SINE","firstEventRate":10000,"nextEventRate":10000,"rateUnit":"PER_SECOND","ratePeriodSec":600,"preloadSeconds":0,"streamTimeout":240,"isRateLimited":false,"useWallclockEventTime":false,"avgPersonByteSize":200,"avgAuctionByteSize":500,"avgBidByteSize":100,"hotAuctionRatio":2,"hotSellersRatio":4,"hotBiddersRatio":4,"windowSizeSec":10,"windowPeriodSec":5,"watermarkHoldbackSec":0,"numInFlightAuctions":100,"numActivePeople":1000,"coderStrategy":"HAND","cpuDelayMs":0,"diskBusyBytes":0,"auctionSkip":123,"fanout":5,"maxAuctionsWaitingTime":600,"occasionalDelaySec":3,"probDelayedEvent":0.1,"maxLogEvents":100000,"usePubsubPublishTime":false,"outOfOrderGroupSize":1}
Configurations:
Conf Description
0000 query:PASSTHROUGH; exportSummaryToBigQuery:true; streamTimeout:60
0001 query:CURRENCY_CONVERSION; exportSummaryToBigQuery:true;
streamTimeout:60
0002 query:SELECTION; exportSummaryToBigQuery:true; streamTimeout:60
Jan 09, 2021 12:12:15 AM org.apache.flink.runtime.taskexecutor.TaskExecutor
closeJobManagerConnection
0003 query:LOCAL_ITEM_SUGGESTION; exportSummaryToBigQuery:true;
streamTimeout:60
0004 query:AVERAGE_PRICE_FOR_CATEGORY; exportSummaryToBigQuery:true;
numEvents:10000; streamTimeout:60
0005 query:HOT_ITEMS; exportSummaryToBigQuery:true; streamTimeout:60
0006 query:AVERAGE_SELLING_PRICE_BY_SELLER; exportSummaryToBigQuery:true;
numEvents:10000; streamTimeout:60
0007 query:HIGHEST_BID; exportSummaryToBigQuery:true; streamTimeout:60
0008 query:MONITOR_NEW_USERS; exportSummaryToBigQuery:true; streamTimeout:60
0009 query:WINNING_BIDS; exportSummaryToBigQuery:true; numEvents:10000;
streamTimeout:60
0010 query:LOG_TO_SHARDED_FILES; exportSummaryToBigQuery:true;
streamTimeout:60
0011 query:USER_SESSIONS; exportSummaryToBigQuery:true; streamTimeout:60
0012 query:PROCESSING_TIME_WINDOWS; exportSummaryToBigQuery:true;
streamTimeout:60
0013 query:BOUNDED_SIDE_INPUT_JOIN; exportSummaryToBigQuery:true;
streamTimeout:60
0014 query:SESSION_SIDE_INPUT_JOIN; exportSummaryToBigQuery:true;
streamTimeout:60
Performance:
Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results
(Baseline)
0000 1.5 64724.9 100000
0001 0.9 116414.4 92000
0002 0.4 278551.5 351
0003 0.6 178253.1 580
0004 1.2 8554.3 40
0005 1.6 64474.5 12
0006 0.5 19531.3 401
0007 18.3 5457.9 1
0008 0.4 230414.7 6000
0009 0.4 24038.5 298
0010 1.3 76982.3 1
0011 1.4 71225.1 1919
0012 0.7 136425.6 1919
0013 29.0 3454.1 92000
0014 1.7 58788.9 92000
INFO: Close JobManager connection for job 82e8345a4540749cb9b63e40ffe6895b.
==========================================================================================
Jan 09, 2021 12:12:15 AM org.apache.flink.runtime.taskexecutor.TaskExecutor
closeJobManagerConnection
INFO: Close JobManager connection for job 82e8345a4540749cb9b63e40ffe6895b.
Jan 09, 2021 12:12:15 AM org.apache.flink.runtime.taskexecutor.JobLeaderService
stop
INFO: Stop job leader service.
Jan 09, 2021 12:12:15 AM
org.apache.flink.runtime.state.TaskExecutorLocalStateStoresManager shutdown
INFO: Shutting down TaskExecutorLocalStateStoresManager.
Jan 09, 2021 12:12:15 AM org.apache.flink.runtime.webmonitor.WebMonitorEndpoint
lambda$shutDownInternal$5
INFO: Removing cache directory /tmp/flink-web-ui
Jan 09, 2021 12:12:15 AM org.apache.flink.runtime.rest.RestServerEndpoint
lambda$closeAsync$1
INFO: Shut down complete.
Jan 09, 2021 12:12:15 AM
org.apache.flink.runtime.io.disk.FileChannelManagerImpl lambda$getFileCloser$0
INFO: FileChannelManager removed spill file directory
/tmp/flink-io-fcd9f1bf-fc6e-46e4-a162-f3eca2eaf71c
Jan 09, 2021 12:12:15 AM
org.apache.flink.runtime.io.network.NettyShuffleEnvironment close
INFO: Shutting down the network environment and its components.
Jan 09, 2021 12:12:15 AM
org.apache.flink.runtime.resourcemanager.ResourceManager deregisterApplication
INFO: Shut down cluster because application is in CANCELED, diagnostics
DispatcherResourceManagerComponent has been closed..
Jan 09, 2021 12:12:15 AM
org.apache.flink.runtime.entrypoint.component.DispatcherResourceManagerComponent
closeAsyncInternal
INFO: Closing components.
Jan 09, 2021 12:12:15 AM
org.apache.flink.runtime.dispatcher.runner.AbstractDispatcherLeaderProcess
closeInternal
INFO: Stopping SessionDispatcherLeaderProcess.
Jan 09, 2021 12:12:15 AM
org.apache.flink.runtime.io.disk.FileChannelManagerImpl lambda$getFileCloser$0
INFO: FileChannelManager removed spill file directory
/tmp/flink-netty-shuffle-3715adfe-681e-47d2-a966-e84244d72efc
Jan 09, 2021 12:12:15 AM org.apache.flink.runtime.dispatcher.Dispatcher onStop
INFO: Stopping dispatcher akka://flink/user/dispatcher.
Jan 09, 2021 12:12:15 AM org.apache.flink.runtime.taskexecutor.KvStateService
shutdown
INFO: Shutting down the kvState service and its components.
Jan 09, 2021 12:12:15 AM
org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl close
INFO: Closing the SlotManager.
Jan 09, 2021 12:12:15 AM org.apache.flink.runtime.dispatcher.Dispatcher
terminateJobManagerRunners
INFO: Stopping all currently running jobs of dispatcher
akka://flink/user/dispatcher.
Jan 09, 2021 12:12:15 AM
org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl suspend
INFO: Suspending the SlotManager.
Jan 09, 2021 12:12:15 AM org.apache.flink.runtime.taskexecutor.JobLeaderService
stop
INFO: Stop job leader service.
Jan 09, 2021 12:12:15 AM org.apache.flink.runtime.filecache.FileCache shutdown
INFO: removed file cache directory
/tmp/flink-dist-cache-8d68f1a7-5a3c-46e5-961c-c621fde4cc64
Jan 09, 2021 12:12:15 AM org.apache.flink.runtime.taskexecutor.TaskExecutor
handleOnStopException
INFO: Stopped TaskExecutor akka://flink/user/taskmanager_28.
Jan 09, 2021 12:12:15 AM
org.apache.flink.runtime.rest.handler.legacy.backpressure.BackPressureRequestCoordinator
shutDown
INFO: Shutting down back pressure request coordinator.
Jan 09, 2021 12:12:15 AM org.apache.flink.runtime.dispatcher.Dispatcher
lambda$onStop$1
INFO: Stopped dispatcher akka://flink/user/dispatcher.
Jan 09, 2021 12:12:15 AM org.apache.flink.runtime.rpc.akka.AkkaRpcService
stopService
INFO: Stopping Akka RPC service.
Jan 09, 2021 12:12:15 AM
akka.event.slf4j.Slf4jLogger$$anonfun$receive$1$$anonfun$applyOrElse$3
apply$mcV$sp
INFO: Shutting down remote daemon.
Jan 09, 2021 12:12:15 AM
akka.event.slf4j.Slf4jLogger$$anonfun$receive$1$$anonfun$applyOrElse$3
apply$mcV$sp
INFO: Remote daemon shut down; proceeding with flushing remote transports.
Jan 09, 2021 12:12:15 AM
akka.event.slf4j.Slf4jLogger$$anonfun$receive$1$$anonfun$applyOrElse$3
apply$mcV$sp
INFO: Remoting shut down.
Jan 09, 2021 12:12:15 AM org.apache.flink.runtime.rpc.akka.AkkaRpcService
stopService
INFO: Stopping Akka RPC service.
Jan 09, 2021 12:12:15 AM org.apache.flink.runtime.rpc.akka.AkkaRpcService
lambda$stopService$5
INFO: Stopped Akka RPC service.
Jan 09, 2021 12:12:15 AM org.apache.flink.runtime.blob.AbstractBlobCache close
INFO: Shutting down BLOB cache
Jan 09, 2021 12:12:15 AM org.apache.flink.runtime.blob.AbstractBlobCache close
INFO: Shutting down BLOB cache
Jan 09, 2021 12:12:15 AM org.apache.flink.runtime.blob.BlobServer close
INFO: Stopped BLOB server at 0.0.0.0:45039
Jan 09, 2021 12:12:15 AM org.apache.flink.runtime.rpc.akka.AkkaRpcService
lambda$stopService$5
INFO: Stopped Akka RPC service.
==========================================================================================
Run started 2021-01-09T00:10:44.489Z and ran for PT104.306S
Default configuration:
{"debug":true,"query":null,"sourceType":"DIRECT","generateEventFilePathPrefix":null,"sinkType":"DEVNULL","exportSummaryToBigQuery":false,"pubSubMode":"COMBINED","pubsubMessageSerializationMethod":"CODER","sideInputType":"DIRECT","sideInputRowCount":500,"sideInputNumShards":3,"sideInputUrl":null,"sessionGap":{"standardDays":0,"standardHours":0,"standardMinutes":10,"standardSeconds":600,"millis":600000},"numEvents":100000,"numEventGenerators":100,"rateShape":"SINE","firstEventRate":10000,"nextEventRate":10000,"rateUnit":"PER_SECOND","ratePeriodSec":600,"preloadSeconds":0,"streamTimeout":240,"isRateLimited":false,"useWallclockEventTime":false,"avgPersonByteSize":200,"avgAuctionByteSize":500,"avgBidByteSize":100,"hotAuctionRatio":2,"hotSellersRatio":4,"hotBiddersRatio":4,"windowSizeSec":10,"windowPeriodSec":5,"watermarkHoldbackSec":0,"numInFlightAuctions":100,"numActivePeople":1000,"coderStrategy":"HAND","cpuDelayMs":0,"diskBusyBytes":0,"auctionSkip":123,"fanout":5,"maxAuctionsWaitingTime":600,"occasionalDelaySec":3,"probDelayedEvent":0.1,"maxLogEvents":100000,"usePubsubPublishTime":false,"outOfOrderGroupSize":1}
Configurations:
Conf Description
0000 query:PASSTHROUGH; exportSummaryToBigQuery:true; streamTimeout:60
0001 query:CURRENCY_CONVERSION; exportSummaryToBigQuery:true;
streamTimeout:60
0002 query:SELECTION; exportSummaryToBigQuery:true; streamTimeout:60
0003 query:LOCAL_ITEM_SUGGESTION; exportSummaryToBigQuery:true;
streamTimeout:60
0004 query:AVERAGE_PRICE_FOR_CATEGORY; exportSummaryToBigQuery:true;
numEvents:10000; streamTimeout:60
0005 query:HOT_ITEMS; exportSummaryToBigQuery:true; streamTimeout:60
0006 query:AVERAGE_SELLING_PRICE_BY_SELLER; exportSummaryToBigQuery:true;
numEvents:10000; streamTimeout:60
0007 query:HIGHEST_BID; exportSummaryToBigQuery:true; streamTimeout:60
0008 query:MONITOR_NEW_USERS; exportSummaryToBigQuery:true; streamTimeout:60
0009 query:WINNING_BIDS; exportSummaryToBigQuery:true; numEvents:10000;
streamTimeout:60
0010 query:LOG_TO_SHARDED_FILES; exportSummaryToBigQuery:true;
streamTimeout:60
0011 query:USER_SESSIONS; exportSummaryToBigQuery:true; streamTimeout:60
0012 query:PROCESSING_TIME_WINDOWS; exportSummaryToBigQuery:true;
streamTimeout:60
Exception in thread "main" 0013 query:BOUNDED_SIDE_INPUT_JOIN;
exportSummaryToBigQuery:true; streamTimeout:60
0014 query:SESSION_SIDE_INPUT_JOIN; exportSummaryToBigQuery:true;
streamTimeout:60
Performance:
Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results
(Baseline)
0000 1.5 64724.9 100000
0001 0.9 116414.4 92000
0002 0.4 278551.5 351
0003 0.6 178253.1 580
0004 1.2 8554.3 40
0005 1.6 64474.5 12
0006 0.5 19531.3 401
0007 18.3 5457.9 1
0008 0.4 230414.7 6000
0009 0.4 24038.5 298
0010 1.3 76982.3 1
0011 1.4 71225.1 1919
0012 0.7 136425.6 1919
0013 29.0 3454.1 92000
0014 1.7 58788.9 92000
==========================================================================================
com.google.cloud.bigquery.BigQueryException: 502 Bad Gateway
POST
https://www.googleapis.com/bigquery/v2/projects/apache-beam-testing/datasets/nexmark/tables/nexmark_12_FlinkRunner_streaming/insertAll?prettyPrint=false
<!DOCTYPE html>
<html lang=en>
<meta charset=utf-8>
<meta name=viewport content="initial-scale=1, minimum-scale=1,
width=device-width">
<title>Error 502 (Server Error)!!1</title>
<style>
*{margin:0;padding:0}html,code{font:15px/22px
arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7%
auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* >
body{background:url(//www.google.com/images/errors/robot.png) 100% 5px
no-repeat;padding-right:205px}p{margin:11px 0
22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media
screen and
(max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png)
no-repeat;margin-left:-5px}@media only screen and
(min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png)
no-repeat 0% 0%/100%
100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png)
0}}@media only screen and
(-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png)
no-repeat;-webkit-background-size:100%
100%}}#logo{display:inline-block;height:54px;width:150px}
</style>
<a href=//www.google.com/><span id=logo aria-label=Google></span></a>
<p><b>502.</b> <ins>That’s an error.</ins>
<p>The server encountered a temporary error and could not complete your
request.<p>Please try again in 30 seconds. <ins>That’s all we know.</ins>
at
com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.translate(HttpBigQueryRpc.java:115)
at
com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.insertAll(HttpBigQueryRpc.java:491)
at
com.google.cloud.bigquery.BigQueryImpl.insertAll(BigQueryImpl.java:1032)
at
org.apache.beam.sdk.testutils.publishing.BigQueryClient.insertAll(BigQueryClient.java:123)
at
org.apache.beam.sdk.testutils.publishing.BigQueryClient.insertRow(BigQueryClient.java:98)
at
org.apache.beam.sdk.testutils.publishing.BigQueryClient.insertRow(BigQueryClient.java:89)
at
org.apache.beam.sdk.testutils.publishing.BigQueryResultsPublisher.publish(BigQueryResultsPublisher.java:47)
at org.apache.beam.sdk.nexmark.Main.savePerfsToBigQuery(Main.java:203)
at org.apache.beam.sdk.nexmark.Main.runAll(Main.java:164)
at org.apache.beam.sdk.nexmark.Main.main(Main.java:494)
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException:
502 Bad Gateway
POST
https://www.googleapis.com/bigquery/v2/projects/apache-beam-testing/datasets/nexmark/tables/nexmark_12_FlinkRunner_streaming/insertAll?prettyPrint=false
<!DOCTYPE html>
<html lang=en>
<meta charset=utf-8>
<meta name=viewport content="initial-scale=1, minimum-scale=1,
width=device-width">
<title>Error 502 (Server Error)!!1</title>
<style>
*{margin:0;padding:0}html,code{font:15px/22px
arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7%
auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* >
body{background:url(//www.google.com/images/errors/robot.png) 100% 5px
no-repeat;padding-right:205px}p{margin:11px 0
22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media
screen and
(max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png)
no-repeat;margin-left:-5px}@media only screen and
(min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png)
no-repeat 0% 0%/100%
100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png)
0}}@media only screen and
(-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png)
no-repeat;-webkit-background-size:100%
100%}}#logo{display:inline-block;height:54px;width:150px}
</style>
<a href=//www.google.com/><span id=logo aria-label=Google></span></a>
<p><b>502.</b> <ins>That’s an error.</ins>
<p>The server encountered a temporary error and could not complete your
request.<p>Please try again in 30 seconds. <ins>That’s all we know.</ins>
at
com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:149)
at
com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:112)
at
com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:39)
at
com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:443)
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1108)
at
com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:541)
at
com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:474)
at
com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:591)
at
com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.insertAll(HttpBigQueryRpc.java:489)
... 8 more
> Task :sdks:java:testing:nexmark:run FAILED
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:java:testing:nexmark:run'.
> Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with
> non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See
https://docs.gradle.org/6.7.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 1m 50s
121 actionable tasks: 2 executed, 119 up-to-date
Publishing build scan...
https://gradle.com/s/v2to3mhxr7nkw
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
