[
https://issues.apache.org/jira/browse/BEAM-6002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16677362#comment-16677362
]
Kenneth Knowles commented on BEAM-6002:
---------------------------------------
https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_Nexmark_Direct/1041/
Also hanging for hours.
{code}
17:14:20 Configurations:
17:14:20 Conf Description
17:14:20 0000 query:PASSTHROUGH; exportSummaryToBigQuery:true
17:14:20 0001 query:CURRENCY_CONVERSION; exportSummaryToBigQuery:true
17:14:20 0002 query:SELECTION; exportSummaryToBigQuery:true
17:14:20 0003 query:LOCAL_ITEM_SUGGESTION; exportSummaryToBigQuery:true
17:14:20 0004 query:AVERAGE_PRICE_FOR_CATEGORY;
exportSummaryToBigQuery:true; numEvents:10000
17:14:20 0005 query:HOT_ITEMS; exportSummaryToBigQuery:true
17:14:20 0006 query:AVERAGE_SELLING_PRICE_BY_SELLER;
exportSummaryToBigQuery:true; numEvents:10000
17:14:20 0007 query:HIGHEST_BID; exportSummaryToBigQuery:true
17:14:20 0008 query:MONITOR_NEW_USERS; exportSummaryToBigQuery:true
17:14:20 0009 query:WINNING_BIDS; exportSummaryToBigQuery:true;
numEvents:10000
17:14:20 0010 query:LOG_TO_SHARDED_FILES; exportSummaryToBigQuery:true
17:14:20 0011 query:USER_SESSIONS; exportSummaryToBigQuery:true
17:14:20 0012 query:PROCESSING_TIME_WINDOWS; exportSummaryToBigQuery:true
17:14:20 0013 query:BOUNDED_SIDE_INPUT_JOIN; exportSummaryToBigQuery:true
17:14:20
17:14:20 Performance:
17:14:20 Conf Runtime(sec) (Baseline) Events(/sec) (Baseline)
Results (Baseline)
17:14:20 0000 6.7 14941.0
100000
17:14:20 0001 3.6 27948.6
92000
17:14:20 0002 *** not run ***
17:14:20 0003 *** not run ***
17:14:20 0004 *** not run ***
17:14:20 0005 *** not run ***
17:14:20 0006 *** not run ***
17:14:20 0007 *** not run ***
17:14:20 0008 *** not run ***
17:14:20 0009 *** not run ***
17:14:20 0010 *** not run ***
17:14:20 0011 *** not run ***
17:14:20 0012 *** not run ***
17:14:20 0013 *** not run ***
17:14:20
==========================================================================================
17:14:20
17:14:20 2018-11-06T01:14:20.667Z Generating 100000 events in batch mode
17:14:22 2018-11-06T01:14:22.295Z Waiting for main pipeline to 'finish'
17:14:22 2018-11-06T01:14:22.296Z DONE Query2
17:14:22 2018-11-06T01:14:22.296Z Running query:LOCAL_ITEM_SUGGESTION;
exportSummaryToBigQuery:true
17:14:22
17:14:22
==========================================================================================
17:14:22 Run started 2018-11-06T01:14:08.264Z and ran for PT14.032S
17:14:22
17:14:22 Default configuration:
17:14:22
{"debug":true,"query":null,"sourceType":"DIRECT","sinkType":"DEVNULL","exportSummaryToBigQuery":false,"pubSubMode":"COMBINED","sideInputType":"DIRECT","sideInputRowCount":500,"sideInputNumShards":3,"sideInputUrl":null,"numEvents":100000,"numEventGenerators":100,"rateShape":"SINE","firstEventRate":10000,"nextEventRate":10000,"rateUnit":"PER_SECOND","ratePeriodSec":600,"preloadSeconds":0,"streamTimeout":240,"isRateLimited":false,"useWallclockEventTime":false,"avgPersonByteSize":200,"avgAuctionByteSize":500,"avgBidByteSize":100,"hotAuctionRatio":2,"hotSellersRatio":4,"hotBiddersRatio":4,"windowSizeSec":10,"windowPeriodSec":5,"watermarkHoldbackSec":0,"numInFlightAuctions":100,"numActivePeople":1000,"coderStrategy":"HAND","cpuDelayMs":0,"diskBusyBytes":0,"auctionSkip":123,"fanout":5,"maxAuctionsWaitingTime":600,"occasionalDelaySec":3,"probDelayedEvent":0.1,"maxLogEvents":100000,"usePubsubPublishTime":false,"outOfOrderGroupSize":1}
17:14:22
17:14:22 Configurations:
17:14:22 Conf Description
17:14:22 0000 query:PASSTHROUGH; exportSummaryToBigQuery:true
17:14:22 0001 query:CURRENCY_CONVERSION; exportSummaryToBigQuery:true
17:14:22 0002 query:SELECTION; exportSummaryToBigQuery:true
17:14:22 0003 query:LOCAL_ITEM_SUGGESTION; exportSummaryToBigQuery:true
17:14:22 0004 query:AVERAGE_PRICE_FOR_CATEGORY;
exportSummaryToBigQuery:true; numEvents:10000
17:14:22 0005 query:HOT_ITEMS; exportSummaryToBigQuery:true
17:14:22 0006 query:AVERAGE_SELLING_PRICE_BY_SELLER;
exportSummaryToBigQuery:true; numEvents:10000
17:14:22 0007 query:HIGHEST_BID; exportSummaryToBigQuery:true
17:14:22 0008 query:MONITOR_NEW_USERS; exportSummaryToBigQuery:true
17:14:22 0009 query:WINNING_BIDS; exportSummaryToBigQuery:true;
numEvents:10000
17:14:22 0010 query:LOG_TO_SHARDED_FILES; exportSummaryToBigQuery:true
17:14:22 0011 query:USER_SESSIONS; exportSummaryToBigQuery:true
17:14:22 0012 query:PROCESSING_TIME_WINDOWS; exportSummaryToBigQuery:true
17:14:22 0013 query:BOUNDED_SIDE_INPUT_JOIN; exportSummaryToBigQuery:true
17:14:22
17:14:22 Performance:
17:14:22 Conf Runtime(sec) (Baseline) Events(/sec) (Baseline)
Results (Baseline)
17:14:22 0000 6.7 14941.0
100000
17:14:22 0001 3.6 27948.6
92000
17:14:22 0002 1.5 67476.4
351
17:14:22 0003 *** not run ***
17:14:22 0004 *** not run ***
17:14:22 0005 *** not run ***
17:14:22 0006 *** not run ***
17:14:22 0007 *** not run ***
17:14:22 0008 *** not run ***
17:14:22 0009 *** not run ***
17:14:22 0010 *** not run ***
17:14:22 0011 *** not run ***
17:14:22 0012 *** not run ***
17:14:22 0013 *** not run ***
17:14:22
==========================================================================================
17:14:22
17:14:22 2018-11-06T01:14:22.298Z Generating 100000 events in batch mode
21:10:52 Build timed out (after 240 minutes). Marking the build as aborted.
21:10:52 Build was aborted
21:10:54 No emails were triggered.
21:10:54 Finished: ABORTED
{code}
> Nexmark tests timing out on all runners (crash loop due to metrics?)
> ---------------------------------------------------------------------
>
> Key: BEAM-6002
> URL: https://issues.apache.org/jira/browse/BEAM-6002
> Project: Beam
> Issue Type: Bug
> Components: test-failures
> Reporter: Kenneth Knowles
> Assignee: Lukasz Gajowy
> Priority: Critical
>
> https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_Nexmark_Spark/992/
> {code}
> 08:58:26 2018-11-06T16:58:26.035Z RUNNING Query0
> 08:58:26 18/11/06 16:58:26 ERROR
> org.apache.beam.sdk.testutils.metrics.MetricsReader: Failed to get metric
> event.elements, from namespace Query0.Events
> 08:58:26 18/11/06 16:58:26 ERROR
> org.apache.beam.sdk.testutils.metrics.MetricsReader: Failed to get metric
> event.bytes, from namespace Query0.Events
> 08:58:26 18/11/06 16:58:26 ERROR
> org.apache.beam.sdk.testutils.metrics.MetricsReader: Failed to get
> distribution metric event.startTime for namespace Query0.Events
> 08:58:26 2018-11-06T16:58:26.035Z no activity
> 08:58:26 18/11/06 16:58:26 ERROR
> org.apache.beam.sdk.testutils.metrics.MetricsReader: Failed to get
> distribution metric event.endTime for namespace Query0.Events
> 08:58:26 18/11/06 16:58:26 ERROR
> org.apache.beam.sdk.testutils.metrics.MetricsReader: Failed to get metric
> result.elements, from namespace Query0.Results
> 08:58:26 18/11/06 16:58:26 ERROR
> org.apache.beam.sdk.testutils.metrics.MetricsReader: Failed to get metric
> result.bytes, from namespace Query0.Results
> 08:58:26 18/11/06 16:58:26 ERROR
> org.apache.beam.sdk.testutils.metrics.MetricsReader: Failed to get
> distribution metric result.startTime for namespace Query0.Results
> 08:58:26 18/11/06 16:58:26 ERROR
> org.apache.beam.sdk.testutils.metrics.MetricsReader: Failed to get
> distribution metric result.endTime for namespace Query0.Results
> 08:58:26 18/11/06 16:58:26 ERROR
> org.apache.beam.sdk.testutils.metrics.MetricsReader: Failed to get
> distribution metric result.startTimestamp for namespace Query0.Results
> 08:58:26 18/11/06 16:58:26 ERROR
> org.apache.beam.sdk.testutils.metrics.MetricsReader: Failed to get
> distribution metric result.endTimestamp for namespace Query0.Results
> 08:58:41 2018-11-06T16:58:41.035Z RUNNING Query0
> 08:58:41 18/11/06 16:58:41 ERROR
> org.apache.beam.sdk.testutils.metrics.MetricsReader: Failed to get metric
> event.elements, from namespace Query0.Events
> 08:58:41 2018-11-06T16:58:41.036Z no activity
> 08:58:41 18/11/06 16:58:41 ERROR
> org.apache.beam.sdk.testutils.metrics.MetricsReader: Failed to get metric
> event.bytes, from namespace Query0.Events
> 08:58:41 18/11/06 16:58:41 ERROR
> org.apache.beam.sdk.testutils.metrics.MetricsReader: Failed to get
> distribution metric event.startTime for namespace Query0.Events
> 08:58:41 18/11/06 16:58:41 ERROR
> org.apache.beam.sdk.testutils.metrics.MetricsReader: Failed to get
> distribution metric event.endTime for namespace Query0.Events
> 08:58:41 18/11/06 16:58:41 ERROR
> org.apache.beam.sdk.testutils.metrics.MetricsReader: Failed to get metric
> result.elements, from namespace Query0.Results
> 08:58:41 18/11/06 16:58:41 ERROR
> org.apache.beam.sdk.testutils.metrics.MetricsReader: Failed to get metric
> result.bytes, from namespace Query0.Results
> 08:58:41 18/11/06 16:58:41 ERROR
> org.apache.beam.sdk.testutils.metrics.MetricsReader: Failed to get
> distribution metric result.startTime for namespace Query0.Results
> 08:58:41 18/11/06 16:58:41 ERROR
> org.apache.beam.sdk.testutils.metrics.MetricsReader: Failed to get
> distribution metric result.endTime for namespace Query0.Results
> 08:58:41 18/11/06 16:58:41 ERROR
> org.apache.beam.sdk.testutils.metrics.MetricsReader: Failed to get
> distribution metric result.startTimestamp for namespace Query0.Results
> 08:58:41 18/11/06 16:58:41 ERROR
> org.apache.beam.sdk.testutils.metrics.MetricsReader: Failed to get
> distribution metric result.endTimestamp for namespace Query0.Results
> 08:58:56 2018-11-06T16:58:56.036Z RUNNING Query0
> 08:58:56 18/11/06 16:58:56 ERROR
> org.apache.beam.sdk.testutils.metrics.MetricsReader: Failed to get metric
> event.elements, from namespace Query0.Events
> 08:58:56 2018-11-06T16:58:56.036Z no activity
> 08:58:56 18/11/06 16:58:56 ERROR
> org.apache.beam.sdk.testutils.metrics.MetricsReader: Failed to get metric
> event.bytes, from namespace Query0.Events
> 08:58:56 18/11/06 16:58:56 ERROR
> org.apache.beam.sdk.testutils.metrics.MetricsReader: Failed to get
> distribution metric event.startTime for namespace Query0.Events
> 08:58:56 18/11/06 16:58:56 ERROR
> org.apache.beam.sdk.testutils.metrics.MetricsReader: Failed to get
> distribution metric event.endTime for namespace Query0.Events
> 08:58:56 18/11/06 16:58:56 ERROR
> org.apache.beam.sdk.testutils.metrics.MetricsReader: Failed to get metric
> result.elements, from namespace Query0.Results
> 08:58:56 18/11/06 16:58:56 ERROR
> org.apache.beam.sdk.testutils.metrics.MetricsReader: Failed to get metric
> result.bytes, from namespace Query0.Results
> 08:58:56 18/11/06 16:58:56 ERROR
> org.apache.beam.sdk.testutils.metrics.MetricsReader: Failed to get
> distribution metric result.startTime for namespace Query0.Results
> 08:58:56 18/11/06 16:58:56 ERROR
> org.apache.beam.sdk.testutils.metrics.MetricsReader: Failed to get
> distribution metric result.endTime for namespace Query0.Results
> 08:58:56 18/11/06 16:58:56 ERROR
> org.apache.beam.sdk.testutils.metrics.MetricsReader: Failed to get
> distribution metric result.startTimestamp for namespace Query0.Results
> 08:58:56 18/11/06 16:58:56 ERROR
> org.apache.beam.sdk.testutils.metrics.MetricsReader: Failed to get
> distribution metric result.endTimestamp for namespace Query0.Results
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
