chenboat commented on issue #4626: Low level realtime consumer (LLC) got into ERROR state due to thread race condition. URL: https://github.com/apache/incubator-pinot/issues/4626#issuecomment-533645720 2019-09-13 00:21:26 INFO FileUploadDownloadClient:318 - Sending request: http://streampinot-prod02-phx3:5983/segmentConsumed?name=testTableCreation__13__5__20190912T1931Z&offset=421474657&instance=Server_streampinot-prod18-phx3_7090&reason=rowLimit&memoryUsedBytes=5947422696&rowCount=375000 to controller: localhost, version: 0.2.12-0053756739acda99e58f8eb9dee9cd9a3c785061 2019-09-13 00:21:26 INFO ServerSegmentCompletionProtocolHandler:183 - Controller response {"buildTimeSec":-1,"isSplitCommitType":false,"status":"KEEP","offset":421474657} for http://streampinot-prod02-phx3:5983/segmentConsumed?name=testTableCreation__13__5__20190912T1931Z&offset=421474657&instance=Server_streampinot-prod18-phx3_7090&reason=rowLimit&memoryUsedBytes=5947422696&rowCount=375000 2019-09-13 00:21:26 INFO LLRealtimeSegmentDataManager_testTableCreation__13__5__20190912T1931Z:652 - Waiting to acquire semaphore for building segment 2019-09-13 00:21:27 INFO HelixTaskExecutor:409 - Scheduling message e2fb09c1-9dad-4cab-bd09-135b25ca8837: testTableCreation_REALTIME:testTableCreation__13__5__20190912T1931Z, CONSUMING->ONLINE 2019-09-13 00:21:27 INFO HelixStateTransitionHandler:105 - handling message: e2fb09c1-9dad-4cab-bd09-135b25ca8837 transit testTableCreation_REALTIME.testTableCreation__13__5__20190912T1931Z|[] from:CONSUMING to:ONLINE, relayedFrom: null 2019-09-13 00:21:27 INFO HelixStateTransitionHandler:408 - Instance Server_streampinot-prod18-phx3_7090, partition testTableCreation__13__5__20190912T1931Z received state transition from CONSUMING to ONLINE on session 167860dbea60244, message id: e2fb09c1-9dad-4cab-bd09-135b25ca8837 2019-09-13 00:21:27 INFO LLRealtimeSegmentDataManager_testTableCreation__13__5__20190912T1931Z:896 - State: RETAINING, transitioning from CONSUMING to ONLINE (startOffset: 421099657, endOffset: 421474657) 2019-09-13 00:27:34 INFO LLRealtimeSegmentDataManager_testTableCreation__13__5__20190912T1931Z:667 - Trying to build segment 2019-09-13 00:31:27 WARN LLRealtimeSegmentDataManager_testTableCreation__13__5__20190912T1931Z:1019 - Failed to stop consumer thread within 10 minutes 2019-09-13 00:31:27 INFO LLRealtimeSegmentDataManager_testTableCreation__13__5__20190912T1931Z:899 - Consumer thread stopped in state RETAINING 2019-09-13 00:31:27 INFO LLRealtimeSegmentDataManager_testTableCreation__13__5__20190912T1931Z:939 - Downloading to replace segment while in state RETAINING 2019-09-13 00:31:32 INFO SegmentV1V2ToV3FormatConverter:66 - Converting segment: /var/upinot/stream-pinot-server/dataDir/testTableCreation_REALTIME/_tmp/tmp-testTableCreation__13__5__20190912T1931Z-1568334454006/testTableCreation__13__5__20190912T1931Z to v3 format 2019-09-13 00:31:40 INFO SegmentV1V2ToV3FormatConverter:87 - v3 segment location for segment: testTableCreation__13__5__20190912T1931Z is /var/upinot/stream-pinot-server/dataDir/testTableCreation_REALTIME/_tmp/tmp-testTableCreation__13__5__20190912T1931Z-1568334454006/testTableCreation__13__5__20190912T1931Z/v3 2019-09-13 00:31:40 INFO SegmentV1V2ToV3FormatConverter:93 - Deleting files in v1 segment directory: /var/upinot/stream-pinot-server/dataDir/testTableCreation_REALTIME/_tmp/tmp-testTableCreation__13__5__20190912T1931Z-1568334454006/testTableCreation__13__5__20190912T1931Z 2019-09-13 00:31:40 INFO HttpSegmentFetcher:62 - Downloaded file from: http://streampinot-prod02-phx3:5983/segments/testTableCreation/testTableCreation__13__5__20190912T1931Z to: /var/upinot/stream-pinot-server/dataDir/testTableCreation_REALTIME/testTableCreation__13__5__20190912T1931Z.tar.gz; Length of downloaded file: 1772239628; Response status code: 200 2019-09-13 00:31:40 INFO testTableCreation_REALTIME-RealtimeTableDataManager:279 - Downloaded file from http://streampinot-prod02-phx3:5983/segments/testTableCreation/testTableCreation__13__5__20190912T1931Z to /var/upinot/stream-pinot-server/dataDir/testTableCreation_REALTIME/testTableCreation__13__5__20190912T1931Z.tar.gz; Length of downloaded file: 1772239628 2019-09-13 00:31:43 INFO LLRealtimeSegmentDataManager_testTableCreation__13__5__20190912T1931Z:678 - Successfully built segment in 249488 ms, after lockWaitTime 367869 ms 2019-09-13 00:31:43 INFO testTableCreation_REALTIME-RealtimeTableDataManager:121 - Adding immutable segment: testTableCreation__13__5__20190912T1931Z to table: testTableCreation_REALTIME 2019-09-13 00:31:43 INFO testTableCreation_REALTIME-RealtimeTableDataManager:133 - Replaced immutable segment: testTableCreation__13__5__20190912T1931Z of table: testTableCreation_REALTIME 2019-09-13 00:32:04 INFO testTableCreation_REALTIME-RealtimeTableDataManager:281 - Uncompressed file /var/upinot/stream-pinot-server/dataDir/testTableCreation_REALTIME/testTableCreation__13__5__20190912T1931Z.tar.gz into tmp dir /var/upinot/stream-pinot-server/dataDir/testTableCreation_REALTIME/tmp-testTableCreation__13__5__20190912T1931Z.1568334687454 2019-09-13 00:32:05 INFO testTableCreation_REALTIME-RealtimeTableDataManager:242 - Closing segment: testTableCreation__13__5__20190912T1931Z of table: testTableCreation_REALTIME 2019-09-13 00:32:05 INFO MutableSegmentImpl_testTableCreation__13__5__20190912T1931Z_hp-api-created_trips:458 - Trying to close RealtimeSegmentImpl : testTableCreation__13__5__20190912T1931Z 2019-09-13 00:32:05 INFO MutableSegmentImpl_testTableCreation__13__5__20190912T1931Z_hp-api-created_trips:466 - Segment used 5947422696 bytes of memory for 375000 rows consumed in 17806 seconds 2019-09-13 00:32:05 INFO MmapMemoryManager:170 - Deleted file /var/upinot/stream-pinot-server/dataDir/testTableCreation_REALTIME/consumers/testTableCreation__13__5__20190912T1931Z.0 2019-09-13 00:32:05 INFO MmapMemoryManager:170 - Deleted file /var/upinot/stream-pinot-server/dataDir/testTableCreation_REALTIME/consumers/testTableCreation__13__5__20190912T1931Z.1 2019-09-13 00:32:05 INFO MmapMemoryManager:170 - Deleted file /var/upinot/stream-pinot-server/dataDir/testTableCreation_REALTIME/consumers/testTableCreation__13__5__20190912T1931Z.2 2019-09-13 00:32:05 INFO MmapMemoryManager:170 - Deleted file /var/upinot/stream-pinot-server/dataDir/testTableCreation_REALTIME/consumers/testTableCreation__13__5__20190912T1931Z.3 2019-09-13 00:32:05 INFO MmapMemoryManager:170 - Deleted file /var/upinot/stream-pinot-server/dataDir/testTableCreation_REALTIME/consumers/testTableCreation__13__5__20190912T1931Z.4 2019-09-13 00:32:05 INFO MmapMemoryManager:170 - Deleted file /var/upinot/stream-pinot-server/dataDir/testTableCreation_REALTIME/consumers/testTableCreation__13__5__20190912T1931Z.5 2019-09-13 00:32:06 INFO MmapMemoryManager:170 - Deleted file /var/upinot/stream-pinot-server/dataDir/testTableCreation_REALTIME/consumers/testTableCreation__13__5__20190912T1931Z.6 2019-09-13 00:32:06 INFO MmapMemoryManager:170 - Deleted file /var/upinot/stream-pinot-server/dataDir/testTableCreation_REALTIME/consumers/testTableCreation__13__5__20190912T1931Z.7 2019-09-13 00:32:06 INFO MmapMemoryManager:170 - Deleted file /var/upinot/stream-pinot-server/dataDir/testTableCreation_REALTIME/consumers/testTableCreation__13__5__20190912T1931Z.8 2019-09-13 00:32:06 INFO MmapMemoryManager:170 - Deleted file /var/upinot/stream-pinot-server/dataDir/testTableCreation_REALTIME/consumers/testTableCreation__13__5__20190912T1931Z.9 2019-09-13 00:32:06 INFO MmapMemoryManager:170 - Deleted file /var/upinot/stream-pinot-server/dataDir/testTableCreation_REALTIME/consumers/testTableCreation__13__5__20190912T1931Z.10 2019-09-13 00:32:06 INFO MmapMemoryManager:170 - Deleted file /var/upinot/stream-pinot-server/dataDir/testTableCreation_REALTIME/consumers/testTableCreation__13__5__20190912T1931Z.11 2019-09-13 00:32:06 INFO testTableCreation_REALTIME-RealtimeTableDataManager:248 - Closed segment: testTableCreation__13__5__20190912T1931Z of table: testTableCreation_REALTIME 2019-09-13 00:32:06 ERROR HelixStateTransitionHandler:359 - Exception while executing a state transition task testTableCreation__13__5__20190912T1931Z Caused by: java.lang.RuntimeException: org.apache.commons.io.FileExistsException: Destination '/var/upinot/stream-pinot-server/dataDir/testTableCreation_REALTIME/testTableCreation__13__5__20190912T1931Z' already exists
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
