[
https://issues.apache.org/jira/browse/IMPALA-11234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Quanlong Huang updated IMPALA-11234:
------------------------------------
Attachment: short-circuit-analysis.txt.gz
> impalad keeps reporting ShortCircuitCache slot release failures in heavy
> workload
> ---------------------------------------------------------------------------------
>
> Key: IMPALA-11234
> URL: https://issues.apache.org/jira/browse/IMPALA-11234
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Reporter: Quanlong Huang
> Assignee: Quanlong Huang
> Priority: Major
> Attachments: short-circuit-analysis.txt.gz
>
>
> I keep seeing this error during a local perf test on my desktop machine:
> {code:java}
> E0410 07:04:10.691095 430 ShortCircuitCache.java:232]
> ShortCircuitCache(0x6e76c6a7): failed to release short-circuit shared memory
> slot Slot(slotIdx=0, shm=DfsClientShm(1effcf56a590fbc371938a368987f4e9)) by
> sending ReleaseShortCircuitAccessRequestProto to
> /var/lib/hadoop-hdfs/socket.31001. Closing shared memory segment.
> Java exception follows:
> java.io.IOException: ERROR_INVALID: there is no shared memory segment
> registered with shmId 1effcf56a590fbc371938a368987f4e9
> at
> org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache$SlotReleaser.run(ShortCircuitCache.java:214)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> I can also find it in our Jenkins jobs, but it only happens in the
> data-loading phase. So I suspend it only happens in heavy workloads.
> HDFS-14701 mentioned that this happens when the DataNode is
> stopped/restarted. But I didn't restart my HDFS cluster and I'm still able to
> see this error log.
> It worth investigating if we are doing something wrong in short-circuit
> related stuffs.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
