[
https://issues.apache.org/jira/browse/IMPALA-11234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17532571#comment-17532571
]
ASF subversion and git services commented on IMPALA-11234:
----------------------------------------------------------
Commit 1f20b7aa51b5c93dea03515423def2d800f43554 in impala's branch
refs/heads/master from stiga-huang
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=1f20b7aa5 ]
IMPALA-11234: Bump CDP_BUILD_NUMBER to 26065553
This patch is to include HDFS-16535, which fixes the bug that HDFS clients
can't release short circuit read slots correctly when there are multiple
DataNodes colocated.
The bug also causes a performance issue when running the same query
repeatedly in our dev box (including the perf-AB-test Jenkins job).
Tests:
- Ran exhaustive build
- Verified there are no error logs about releasing short-circuit read
slots
Change-Id: I898a246f49c5ec1b34d92053849db6c3e3a0e3e3
Reviewed-on: http://gerrit.cloudera.org:8080/18495
Reviewed-by: Tamas Mate <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> impalad keeps reporting ShortCircuitCache slot release failures in heavy
> workload
> ---------------------------------------------------------------------------------
>
> Key: IMPALA-11234
> URL: https://issues.apache.org/jira/browse/IMPALA-11234
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Reporter: Quanlong Huang
> Assignee: Quanlong Huang
> Priority: Major
> Attachments: hdfs-impalad-logs.tar.gz, short-circuit-analysis.txt.gz
>
>
> I keep seeing this error during a local perf test on my desktop machine:
> {code:java}
> E0410 07:04:10.691095 430 ShortCircuitCache.java:232]
> ShortCircuitCache(0x6e76c6a7): failed to release short-circuit shared memory
> slot Slot(slotIdx=0, shm=DfsClientShm(1effcf56a590fbc371938a368987f4e9)) by
> sending ReleaseShortCircuitAccessRequestProto to
> /var/lib/hadoop-hdfs/socket.31001. Closing shared memory segment.
> Java exception follows:
> java.io.IOException: ERROR_INVALID: there is no shared memory segment
> registered with shmId 1effcf56a590fbc371938a368987f4e9
> at
> org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache$SlotReleaser.run(ShortCircuitCache.java:214)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> I can also find it in our Jenkins jobs, but it only happens in the
> data-loading phase. So I suspend it only happens in heavy workloads.
> HDFS-14701 mentioned that this happens when the DataNode is
> stopped/restarted. But I didn't restart my HDFS cluster and I'm still able to
> see this error log.
> It worth investigating if we are doing something wrong in short-circuit
> related stuffs.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
