Fred Tzeng created HDFS-14676:
---------------------------------
Summary: Calls to libhdfs (via pyarrow) hang after a while
Key: HDFS-14676
URL: https://issues.apache.org/jira/browse/HDFS-14676
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs, libhdfs
Affects Versions: 3.0.3
Environment: hadoop-3.0.3
python 3.6
Centos7
Reporter: Fred Tzeng
I'm using the pyarrow HDFS client in a long running (forever) app that makes
connections to HDFS (via libhdfs) as external requests come in and destroys the
connection as soon as the request is handled. This happens a large amount of
times on separate threads and everything works great.
The problem is, after the app idles for a while (perhaps hours) and no HDFS
connections are made during this time, when the next connection is attempted,
it hangs. No exceptions are thrown. As soon as I restart my python app, the
HDFS connection works just fine again.
I'm using the precompiled libhdfs.so directly from the hadoop-3.0.3
distribution. Do I typically need to recompile libhdfs.so for my OS, or is the
one out of the box typically fine?
I've checked with the Arrow community first- they've recommended I check with
the Hadoop community since all the pyarrow client does is pass through the
commands to libhdfs.
Any suggestions on debugging this hanging issue would be appreciated.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
