JulienLrx opened a new issue, #37871:
URL: https://github.com/apache/arrow/issues/37871
### Describe the bug, including details regarding any error messages,
version, and platform.
test done on pyarrow version 13.0.0 / python 3.11.5 / hadoop 3.3.5.
In a containerized app, I'm using hostname to address my datanodes as
described in hadoop multihomed configuration. For that to work you need to tell
hdfs clients to use hostname instead of IP by setting
`dfs.client.use.datanode.hostname` to true in hdfs-site.xml.
This is working with other clients but doesn't seem to work with pyarrow. I
can make it work by passing extra_conf to the fs.HadoopFileSystem:
`hdfs = fs.HadoopFileSystem("hostname of namenode",user='yyyy', port=9000,
extra_conf={"dfs.client.use.datanode.hostname":"true"}`
But this is not very pratical as it will require a lot of code editing
instead of just changing the configuration file of the docker image of the
client application.
From other/past issue I've read, this should work, so i guess there's a
bug/issue somewhere.
(ps : etc/hadoopcontaining the hdfs-site.xml dir is in classpath)
### Component(s)
Integration, Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
