pitrou commented on a change in pull request #8065:
URL: https://github.com/apache/arrow/pull/8065#discussion_r481251515
##########
File path: docs/source/python/filesystems.rst
##########
@@ -69,3 +135,66 @@ here is how you can read contents from a S3 bucket::
>>> f = s3.open_input_stream('my-test-bucket/Dir1/File2')
>>> f.readall()
b'some data'
+
+
+Hadoop File System (HDFS)
+-------------------------
+
+PyArrow comes with bindings to the Hadoop File System (based on C++ bindings
+using ``libhdfs``, a JNI-based interface to the Java Hadoop client). You
connect
+using the `class`:HadoopFileSystem: constructor::
+
+.. code-block:: python
+
+ from pyarrow import fs
+ hdfs = fs.HadoopFileSystem(host, port, user=user,
kerb_ticket=ticket_cache_path)
+
+The ``libhdfs`` library is loaded **at runtime**
+(rather than at link / library load time, since the library may not be in your
+LD_LIBRARY_PATH), and relies on some environment variables.
+
+* ``HADOOP_HOME``: the root of your installed Hadoop distribution. Often has
+ `lib/native/libhdfs.so`.
+
+* ``JAVA_HOME``: the location of your Java SDK installation.
+
+* ``ARROW_LIBHDFS_DIR`` (optional): explicit location of ``libhdfs.so`` if it
is
+ installed somewhere other than ``$HADOOP_HOME/lib/native``.
+
+* ``CLASSPATH``: must contain the Hadoop jars. You can set these using:
+
+ .. code-block:: shell
+
+ export CLASSPATH=`$HADOOP_HOME/bin/hdfs classpath --glob`
+
+ If ``CLASSPATH`` is not set, then it will be set automatically if the
+ ``hadoop`` executable is in your system path, or if ``HADOOP_HOME`` is set.
+
+
+Using fsspec-compatible filesystems
+-----------------------------------
+
+The filesystems mentioned above are natively supported by Arrow C++ / PyArrow.
+The Python ecosystem, however, also has several filesystem packages. Those
+packages following the
+`fsspec <https://filesystem-spec.readthedocs.io/en/latest/>`__ interface can be
Review comment:
Ah, thank you, I've just learned something.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
