pitrou commented on a change in pull request #8065:
URL: https://github.com/apache/arrow/pull/8065#discussion_r481256952
##########
File path: docs/source/python/filesystems.rst
##########
@@ -69,3 +138,66 @@ here is how you can read contents from a S3 bucket::
>>> f = s3.open_input_stream('my-test-bucket/Dir1/File2')
>>> f.readall()
b'some data'
+
+
+Hadoop File System (HDFS)
+-------------------------
+
+PyArrow comes with bindings to the Hadoop File System (based on C++ bindings
+using ``libhdfs``, a JNI-based interface to the Java Hadoop client). You
connect
+using the :class:`HadoopFileSystem` constructor::
+
+.. code-block:: python
+
+ from pyarrow import fs
+ hdfs = fs.HadoopFileSystem(host, port, user=user,
kerb_ticket=ticket_cache_path)
+
+The ``libhdfs`` library is loaded **at runtime** (rather than at link / library
+load time, since the library may not be in your LD_LIBRARY_PATH), and relies on
+some environment variables.
+
+* ``HADOOP_HOME``: the root of your installed Hadoop distribution. Often has
+ `lib/native/libhdfs.so`.
+
+* ``JAVA_HOME``: the location of your Java SDK installation.
+
+* ``ARROW_LIBHDFS_DIR`` (optional): explicit location of ``libhdfs.so`` if it
is
+ installed somewhere other than ``$HADOOP_HOME/lib/native``.
+
+* ``CLASSPATH``: must contain the Hadoop jars. You can set these using:
+
+ .. code-block:: shell
+
+ export CLASSPATH=`$HADOOP_HOME/bin/hdfs classpath --glob`
+
+ If ``CLASSPATH`` is not set, then it will be set automatically if the
+ ``hadoop`` executable is in your system path, or if ``HADOOP_HOME`` is set.
+
+
+Using fsspec-compatible filesystems
+-----------------------------------
+
+The filesystems mentioned above are natively supported by Arrow C++ / PyArrow.
+The Python ecosystem, however, also has several filesystem packages. Those
+packages following the
+`fsspec <https://filesystem-spec.readthedocs.io/en/latest/>`__ interface can be
+used in PyArrow as well.
+
+Functions accepting a filesystem object will also accept an fsspec subclass.
+For example::
+
+ # creating an ffspec-based filesystem object for Google Cloud Storage
Review comment:
"fsspec" :-)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
