[jira] [Updated] (HDDS-2443) Python client/interface for Ozone
[ https://issues.apache.org/jira/browse/HDDS-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Cheng updated HDDS-2443: --- Attachment: (was: OzoneS3.py) > Python client/interface for Ozone > - > > Key: HDDS-2443 > URL: https://issues.apache.org/jira/browse/HDDS-2443 > Project: Hadoop Distributed Data Store > Issue Type: New Feature > Components: Ozone Client >Reporter: Li Cheng >Priority: Major > Attachments: OzoneS3.py > > > Original ideas: item#25 in > [https://cwiki.apache.org/confluence/display/HADOOP/Ozone+project+ideas+for+new+contributors] > Ozone Client(Python) for Data Science Notebook such as Jupyter. > # Size: Large > # PyArrow: [https://pypi.org/project/pyarrow/] > # Python -> libhdfs HDFS JNI library (HDFS, S3,...) -> Java client API > Impala uses libhdfs > > Path to try: > # s3 interface: Ozone s3 gateway(already supported) + AWS python client > (boto3) > # python native RPC > # pyarrow + libhdfs, which use the Java client under the hood. > # python + C interface of go / rust ozone library. I created POC go / rust > clients earlier which can be improved if the libhdfs interface is not good > enough. [By [~elek]] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2443) Python client/interface for Ozone
[ https://issues.apache.org/jira/browse/HDDS-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Cheng updated HDDS-2443: --- Attachment: OzoneS3.py > Python client/interface for Ozone > - > > Key: HDDS-2443 > URL: https://issues.apache.org/jira/browse/HDDS-2443 > Project: Hadoop Distributed Data Store > Issue Type: New Feature > Components: Ozone Client >Reporter: Li Cheng >Priority: Major > Attachments: OzoneS3.py > > > Original ideas: item#25 in > [https://cwiki.apache.org/confluence/display/HADOOP/Ozone+project+ideas+for+new+contributors] > Ozone Client(Python) for Data Science Notebook such as Jupyter. > # Size: Large > # PyArrow: [https://pypi.org/project/pyarrow/] > # Python -> libhdfs HDFS JNI library (HDFS, S3,...) -> Java client API > Impala uses libhdfs > > Path to try: > # s3 interface: Ozone s3 gateway(already supported) + AWS python client > (boto3) > # python native RPC > # pyarrow + libhdfs, which use the Java client under the hood. > # python + C interface of go / rust ozone library. I created POC go / rust > clients earlier which can be improved if the libhdfs interface is not good > enough. [By [~elek]] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2443) Python client/interface for Ozone
[ https://issues.apache.org/jira/browse/HDDS-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] YiSheng Lien updated HDDS-2443: --- Description: Original ideas: item#25 in [https://cwiki.apache.org/confluence/display/HADOOP/Ozone+project+ideas+for+new+contributors] Ozone Client(Python) for Data Science Notebook such as Jupyter. # Size: Large # PyArrow: [https://pypi.org/project/pyarrow/] # Python -> libhdfs HDFS JNI library (HDFS, S3,...) -> Java client API Impala uses libhdfs Path to try: # s3 interface: Ozone s3 gateway(already supported) + AWS python client (boto3) # python native RPC # pyarrow + libhdfs, which use the Java client under the hood. # python + C interface of go / rust ozone library. I created POC go / rust clients earlier which can be improved if the libhdfs interface is not good enough. [By [~elek]] was: Original ideas: item#25 in [https://cwiki.apache.org/confluence/display/HADOOP/Ozone+project+ideas+for+new+contributors] Ozone Client(Python) for Data Science Notebook such as Jupyter. # Size: Large # PyArrow: [https://pypi.org/project/pyarrow/] # Python -> libhdfs HDFS JNI library (HDFS, S3,...) -> Java client API Impala uses libhdfs Path to try: # s3 interface: Ozone s3 gateway(already supported) + AWS python client (boto3) # python native RPC # pyarrow + libhdfs, which use the Java client under the hood. > Python client/interface for Ozone > - > > Key: HDDS-2443 > URL: https://issues.apache.org/jira/browse/HDDS-2443 > Project: Hadoop Distributed Data Store > Issue Type: New Feature > Components: Ozone Client >Reporter: Li Cheng >Priority: Major > Attachments: OzoneS3.py > > > Original ideas: item#25 in > [https://cwiki.apache.org/confluence/display/HADOOP/Ozone+project+ideas+for+new+contributors] > Ozone Client(Python) for Data Science Notebook such as Jupyter. > # Size: Large > # PyArrow: [https://pypi.org/project/pyarrow/] > # Python -> libhdfs HDFS JNI library (HDFS, S3,...) -> Java client API > Impala uses libhdfs > > Path to try: > # s3 interface: Ozone s3 gateway(already supported) + AWS python client > (boto3) > # python native RPC > # pyarrow + libhdfs, which use the Java client under the hood. > # python + C interface of go / rust ozone library. I created POC go / rust > clients earlier which can be improved if the libhdfs interface is not good > enough. [By [~elek]] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2443) Python client/interface for Ozone
[ https://issues.apache.org/jira/browse/HDDS-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Cheng updated HDDS-2443: --- Attachment: OzoneS3.py > Python client/interface for Ozone > - > > Key: HDDS-2443 > URL: https://issues.apache.org/jira/browse/HDDS-2443 > Project: Hadoop Distributed Data Store > Issue Type: New Feature > Components: Ozone Client >Reporter: Li Cheng >Priority: Major > Attachments: OzoneS3.py > > > Original ideas: item#25 in > [https://cwiki.apache.org/confluence/display/HADOOP/Ozone+project+ideas+for+new+contributors] > Ozone Client(Python) for Data Science Notebook such as Jupyter. > # Size: Large > # PyArrow: [https://pypi.org/project/pyarrow/] > # Python -> libhdfs HDFS JNI library (HDFS, S3,...) -> Java client API > Impala uses libhdfs > > Path to try: > # s3 interface: Ozone s3 gateway(already supported) + AWS python client > (boto3) > # python native RPC > # pyarrow + libhdfs, which use the Java client under the hood. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2443) Python client/interface for Ozone
[ https://issues.apache.org/jira/browse/HDDS-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] YiSheng Lien updated HDDS-2443: --- Description: Original ideas: item#25 in [https://cwiki.apache.org/confluence/display/HADOOP/Ozone+project+ideas+for+new+contributors] Ozone Client(Python) for Data Science Notebook such as Jupyter. # Size: Large # PyArrow: [https://pypi.org/project/pyarrow/] # Python -> libhdfs HDFS JNI library (HDFS, S3,...) -> Java client API Impala uses libhdfs Path to try: # s3 interface: Ozone s3 gateway(already supported) + AWS python client (boto3) # python native RPC # pyarrow + libhdfs, which use the Java client under the hood. was: Original ideas: Ozone Client(Python) for Data Science Notebook such as Jupyter. # Size: Large # PyArrow: [https://pypi.org/project/pyarrow/] # Python -> libhdfs HDFS JNI library (HDFS, S3,...) -> Java client API Impala uses libhdfs Path to try: 1. s3 interface: Ozone s3 gateway(already supported) + AWS python client (boto3) 2. python native RPC 3. pyarrow + libhdfs, which use the Java client under the hood. > Python client/interface for Ozone > - > > Key: HDDS-2443 > URL: https://issues.apache.org/jira/browse/HDDS-2443 > Project: Hadoop Distributed Data Store > Issue Type: New Feature > Components: Ozone Client >Reporter: Li Cheng >Priority: Major > > Original ideas: item#25 in > [https://cwiki.apache.org/confluence/display/HADOOP/Ozone+project+ideas+for+new+contributors] > Ozone Client(Python) for Data Science Notebook such as Jupyter. > # Size: Large > # PyArrow: [https://pypi.org/project/pyarrow/] > # Python -> libhdfs HDFS JNI library (HDFS, S3,...) -> Java client API > Impala uses libhdfs > > Path to try: > # s3 interface: Ozone s3 gateway(already supported) + AWS python client > (boto3) > # python native RPC > # pyarrow + libhdfs, which use the Java client under the hood. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2443) Python client/interface for Ozone
[ https://issues.apache.org/jira/browse/HDDS-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] YiSheng Lien updated HDDS-2443: --- Description: Original ideas: Ozone Client(Python) for Data Science Notebook such as Jupyter. # Size: Large # PyArrow: [https://pypi.org/project/pyarrow/] # Python -> libhdfs HDFS JNI library (HDFS, S3,...) -> Java client API Impala uses libhdfs Path to try: 1. s3 interface: Ozone s3 gateway(already supported) + AWS python client (boto3) 2. python native RPC 3. pyarrow + libhdfs, which use the Java client under the hood. was: Original ideas: Ozone Client(Python) for Data Science Notebook such as Jupyter. # Size: Large # PyArrow: [https://pypi.org/project/pyarrow/] # Python -> libhdfs HDFS JNI library (HDFS, S3,...) -> Java client API Impala uses libhdfs # How Jupyter iPython work: [https://jupyter.readthedocs.io/en/latest/architecture/how_jupyter_ipython_work.html] # Eco, Architecture:[https://ipython-books.github.io/chapter-3-mastering-the-jupyter-notebook/] Path to try: 1. s3 interface: Ozone s3 gateway(already supported) + AWS python client (boto3) 2. python native RPC 3. pyarrow + libhdfs, which use the Java client under the hood. > Python client/interface for Ozone > - > > Key: HDDS-2443 > URL: https://issues.apache.org/jira/browse/HDDS-2443 > Project: Hadoop Distributed Data Store > Issue Type: New Feature > Components: Ozone Client >Reporter: Li Cheng >Priority: Major > > Original ideas: > Ozone Client(Python) for Data Science Notebook such as Jupyter. > # Size: Large > # PyArrow: [https://pypi.org/project/pyarrow/] > # Python -> libhdfs HDFS JNI library (HDFS, S3,...) -> Java client API > Impala uses libhdfs > > Path to try: > 1. s3 interface: Ozone s3 gateway(already supported) + AWS python client > (boto3) > 2. python native RPC > 3. pyarrow + libhdfs, which use the Java client under the hood. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
