[
https://issues.apache.org/jira/browse/HDDS-4977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marton Elek updated HDDS-4977:
------------------------------
Description:
Based on the
[discussion|https://lists.apache.org/thread.html/rf7dc7f417188103cd76cdc0536b11af8a089bb23e3faa388e82c7d23%40%3Cdev.ozone.apache.org%3E]
on the dev list a Golang client can help the adoption of Apache Ozone in
non-jvm environment.
Ozone go-client
Apache Ozone is an object store for Hadoop ecosystem and cloud-native
environments. As it was originally developed as a Hadoop subproject it was
implemented in Java language.
One of the power of Ozone compared to Hdfs that it can be used with multiple
interfaces: it supports Hadoop Compatible File Systems, S3 compatible
interfaces, and it also supports CSI specification (but today only with the
help of 3rd party fuse driver.
These interfaces help to use Ozone from multiple type of environments, but
Ozone provides only Java based SDK which makes harder the adoption for non JVM
languages.
This Jira proposes to create a Go based Apache Ozone SDK and additional tools
on top of it:
• With the help of a Go SDK we can create easily native Fuse drivers which
can be an additional option for the existing Ozone CSI driver (in addition to
the existing, 3rd party S3 fuse drivers)
• With the help of the Go SDK we can create a native C library which can be
used from many other languages (like Python or Rust)
*Use-cases / goals:*
* Provide Go SDK for Apache Ozone to connect to Om and Datanode for downloading
/ uploading data and managing keys/buckets/volumes view the native Ozone RPC
(Hadoop RPC and GRPC) protocols. This would be the re-implementation of Ozone
client in golang
* Provide Fuse file system implementation to mount buckets to directories
efficiently (long-term goal: use this fuse driver in Ozone CSI daemon)
* Create an adapter layer for a shared C library with the help of Ozone Go SDK
which can be used from python/rust/c/c++.
*Implementation details:*
*This is a full client side effort, no API change is required in Ozone side.*
The Go SDK would follow the design of the Java based Ozone Client:
• We need low-level Om and Datanode (Grpc and Ratis) clients
• And we can provide a higher level interface which uses those clients and
provides better programming experience
• As a first step, fuse driver can be based (github.com/hanwen/go-fuse)
which already implements the file-system like interface/abstract. This
implementation would use the low-level clients of the Go SDK.
• C library (and headers) also can be provided using the low-level API
which can be tested with Python
• A simple command line tool can be created (similar to ozone sh) which
uses the higher level interface. It can be used to test the clients from
similar approach as the Ozone main repository is tested (robotframework,
docker-compose)
• Goals for the first phase: Implement fully functional Fuse driver + CLI
for non-secure, non-HA clusters. It would be enough to test the functionality
in multiple environments (including testing it as an underlying implementation
for the CSI driver)
• In next phase we can incrementally implement HA support and security
In the current form, it’s not supposed to be released, and it should be clearly
communicated that all of these elements are just experiments and poc. Once we
have production ready SDK / Fuse driver / CLI we can start separate discussion
about release lifecycle.
Initial features (poc) are demonstrated here: https://youtu.be/ouNxn9i5DcM
was:
Based on the
[discussion|https://lists.apache.org/thread.html/rf7dc7f417188103cd76cdc0536b11af8a089bb23e3faa388e82c7d23%40%3Cdev.ozone.apache.org%3E]
on the dev list a Golang client can help the adoption of Apache Ozone in
non-jvm environment.
Ozone go-client
Apache Ozone is an object store for Hadoop ecosystem and cloud-native
environments. As it was originally developed as a Hadoop subproject it was
implemented in Java language.
One of the power of Ozone compared to Hdfs that it can be used with multiple
interfaces: it supports Hadoop Compatible File Systems, S3 compatible
interfaces, and it also supports CSI specification (but today only with the
help of 3rd party fuse driver.
These interfaces help to use Ozone from multiple type of environments but Ozone
provides only Java based SDK which makes harder the adoption for non JVM
languages.
This Jira proposes to create a Go based Apache Ozone SDK and additional tools
on top of it:
• With the help of a Go SDK we can create easily native Fuse drivers which
can be an additional option for the existing Ozone CSI driver (in addition to
the existing, 3rd party S3 fuse drivers)
• With the help of the Go SDK we can create a native C library which can be
used from many other languages (like Python or Rust)
Implementation details:
The Go SDK would follow the design of the Java based Ozone Client:
• We need low-level Om and Datanode (Grpc and Ratis) clients
• And we can provide a higher level interface which uses those clients and
provides better programming experience
• As a first step, fuse driver can be based (github.com/hanwen/go-fuse)
which already implements the file-system like interface/abstract. This
implementation would use the low-level clients of the Go SDK.
• C library (and headers) also can be provided using the low-level API
which can be tested with Python
• A simple command line tool can be created (similar to ozone sh) which
uses the higher level interface. It can be used to test the clients from
similar approach as the Ozone main repository is tested (robotframework,
docker-compose)
• Goals for the first phase: Implement fully functional Fuse driver + CLI
for non-secure, non-HA clusters. It would be enough to test the functionality
in multiple environments (including testing it as an underlying implementation
for the CSI driver)
• In next phase we ca incrementally implement HA support and security
In the current form, it’s not supposed to be released, and it should be clearly
communicated that all of these elements are just experiments and poc. Once we
have production ready SDK / Fuse driver / CLI we can start separate discussion
about release lifecycle.
Initial features (poc) are demostrated here: https://youtu.be/ouNxn9i5DcM
> Ozone go client and FUSE driver
> -------------------------------
>
> Key: HDDS-4977
> URL: https://issues.apache.org/jira/browse/HDDS-4977
> Project: Apache Ozone
> Issue Type: New Feature
> Reporter: Marton Elek
> Priority: Major
>
> Based on the
> [discussion|https://lists.apache.org/thread.html/rf7dc7f417188103cd76cdc0536b11af8a089bb23e3faa388e82c7d23%40%3Cdev.ozone.apache.org%3E]
> on the dev list a Golang client can help the adoption of Apache Ozone in
> non-jvm environment.
> Ozone go-client
> Apache Ozone is an object store for Hadoop ecosystem and cloud-native
> environments. As it was originally developed as a Hadoop subproject it was
> implemented in Java language.
> One of the power of Ozone compared to Hdfs that it can be used with multiple
> interfaces: it supports Hadoop Compatible File Systems, S3 compatible
> interfaces, and it also supports CSI specification (but today only with the
> help of 3rd party fuse driver.
> These interfaces help to use Ozone from multiple type of environments, but
> Ozone provides only Java based SDK which makes harder the adoption for non
> JVM languages.
> This Jira proposes to create a Go based Apache Ozone SDK and additional tools
> on top of it:
> • With the help of a Go SDK we can create easily native Fuse drivers
> which can be an additional option for the existing Ozone CSI driver (in
> addition to the existing, 3rd party S3 fuse drivers)
> • With the help of the Go SDK we can create a native C library which can
> be used from many other languages (like Python or Rust)
> *Use-cases / goals:*
> * Provide Go SDK for Apache Ozone to connect to Om and Datanode for
> downloading / uploading data and managing keys/buckets/volumes view the
> native Ozone RPC (Hadoop RPC and GRPC) protocols. This would be the
> re-implementation of Ozone client in golang
> * Provide Fuse file system implementation to mount buckets to directories
> efficiently (long-term goal: use this fuse driver in Ozone CSI daemon)
> * Create an adapter layer for a shared C library with the help of Ozone Go
> SDK which can be used from python/rust/c/c++.
> *Implementation details:*
> *This is a full client side effort, no API change is required in Ozone side.*
> The Go SDK would follow the design of the Java based Ozone Client:
> • We need low-level Om and Datanode (Grpc and Ratis) clients
> • And we can provide a higher level interface which uses those clients
> and provides better programming experience
> • As a first step, fuse driver can be based (github.com/hanwen/go-fuse)
> which already implements the file-system like interface/abstract. This
> implementation would use the low-level clients of the Go SDK.
> • C library (and headers) also can be provided using the low-level API
> which can be tested with Python
> • A simple command line tool can be created (similar to ozone sh) which
> uses the higher level interface. It can be used to test the clients from
> similar approach as the Ozone main repository is tested (robotframework,
> docker-compose)
> • Goals for the first phase: Implement fully functional Fuse driver + CLI
> for non-secure, non-HA clusters. It would be enough to test the functionality
> in multiple environments (including testing it as an underlying
> implementation for the CSI driver)
> • In next phase we can incrementally implement HA support and security
> In the current form, it’s not supposed to be released, and it should be
> clearly communicated that all of these elements are just experiments and poc.
> Once we have production ready SDK / Fuse driver / CLI we can start separate
> discussion about release lifecycle.
> Initial features (poc) are demonstrated here: https://youtu.be/ouNxn9i5DcM
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
