This is an automated email from the ASF dual-hosted git repository.
lzljs3620320 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/paimon.git
The following commit(s) were added to refs/heads/master by this push:
new 2c3caae04 [doc] Add python-api.md (#4175)
2c3caae04 is described below
commit 2c3caae04b6940abbae7c7a658deab5f07b20244
Author: yuzelin <[email protected]>
AuthorDate: Wed Sep 25 16:38:50 2024 +0800
[doc] Add python-api.md (#4175)
---
docs/content/program-api/catalog-api.md | 2 +-
docs/content/program-api/python-api.md | 216 ++++++++++++++++++++++++++++++++
2 files changed, 217 insertions(+), 1 deletion(-)
diff --git a/docs/content/program-api/catalog-api.md
b/docs/content/program-api/catalog-api.md
index 570577437..d016cfa7b 100644
--- a/docs/content/program-api/catalog-api.md
+++ b/docs/content/program-api/catalog-api.md
@@ -1,6 +1,6 @@
---
title: "Catalog API"
-weight: 3
+weight: 4
type: docs
aliases:
- /api/catalog-api.html
diff --git a/docs/content/program-api/python-api.md
b/docs/content/program-api/python-api.md
new file mode 100644
index 000000000..fbaa61811
--- /dev/null
+++ b/docs/content/program-api/python-api.md
@@ -0,0 +1,216 @@
+---
+title: "Python API"
+weight: 3
+type: docs
+aliases:
+- /api/python-api.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Java-based Implementation For Python API
+
+[Python SDK ](https://github.com/apache/paimon-python) has defined Python API
for Paimon. Currently, there is only a Java-based implementation.
+
+Java-based implementation will launch a JVM and use `py4j` to execute Java
code to read and write Paimon table.
+
+## Environment Settings
+
+### SDK Installing
+
+SDK is published at [paimon-python](https://pypi.org/project/paimon-python/).
You can install by
+```shell
+pip install paimon-python
+```
+
+### Java Runtime Environment
+
+This SDK needs JRE 1.8. After installing JRE, make sure that at least one of
the following conditions is met:
+1. `java` command is available. You can verify it by `java -version`.
+2. `JAVA_HOME` and `PATH` variables are set correctly. For example, you can
set:
+```shell
+export JAVA_HOME=/path/to/java-directory
+export PATH=$JAVA_HOME/bin:$PATH
+```
+
+### Set Environment Variables
+
+Because we need to launch a JVM to access Java code, JVM environment need to
be set. Besides, the java code need Hadoop
+dependencies, so hadoop environment should be set.
+
+#### Java classpath
+
+The package has set necessary paimon core dependencies (Local/Hadoop FileIO,
Avro/Orc/Parquet format support and
+FileSystem/Jdbc/Hive catalog), so If you just test codes in local or in hadoop
environment, you don't need to set classpath.
+
+If you need other dependencies such as OSS/S3 filesystem jars, or special
format and catalog ,please prepare jars and set
+classpath via one of the following ways:
+
+1. Set system environment variable: ```export
_PYPAIMON_JAVA_CLASSPATH=/path/to/jars/*```
+2. Set environment variable in Python code:
+
+```python
+import os
+from paimon_python_java import constants
+
+os.environ[constants.PYPAIMON_JAVA_CLASSPATH] = '/path/to/jars/*'
+```
+
+#### JVM args (optional)
+
+You can set JVM args via one of the following ways:
+
+1. Set system environment variable: ```export _PYPAIMON_JVM_ARGS='arg1 arg2
...'```
+2. Set environment variable in Python code:
+
+```python
+import os
+from paimon_python_java import constants
+
+os.environ[constants.PYPAIMON_JVM_ARGS] = 'arg1 arg2 ...'
+```
+
+#### Hadoop classpath
+
+If the machine is in a hadoop environment, please ensure the value of the
environment variable HADOOP_CLASSPATH include
+path to the common Hadoop libraries, then you don't need to set hadoop.
+
+Otherwise, you should set hadoop classpath via one of the following ways:
+
+1. Set system environment variable: ```export
_PYPAIMON_HADOOP_CLASSPATH=/path/to/jars/*```
+2. Set environment variable in Python code:
+
+```python
+import os
+from paimon_python_java import constants
+
+os.environ[constants.PYPAIMON_HADOOP_CLASSPATH] = '/path/to/jars/*'
+```
+
+If you just want to test codes in local, we recommend to use [Flink
Pre-bundled hadoop
jar](https://flink.apache.org/downloads/#additional-components).
+
+
+## Create Catalog
+
+Before coming into contact with the Table, you need to create a Catalog.
+
+```python
+from paimon_python_java import Catalog
+
+# Note that keys and values are all string
+catalog_options = {
+ 'metastore': 'filesystem',
+ 'warehouse': 'file:///path/to/warehouse'
+}
+catalog = Catalog.create(catalog_options)
+```
+
+## Get Table
+
+The Table interface provides tools to read and write table.
+
+```python
+table = catalog.get_table('database_name.table_name')
+```
+
+## Batch Read
+
+TableRead interface provides parallelly reading for multiple splits. You can
set `'max-workers': 'N'` in `catalog_options`
+to set thread numbers when reading splits. `max-workers` is 1 by default, that
means TableRead will read splits sequentially
+if you doesn't set `max-workers`.
+
+```python
+table = catalog.get_table('database_name.table_name')
+
+# 1. Create table scan and read
+read_builder = table.new_read_builder()
+table_scan = read_builder.new_scan()
+table_read = read_builder.new_read()
+
+# 2. Get splits
+splits = table_scan.plan().splits()
+
+# 3. Read splits. Support 3 methods:
+# 3.1 Read as pandas.DataFrame
+dataframe = table_read.to_pandas(splits)
+
+# 3.2 Read as pyarrow.Table
+pa_table = table_read.to_arrow(splits)
+
+# 3.3 Read as pyarrow.RecordBatchReader
+record_batch_reader = table_read.to_arrow_batch_reader(splits)
+```
+
+## Batch Write
+
+Paimon table write is Two-Phase Commit, you can write many times, but once
committed, no more data can be write.
+
+{{< hint warning >}}
+Currently, Python SDK doesn't support writing primary key table with
`bucket=-1`.
+{{< /hint >}}
+
+```python
+table = catalog.get_table('database_name.table_name')
+
+# 1. Create table write and commit
+write_builder = table.new_batch_write_builder()
+# By default, write data will be appended to table.
+# If you want to overwrite table:
+# write_builder.overwrite()
+# If you want to overwrite partition 'dt=2024-01-01':
+# write_builder.overwrite({'dt': '2024-01-01'})
+
+table_write = write_builder.new_write()
+table_commit = write_builder.new_commit()
+
+# 2. Write data. Support 3 methods:
+# 2.1 Write pandas.DataFrame
+dataframe = ...
+table_write.write_pandas(dataframe)
+
+# 2.2 Write pyarrow.Table
+pa_table = ...
+table_write.write_arrow(pa_table)
+
+# 2.3 Write pyarrow.RecordBatch
+record_batch = ...
+table_write.write_arrow_batch(record_batch)
+
+# 3. Commit data
+commit_messages = table_write.prepare_commit()
+table_commit.commit(commit_messages)
+
+# 4. Close resources
+table_write.close()
+table_commit.close()
+
+```
+
+## Data Types
+
+| pyarrow | Paimon |
+|:-----------------------------------------|:---------|
+| pyarrow.int8() | TINYINT |
+| pyarrow.int16() | SMALLINT |
+| pyarrow.int32() | INT |
+| pyarrow.int64() | BIGINT |
+| pyarrow.float16() <br/>pyarrow.float32() | FLOAT |
+| pyarrow.float64() | DOUBLE |
+| pyarrow.string() | STRING |
+| pyarrow.boolean() | BOOLEAN |
