This is an automated email from the ASF dual-hosted git repository.
xushiyan pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/hudi-rs.git
The following commit(s) were added to refs/heads/main by this push:
new c454088 docs: update readme, contributing guide, and issue template
(#57)
c454088 is described below
commit c454088f32e352309693ffcd5b408ebfe88a1064
Author: Shiyan Xu <[email protected]>
AuthorDate: Tue Jul 9 03:15:07 2024 -0500
docs: update readme, contributing guide, and issue template (#57)
- add project intro and detailed examples in README.md
- write `CONTRIBUTING.md` for all the info about making contributions to
hudi-rs
- also define an issue template to file bug
---
.github/ISSUE_TEMPLATE/bug_report.yml | 84 +++++++++++++++++++++++++
.github/ISSUE_TEMPLATE/config.yml | 36 +++++++++++
.github/workflows/ci.yml | 16 +----
CONTRIBUTING.md | 111 ++++++++++++++++++++++++++++++++++
Makefile | 31 ----------
README.md | 82 +++++++++++++++++++++++--
python/Makefile | 17 ++++++
7 files changed, 327 insertions(+), 50 deletions(-)
diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml
b/.github/ISSUE_TEMPLATE/bug_report.yml
new file mode 100644
index 0000000..d0d655b
--- /dev/null
+++ b/.github/ISSUE_TEMPLATE/bug_report.yml
@@ -0,0 +1,84 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: Bug report 🐛
+description: Report a bug to help us improve
+labels: ['bug']
+body:
+ - type: markdown
+ attributes:
+ value: |
+ Thanks for taking the time to fill out this bug report!
+ - type: checkboxes
+ id: new-bug
+ attributes:
+ label: Is there an existing issue for this?
+ description: Please search to see if an issue already exists for the bug
you encountered.
+ options:
+ - label: I have searched the existing issues
+ required: true
+ - type: textarea
+ id: bug-description
+ attributes:
+ label: Description of the bug
+ description: Tell us what bug you encountered and what should have
happened
+ validations:
+ required: true
+ - type: textarea
+ id: steps-to-reproduce
+ attributes:
+ label: Steps To Reproduce
+ description: Steps to reproduce the behavior.
+ placeholder: |
+ Please tell us how to reproduce this bug, for example:
+ 1. Build with '...' arguments
+ 2. Run with '....' arguments
+ 4. Use '...' configuration options
+ 5. See the error
+ validations:
+ required: true
+ - type: textarea
+ id: expected-behavior
+ attributes:
+ label: Expected behavior
+ description: What should be the expected behavior.
+ placeholder: A clear and concise description of what you expected to
happen.
+ validations:
+ required: true
+ - type: textarea
+ id: screenshots
+ attributes:
+ label: Screenshots / Logs
+ description: If applicable, add screenshots to help explain your problem.
+ placeholder: Paste your screenshots here.
+ - type: textarea
+ id: software-info
+ attributes:
+ label: Software information
+ description: Please provide the following information about your
environment.
+ value: |
+ - Operating system:
+ - Rust version:
+ - Project version:
+ validations:
+ required: true
+ - type: textarea
+ id: additional-context
+ attributes:
+ label: Additional context
+ description: Do you want to share any additional context about this bug?
+ placeholder: Add any other context about the problem here.
diff --git a/.github/ISSUE_TEMPLATE/config.yml
b/.github/ISSUE_TEMPLATE/config.yml
new file mode 100644
index 0000000..ac389d2
--- /dev/null
+++ b/.github/ISSUE_TEMPLATE/config.yml
@@ -0,0 +1,36 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+contact_links:
+ - name: LinkedIn updates
+ url: https://www.linkedin.com/company/apache-hudi
+ about: Get the latest updates about Hudi from LinkedIn
+ - name: X/Twitter updates
+ url: https://x.com/apachehudi
+ about: Get the latest updates about Hudi from X/Twitter
+ - name: YouTube channel
+ url: https://www.youtube.com/@apachehudi
+ about: Watch talks and guides about Hudi from YouTube
+ - name: Slack support
+ url:
https://join.slack.com/t/apache-hudi/shared_invite/zt-2ggm1fub8-_yt4Reu9djwqqVRFC7X49g
+ about: Ask questions and have discussions about Hudi on Slack
+ - name: 微信公众号 WeChat updates (Chinese)
+ url: 微信公众号搜索关注 "apachehudi"
+ about: Get the latest updates about Hudi from WeChat
+ - name: 钉钉群 DingTalk support (Chinese)
+ url: Apache Hudi交流群 - 钉钉群号 35087066
+ about: Ask questions and have discussions about Hudi on DingTalk
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index e8182c7..18d9f8e 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -37,21 +37,7 @@ jobs:
- name: Check license header
uses: apache/skywalking-eyes/[email protected]
- name: Check code style
- run: make check
-
- build:
- runs-on: ${{ matrix.os }}
- if: false # disable OS build matrix until needs arise
- strategy:
- matrix:
- os:
- - ubuntu-latest
- - macos-latest
- - windows-latest
- steps:
- - uses: actions/checkout@v4
- - name: Build
- run: cargo build
+ run: cd python && make check-rust
test:
runs-on: ubuntu-latest
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
new file mode 100644
index 0000000..0698faa
--- /dev/null
+++ b/CONTRIBUTING.md
@@ -0,0 +1,111 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# Contributing to hudi-rs
+
+Welcome to the Apache Hudi community! We appreciate your interest in
contributing to this open-source data lake
+platform. This guide will walk you through the process of making your first
contribution.
+
+## File an issue
+
+Testing and reporting bugs are also valueable contributions. Please follow
+the [issue
template](https://github.com/apache/hudi-rs/issues/new?template=bug_report.md)
to file bug reports.
+
+## Prepare for development
+
+- Install Rust, e.g. as described
[here](https://doc.rust-lang.org/cargo/getting-started/installation.html)
+- Have a compatible Python version installed (check `python/pyproject.toml`
for current requirement)
+
+## Commonly used dev commands
+
+For most of the time, use dev commands specified in `python/Makefile`, it
applies to both Python and Rust modules. You
+don't need to
+CD to the root directory and run `cargo` commands.
+
+To setup python virtual env, run
+
+```shell
+make setup-env
+```
+
+Once activate virtual env, build the project for development by
+
+```shell
+make develop
+```
+
+This will install `hudi` dependency built from your local repo to the virtual
env.
+
+## Run tests locally
+
+For Rust,
+
+```shell
+# for all tests
+make test-rust
+# or
+cargo test --workspace
+
+# for all tests in a crate / package
+cargo test -p hudi-core
+
+# for a specific test case
+cargo test -p hudi-core table::tests::hudi_table_get_schema
+```
+
+For Python,
+
+```shell
+# for all tests
+make test-python
+# or
+pytest -s
+
+# for a specific test case
+pytest tests/test_table_read.py -s -k "test_sample_table"
+```
+
+## Before creating a pull request
+
+Run check command and follow the suggestions to fix the code.
+
+```shell
+make check-rust
+```
+
+## Create a pull request
+
+### Title
+
+The pull request title must follow the format outlined in
+the [conventional commits spec](https://www.conventionalcommits.org).
[Conventional commits](https://www.conventionalcommits.org)
+is a standardized format for commit messages, and also allows us to
auto-generate change logs and release notes. Since
+only the `main` branch requires this format, and we always squash commits and
then merge the PR, incremental commits'
+messages
+do not need to conform to it.
+
+### Code coverage
+
+We use [codecov](https://app.codecov.io/github/apache/hudi-rs) to generate
code coverage report and enforce net positive
+coverage change for PRs, with a 5% lenacy.
+
+## Code of Conduct
+
+We expect all community members to follow
+our [Code of Conduct](https://www.apache.org/foundation/policies/conduct.html).
diff --git a/Makefile b/Makefile
deleted file mode 100644
index 3a5ba3f..0000000
--- a/Makefile
+++ /dev/null
@@ -1,31 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-.EXPORT_ALL_VARIABLES:
-
-RUST_LOG = debug
-
-build:
- cargo build
-
-check-fmt:
- cargo fmt --all -- --check
-
-check-clippy:
- cargo clippy --all-targets --all-features --workspace -- -D warnings
-
-check: check-fmt check-clippy
diff --git a/README.md b/README.md
index bd3b3a5..ad0c251 100644
--- a/README.md
+++ b/README.md
@@ -17,13 +17,77 @@
~ under the License.
-->
-# hudi-rs
+<p align="center">
+ <a href="https://hudi.apache.org/";>
+ <img src="https://hudi.apache.org/assets/images/hudi-logo-medium.png";
alt="Hudi logo" height="80px">
+ </a>
+</p>
+<p align="center">
+ A native Rust library for Apache Hudi, with bindings to Python
+ <br>
+ <br>
+ <a href="https://github.com/apache/hudi-rs/actions/workflows/ci.yml";>
+ <img alt="hudi-rs ci"
src="https://github.com/apache/hudi-rs/actions/workflows/ci.yml/badge.svg";>
+ </a>
+ <a href="https://codecov.io/github/apache/hudi-rs";>
+ <img alt="hudi-rs codecov"
src="https://codecov.io/github/apache/hudi-rs/graph/badge.svg";>
+ </a>
+ <a
href="https://join.slack.com/t/apache-hudi/shared_invite/zt-2ggm1fub8-_yt4Reu9djwqqVRFC7X49g";>
+ <img alt="join hudi slack"
src="https://img.shields.io/badge/slack-%23hudi-72eff8?logo=slack&color=48c628";>
+ </a>
+ <a href="https://x.com/apachehudi";>
+ <img alt="follow hudi x/twitter"
src="https://img.shields.io/twitter/follow/apachehudi?label=apachehudi";>
+ </a>
+ <a href="https://www.linkedin.com/company/apache-hudi";>
+ <img alt="follow hudi linkedin"
src="https://img.shields.io/badge/apache%E2%80%93hudi-0077B5?logo=linkedin";>
+ </a>
+</p>
-[](https://codecov.io/github/apache/hudi-rs)
+The `hudi-rs` project aims to broaden the use of [Apache
Hudi](https://github.com/apache/hudi) for a diverse range of
+users and projects.
-## Quick Start
+| Source | Installation Command |
+|---------------|----------------------|
+| **PyPi** | `pip install hudi` |
+| **Crates.io** | `cargo add hudi` |
-### Apache DataFusion
+*Note: not yet available until the first release.*
+
+## Example usage
+
+### Python
+
+Add `python/hudi` as a dependency to your Python application.
+
+For example, use these commands to setup a virtualenv.
+
+```shell
+cd python
+python3 -m venv my_venv
+source my_venv/bin/activate
+pip install -e .
+```
+
+Then query a Hudi table.
+
+```python
+from hudi import HudiTable
+
+hudi_table = HudiTable("/tmp/trips_table")
+records = hudi_table.read_snapshot()
+
+import pyarrow as pa
+
+arrow_table = pa.Table.from_batches(records)
+result = arrow_table.select(
+ ["rider", "ts", "fare"]).filter(
+ pa.compute.field("fare") > 20.0)
+print(result)
+```
+
+### Rust
+
+Add `hudi-datafusion` as a dependency to your Rust application, and query a
Hudi table.
```rust
use std::sync::Arc;
@@ -43,3 +107,13 @@ async fn main() -> Result<()> {
Ok(())
}
```
+
+### Work with cloud storage
+
+Ensure cloud storage credentials are set properly as environment variables,
e.g., `AWS_*`, `AZURE_*`, or `GOOGLE_*`.
+Relevant storage environment variables will then be picked up. The target
table's base uri with schemes such
+as `s3://`, `az://`, or `gs://` will be processed accordingly.
+
+## Contributing
+
+Check out the [contributing guide](./CONTRIBUTING.md) for all the details
about making contributions to the project.
diff --git a/python/Makefile b/python/Makefile
index 8017d2f..4badaca 100644
--- a/python/Makefile
+++ b/python/Makefile
@@ -40,3 +40,20 @@ build: setup ## Build Python binding of delta-rs
develop: setup ## Install Python binding of delta-rs
$(info --- Develop with Python binding ---)
maturin develop --extras=devel,pandas $(MATURIN_EXTRA_ARGS)
+
+.PHONY: check-rust
+check-rust: ## Run check on Rust
+ $(info --- Check Rust clippy ---)
+ cargo clippy --all-targets --all-features --workspace -- -D warnings
+ $(info --- Check Rust format ---)
+ cargo fmt --all -- --check
+
+.PHONY: test-rust
+test-rust: ## Run tests on Rust
+ $(info --- Run Rust tests ---)
+ cargo test --no-fail-fast --all-targets --all-features --workspace
+
+.PHONY: test-python
+test-python: ## Run tests on Python
+ $(info --- Run Python tests ---)
+ pytest -s
![https://hudi.apache.org/assets/images/hudi-logo-medium.png"](https://hudi.apache.org/assets/images/hudi-logo-medium.png"){kind=link}
![https://github.com/apache/hudi-rs/actions/workflows/ci.yml/badge.svg"](https://github.com/apache/hudi-rs/actions/workflows/ci.yml/badge.svg"){kind=link}
![https://codecov.io/github/apache/hudi-rs/graph/badge.svg"](https://codecov.io/github/apache/hudi-rs/graph/badge.svg"){kind=link}
