xhochy commented on a change in pull request #69:
URL: https://github.com/apache/arrow-datafusion/pull/69#discussion_r620234294
##########
File path: .github/workflows/python_build.yml
##########
@@ -0,0 +1,72 @@
+name: Build
Review comment:
The tag release probably won't work in the context of an ASF repo
anymore?
##########
File path: python/README.md
##########
@@ -0,0 +1,127 @@
+## DataFusion in Python
+
+This is a Python library that binds to [Apache
Arrow](https://arrow.apache.org/) in-memory query engine
[DataFusion](https://github.com/apache/arrow/tree/master/rust/datafusion).
+
+Like pyspark, it allows you to build a plan through SQL or a DataFrame API
against in-memory data, parquet or CSV files, run it in a multi-threaded
environment, and obtain the result back in Python.
+
+It also allows you to use UDFs and UDAFs for complex operations.
+
+The major advantage of this library over other execution engines is that this
library achieves zero-copy between Python and its execution engine: there is no
cost in using UDFs, UDAFs, and collecting the results to Python apart from
having to lock the GIL when running those operations.
+
+Its query engine, DataFusion, is written in
[Rust](https://www.rust-lang.org/), which makes strong assumptions about thread
safety and lack of memory leaks.
+
+Technically, zero-copy is achieved via the [c data
interface](https://arrow.apache.org/docs/format/CDataInterface.html).
+
+## How to use it
+
+Simple usage:
+
+```python
+import datafusion
+import pyarrow
+
+# an alias
+f = datafusion.functions
+
+# create a context
+ctx = datafusion.ExecutionContext()
+
+# create a RecordBatch and a new DataFrame from it
+batch = pyarrow.RecordBatch.from_arrays(
+ [pyarrow.array([1, 2, 3]), pyarrow.array([4, 5, 6])],
+ names=["a", "b"],
+)
+df = ctx.create_dataframe([[batch]])
+
+# create a new statement
+df = df.select(
+ f.col("a") + f.col("b"),
+ f.col("a") - f.col("b"),
+)
+
+# execute and collect the first (and only) batch
+result = df.collect()[0]
+
+assert result.column(0) == pyarrow.array([5, 7, 9])
+assert result.column(1) == pyarrow.array([-3, -3, -3])
+```
+
+### UDFs
+
+```python
+def is_null(array: pyarrow.Array) -> pyarrow.Array:
+ return array.is_null()
+
+udf = f.udf(is_null, [pyarrow.int64()], pyarrow.bool_())
+
+df = df.select(udf(f.col("a")))
+```
+
+### UDAF
+
+```python
+import pyarrow
+import pyarrow.compute
+
+
+class Accumulator:
+ """
+ Interface of a user-defined accumulation.
+ """
+ def __init__(self):
+ self._sum = pyarrow.scalar(0.0)
+
+ def to_scalars(self) -> [pyarrow.Scalar]:
+ return [self._sum]
+
+ def update(self, values: pyarrow.Array) -> None:
+ # not nice since pyarrow scalars can't be summed yet. This breaks on
`None`
+ self._sum = pyarrow.scalar(self._sum.as_py() +
pyarrow.compute.sum(values).as_py())
+
+ def merge(self, states: pyarrow.Array) -> None:
+ # not nice since pyarrow scalars can't be summed yet. This breaks on
`None`
+ self._sum = pyarrow.scalar(self._sum.as_py() +
pyarrow.compute.sum(states).as_py())
+
+ def evaluate(self) -> pyarrow.Scalar:
+ return self._sum
+
+
+df = ...
+
+udaf = f.udaf(Accumulator, pyarrow.float64(), pyarrow.float64(),
[pyarrow.float64()])
+
+df = df.aggregate(
+ [],
+ [udaf(f.col("a"))]
+)
+```
+
+## How to install
+
+```bash
+pip install datafusion
+```
Review comment:
Adding here as a suggestion but I'll take a look at packaging it as a
conda package. I'll cc you on the PR once I got a bit working.
```suggestion
```
or via `conda`/`mamba`:
```
conda install -c conda-forge datafusion
mamba install -c conda-forge datafusion
```
##########
File path: python/tests/test_df.py
##########
@@ -0,0 +1,98 @@
+import unittest
Review comment:
Out of curiosity: Why not `pytest`?
##########
File path: python/Cargo.toml
##########
@@ -0,0 +1,39 @@
+[package]
+name = "datafusion"
+version = "0.2.1"
+authors = ["Jorge C. Leitao <[email protected]>"]
+description = "Build and run queries against data"
+readme = "README.md"
+repository = "https://github.com/jorgecarleitao/datafusion-python";
+license = "MIT OR Apache-2.0"
+edition = "2018"
+
+[dependencies]
+tokio = "0.2.22"
+rand = "0.7"
+pyo3 = { version = "0.12.1", features = ["extension-module"] }
+datafusion = { git = "https://github.com/apache/arrow.git";, rev = "f945eba",
features = ["simd"] }
+arrow = { git = "https://github.com/apache/arrow.git";, rev = "f945eba",
features = ["simd"] }
+
+[lib]
+name = "datafusion"
+crate-type = ["cdylib"]
+
+[package.metadata.maturin]
+requires-dist = ["pyarrow>=1"]
+
+classifier = [
+ "Development Status :: 2 - Pre-Alpha",
+ "Intended Audience :: Developers",
+ "License :: OSI Approved :: Apache Software License",
+ "License :: OSI Approved",
+ "Operating System :: MacOS",
+ "Operating System :: Microsoft :: Windows",
+ "Operating System :: POSIX :: Linux",
+ "Programming Language :: Python :: 3",
+ "Programming Language :: Python :: 3.6",
+ "Programming Language :: Python :: 3.7",
+ "Programming Language :: Python :: 3.8",
Review comment:
Everything listed here should also work with Python 3.9
```suggestion
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
```
##########
File path: .github/workflows/python_test.yaml
##########
@@ -0,0 +1,41 @@
+name: Tests
+on: [push, pull_request]
+
+jobs:
+ test:
+ runs-on: ubuntu-latest
+ steps:
+ - uses: actions/checkout@v2
+ - uses: actions-rs/toolchain@v1
+ with:
+ toolchain: nightly-2020-11-24
+ default: true
+ components: rustfmt
+ - name: Cache Cargo
+ uses: actions/cache@v2
+ with:
+ path: /home/runner/.cargo
+ key: cargo-maturin-cache-
+ - name: Cache Rust dependencies
+ uses: actions/cache@v2
+ with:
+ path: /home/runner/target
+ key: target-maturin-cache-
+ - uses: actions/setup-python@v2
+ with:
+ python-version: '3.7'
+ - name: Install Python dependencies
+ run: python -m pip install --upgrade pip setuptools wheel
+ - name: Run tests
+ run: |
+ cd python/
+ export CARGO_HOME="/home/runner/.cargo"
+ export CARGO_TARGET_DIR="/home/runner/target"
+
+ python -m venv venv
+ source venv/bin/activate
+
+ pip install maturin==0.8.2 toml==0.10.1 pyarrow==1.0.0
Review comment:
`pyarrow=1.0` 😠What's holding this back?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
