abyssnlp commented on issue #108:
URL: https://github.com/apache/hudi-rs/issues/108#issuecomment-2270566221
Hi @aditya-raval-genea,
I'm not a maintainer but I did go through the source and it looks like the
Python bindings rely on `HudiTable` so datafusion is not the issue here. It
should return a `List[pyarrow.RecordBatch]` when querying via python.
I tried it out and was able to query S3.
`.env` with the relevant env vars:
```
AWS_ACCESS_KEY_ID="abc"
AWS_SECRET_ACCESS_KEY="123abc"
AWS_DEFAULT_REGION="eu-central-1"
```
```python
from hudi import HudiTable
import pyarrow as pa
import duckdb
from dotenv import load_dotenv
load_dotenv()
table = HudiTable("s3://test-hudi-rs/v6_nonpartitioned/")
records = table.read_snapshot()
arrow_table = pa.Table.from_batches(records)
conn = duckdb.connect()
result = conn.execute("select * from arrow_table").arrow()
print(result)
```
It works for me and gives back the results I expect. Could you share the
logs/exceptions that you encounter?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
