hadrian-reppas commented on issue #46629: URL: https://github.com/apache/arrow/issues/46629#issuecomment-3020425208
[This](https://github.com/hadrian-reppas/arrow/commit/62c48a1dfa3145bbf75e54dfbce73ea31c021047) commit allows me to write ```python factory = ds.FileSystemDatasetFactory(fs.LocalFileSystem(), [path1, path2], ds.ParquetFileFormat()) schema = factory.inspect("permissive") dataset = factory.finish(schema) table = dataset.to_table() ``` (just like C++ example above) and get the desired behavior. It also gives correct results for other parameters when I tested in the REPL. Two issues: - I can't figure out how to access the static constexpr `kInspectAllFragments` in `InspectOptions`, so I just hardcoded `-1`. - This branches off of `maint-20.0.0` because when I branch off main I get the following error trying to import `pyarrow.dataset`: ``` >>> import pyarrow.dataset as ds <frozen importlib._bootstrap>:241: RuntimeWarning: pyarrow.lib.IpcReadOptions size changed, may indicate binary incompatibility. Expected 104 from C header, got 112 from PyObject Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Users/hadrian/dev/arrow/python/pyarrow/dataset.py", line 85, in <module> from pyarrow._dataset_parquet import ( # noqa File "pyarrow/_dataset_parquet.pyx", line 57, in init pyarrow._dataset_parquet from pyarrow._dataset_parquet_encryption import ( File "pyarrow/_dataset_parquet_encryption.pyx", line 1, in init pyarrow._dataset_parquet_encryption # Licensed to the Apache Software Foundation (ASF) under one AttributeError: module 'pyarrow.lib' has no attribute 'PyExtensionType'. Did you mean: 'ExtensionType'? ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
