[
https://issues.apache.org/jira/browse/ARROW-17308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17620323#comment-17620323
]
Joris Van den Bossche edited comment on ARROW-17308 at 10/20/22 8:21 AM:
-------------------------------------------------------------------------
Yes, the new Dataset API doesn't yet support the {{validate_schema}} keyword,
but the work is in progress and is tracked with ARROW-8221 (as it is a
duplicated Jira will be closing this issue).
The default in {{ParquetDataset}} is still the legacy API and so the keyword
{{validate_schema}} should work (without supplying the new
[FileSystem|https://arrow.apache.org/docs/dev/python/generated/pyarrow.fs.FileSystem.html#pyarrow.fs.FileSystem],
see:
[https://github.com/apache/arrow/blob/962121062e4b13c148f24a6d4fa4b1a2f1be0d88/python/pyarrow/parquet/core.py#L1717-L1718],
which is relevant for 9.x and I think it was the same in 6.x, could be wrong
though)
was (Author: alenkaf):
Yes, the new Dataset API doesn't yet support the {{validate_schema}} keyword,
but the work is in progress and is tracked with
https://issues.apache.org/jira/browse/ARROW-8221 (as it is a duplicated Jira
will be closing this issue).
The default in {{ParquetDataset}} is still the legacy API and so the keyword
{{validate_schema}} should work (without supplying the new
[FileSystem|https://arrow.apache.org/docs/dev/python/generated/pyarrow.fs.FileSystem.html#pyarrow.fs.FileSystem],
see:
[https://github.com/apache/arrow/blob/962121062e4b13c148f24a6d4fa4b1a2f1be0d88/python/pyarrow/parquet/core.py#L1717-L1718],
which is relevant for 9.x and I think it was the same in 6.x, could be wrong
though)
> ValueError: Keyword 'validate_schema' is not yet supported with the new
> Dataset API
> -----------------------------------------------------------------------------------
>
> Key: ARROW-17308
> URL: https://issues.apache.org/jira/browse/ARROW-17308
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Reporter: Abderrahmane Jaidi
> Priority: Major
> Labels: dataset-parquet-legacy, dataset-parquet-read
>
> Documentation for PyArrow 6.x and 7.x both indicate that the
> `validate_schema` argument is supported in the `ParquetDataset` class. Yet
> passing that argument to an instance results in:
> ValueError: Keyword 'validate_schema' is not yet supported with the new
> Dataset API
> Code:
> {code:python}
> parquet_dataset = pyarrow.parquet.ParquetDataset(
> path_or_paths=paths,
> validate_schema=validate_schema,
> filesystem=filesystem,
> partitioning=partitioning,
> use_legacy_dataset=False,
> ){code}
> Docs link:
> [https://arrow.apache.org/docs/6.0/python/generated/pyarrow.parquet.ParquetDataset.html]
> [https://arrow.apache.org/docs/7.0/python/generated/pyarrow.parquet.ParquetDataset.html]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
