pitrou commented on a change in pull request #11724:
URL: https://github.com/apache/arrow/pull/11724#discussion_r756903239
##########
File path: python/pyarrow/_parquet.pyx
##########
@@ -1278,11 +1297,43 @@ cdef shared_ptr[WriterProperties]
_create_writer_properties(
if isinstance(use_byte_stream_split, bool):
if use_byte_stream_split:
- props.encoding(ParquetEncoding_BYTE_STREAM_SPLIT)
+ if column_encoding is not None:
+ raise ValueError(
+ "'column_encoding' is not None")
+ else:
+ props.encoding(ParquetEncoding_BYTE_STREAM_SPLIT)
elif use_byte_stream_split is not None:
for column in use_byte_stream_split:
- props.encoding(tobytes(column),
- ParquetEncoding_BYTE_STREAM_SPLIT)
+ if column_encoding is None:
+ column_encoding = {column: 'BYTE_STREAM_SPLIT'}
+ elif column_encoding.get(column, None) is None:
+ column_encoding[column]='BYTE_STREAM_SPLIT'
+ else:
+ raise ValueError(
+ "Column {0} is already specified in 'column_encoding'"
+ .format(column))
+
+ # column_encoding
+ # encoding map - encode individual columns
+
+ if column_encoding is not None:
+ if isinstance(column_encoding, dict):
+ for column, _encoding in column_encoding.items():
+ if encoding_enum_from_name(_encoding) is None:
+ raise ValueError("Unsupported column encoding: {0}"
+ .format(_encoding))
Review comment:
It does, but I think you can also do it all-in-one:
```cython
cdef encoding_enum_from_name(str encoding_name):
enc = {
'PLAIN': ParquetEncoding_PLAIN,
'BIT_PACKED': ParquetEncoding_BIT_PACKED,
'RLE': ParquetEncoding_RLE,
'BYTE_STREAM_SPLIT': ParquetEncoding_BYTE_STREAM_SPLIT,
'DELTA_BINARY_PACKED': ParquetEncoding_DELTA_BINARY_PACKED,
'DELTA_BYTE_ARRAY': ParquetEncoding_DELTA_BYTE_ARRAY,
}.get(encoding_name)
if enc is None:
raise ValueError(f"Unsupported column encoding: {name!r}")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
