AlenkaF commented on a change in pull request #11724:
URL: https://github.com/apache/arrow/pull/11724#discussion_r756869462
##########
File path: python/pyarrow/_parquet.pyx
##########
@@ -1278,11 +1297,43 @@ cdef shared_ptr[WriterProperties]
_create_writer_properties(
if isinstance(use_byte_stream_split, bool):
if use_byte_stream_split:
- props.encoding(ParquetEncoding_BYTE_STREAM_SPLIT)
+ if column_encoding is not None:
+ raise ValueError(
+ "'column_encoding' is not None")
+ else:
+ props.encoding(ParquetEncoding_BYTE_STREAM_SPLIT)
elif use_byte_stream_split is not None:
for column in use_byte_stream_split:
- props.encoding(tobytes(column),
- ParquetEncoding_BYTE_STREAM_SPLIT)
+ if column_encoding is None:
+ column_encoding = {column: 'BYTE_STREAM_SPLIT'}
+ elif column_encoding.get(column, None) is None:
+ column_encoding[column]='BYTE_STREAM_SPLIT'
+ else:
+ raise ValueError(
+ "Column {0} is already specified in 'column_encoding'"
+ .format(column))
+
+ # column_encoding
+ # encoding map - encode individual columns
+
+ if column_encoding is not None:
+ if isinstance(column_encoding, dict):
+ for column, _encoding in column_encoding.items():
+ if encoding_enum_from_name(_encoding) is None:
+ raise ValueError("Unsupported column encoding: {0}"
+ .format(_encoding))
Review comment:
Oh, that is ugly =) I would put it in line 1320 after
https://github.com/apache/arrow/blob/19ea3980568560658629361bdf940973be07e152/python/pyarrow/_parquet.pyx#L1319
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
