etseidl commented on code in PR #49377:
URL: https://github.com/apache/arrow/pull/49377#discussion_r2982110616
##########
python/pyarrow/_parquet.pyx:
##########
@@ -2122,6 +2128,48 @@ cdef shared_ptr[WriterProperties]
_create_writer_properties(
raise TypeError(
"'column_encoding' should be a dictionary or a string")
+ # bloom filters
+ if bloom_filter_options is not None:
+ if isinstance(bloom_filter_options, dict):
+ # for each entry in bloom_filter_options, {"path": {"ndv": ndv,
"fpp", fpp}}
+ # convert (ndv,fpp) to BloomFilterOptions struct and pass to props
+ for column, _bloom_opts in bloom_filter_options.items():
+ # set defaults
+ bloom_opts.ndv = _DEFAULT_BLOOM_FILTER_NDV
+ bloom_opts.fpp = _DEFAULT_BLOOM_FILTER_FPP
+ if isinstance(_bloom_opts, dict):
+ if "ndv" in _bloom_opts:
+ ndv = _bloom_opts["ndv"]
+ if isinstance(ndv, int):
+ if ndv < 0:
Review Comment:
I'm not sure if NDV can be 0 or not...I can't seem to find validation in
either the C++ nor rust implementations. This test just ensures it's a positive
value. Honestly, I don't see why you'd need a bloom filter for a column with no
values, so I'm fine following your suggestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
