NikitaMatskevich commented on code in PR #5724:
URL: https://github.com/apache/datafusion-comet/pull/5724#discussion_r4584378882


##########
native/core/src/execution/operators/iceberg_write.rs:
##########
@@ -2310,8 +2313,147 @@ fn build_writer_properties(settings: 
&IcebergParquetWriteSettings) -> DFResult<W
         .set_dictionary_page_size_limit(settings.dict_size_bytes as usize)
         .set_data_page_row_count_limit(settings.page_row_limit as usize)
         .set_statistics_enabled(EnabledStatistics::Page)
-        .set_statistics_truncate_length(None)
-        .build())
+        .set_statistics_truncate_length(None);
+    for bloom_filter in &settings.bloom_filters {
+        let path = parquet_column_path(&bloom_filter.parquet_path);
+        let fpp = bloom_filter.fpp;
+        let ndv = bloom_filter.ndv;
+        let max_bytes = settings.bloom_filter_max_bytes as usize;
+        validate_bloom_filter_inputs(fpp, max_bytes)?;
+        let target_bytes = parquet_mr_bloom_filter_bytes(ndv, fpp, max_bytes);
+        let synthetic_ndv = synthetic_ndv_for_bloom_filter_bytes(target_bytes, 
fpp)?;
+        builder = builder
+            .set_column_bloom_filter_enabled(path.clone(), true)
+            .set_column_bloom_filter_fpp(path.clone(), fpp)
+            .set_column_bloom_filter_max_ndv(path, synthetic_ndv);

Review Comment:
   Thank you, fixed in commit "fix: account for Iceberg Bloom filter memory".



##########
docs/source/user-guide/latest/iceberg.md:
##########
@@ -653,13 +655,77 @@ unchanged.
 
 ### Accepted divergences
 
-Some differences between parquet-mr and the pinned parquet-rs / iceberg-rust 
are unconditional —
-they apply to every native write and cannot be configured away. Enabling
-`spark.comet.write.iceberg.enabled` accepts them. They fall into two classes 
with very
-different blast radius: differences confined to the physical bytes of a data 
file (cosmetic —
-no reader decision is based on them), and differences visible in manifest 
metadata (these
-outlive the write and feed later readers' pruning decisions, so each one is 
analyzed
-individually below).
+Enabling `spark.comet.write.iceberg.enabled` accepts the physical file-layout 
and manifest-metadata
+differences described below. Bloom-filter differences require the additional
+`spark.comet.write.iceberg.bloomFilterFolding.enabled` opt-in, because they 
can affect pruning.
+
+#### Bloom filters
+
+Comet's native Iceberg writer supports tables with column Bloom filters. 
However, byte-identical
+behavior to classic Spark cannot be guaranteed for all settings: parquet-rs 
enables folding by
+default, while Spark's non-adaptive parquet-java writer does not.
+
+Folding automatically shrinks a filter when its initial allocation from NDV or 
`max-bytes` is
+larger than the actual values need at the requested false-positive probability 
(FPP). It saves
+serialized space by shrinking toward the target FPP; Java retains the larger 
filter. Folding
+retains the original buffer capacity, so this does not imply lower peak writer 
memory.
+
+In addition to the [general native-write eligibility 
conditions](#native-parquet-write-eligibility),
+native writes for configured Bloom columns require all of the following:
+
+- `spark.comet.write.iceberg.enabled=true`.
+- `spark.comet.write.iceberg.bloomFilterFolding.enabled=true`.
+- The Iceberg runtime reports an identifiable version of 1.10 or later. 
Earlier or
+  unidentified runtimes use Java whenever a per-column Bloom property is 
configured,
+  including `false` enablement and standalone FPP/NDV properties. The opt-in 
cannot enable native Bloom writes there.
+- The effective `write.parquet.bloom-filter-max-bytes` (default 1 MiB) is a 
power of two from
+  32 bytes through 128 MiB.
+- An interpreted NDV is absent or a positive Java long no greater than 
`Long.MAX_VALUE / 8`.
+- Each interpreted FPP is finite, strictly between 0 and 1, and representable 
at the required
+  native filter sizes.
+- With a 32-byte cap and interpreted explicit NDV, the NDV/FPP request fits in 
32 bytes, because
+  Java ignores that boundary as a maximum for larger requests.
+- Bloom-filter columns use physical Parquet paths supported by the native 
writer.
+- Bloom-filter columns are not FLOAT or DOUBLE, including nested leaves. 
Floating-point columns
+  without Bloom filters remain eligible for native writes.
+
+The default Spark profiles have the following Bloom-write eligibility. The 
loaded Iceberg
+version determines eligibility; Spark's version alone does not enable this 
feature.
+
+| Spark profile | Pinned Iceberg | Bloom opt-in behavior                       
  |
+| ------------- | -------------- | 
--------------------------------------------- |
+| 3.4           | 1.5.2          | Java writer, even with the opt-in enabled   
  |
+| 3.5           | 1.8.1          | Java writer, even with the opt-in enabled   
  |
+| 4.0           | 1.10.0         | Native eligible; FPP interpreted, NDV 
ignored |
+| 4.1           | 1.11.0         | Native eligible; FPP and NDV interpreted    
  |
+
+The cap range applies whenever a `write.parquet.bloom-filter-enabled.column.*` 
property is
+present, even if set to `false`. With no such properties, the cap is unused: 
unset or any
+positive Java int, such as `100`, keeps an otherwise eligible write native. 
Malformed, zero,
+or negative caps select Java, which rejects them through its normal validation.
+
+The Bloom option defaults to `false`, selecting Java's writer for 
Bloom-enabled tables. Enabling
+it accepts byte differences and the pruning tradeoff below; it cannot turn 
folding off inside
+the pinned parquet-rs writer.
+Enabling it is generally safe when the requested FPP meets your pruning needs: 
folding preserves
+inserted-value membership and introduces no false negatives. Unsupported 
settings select Java's
+writer, where normal property validation applies. FLOAT/DOUBLE Bloom writes 
use Java until native
+writes handle NaN equality consistently with Spark. The fallback is decided 
from the schema and
+effective write properties before execution, and its reason identifies the 
affected columns.
+
+Pruning can regress when existing Java-written filters are oversized for their 
actual values
+and therefore have a much lower FPP than requested. For example, with no 
explicit NDV, a 64-MiB
+maximum and only 256 distinct values, Java can retain a 64-MiB filter with 
extra pruning accuracy.
+Rust can shrink that filter toward the target FPP and produce more false 
positives. Leave the
+option off if your workload relies on that extra accuracy. This can happen 
with an overestimated
+NDV, skew, or a partly filled row group. With explicit NDV, `max-bytes` is a 
ceiling; a large cap
+alone does not enlarge the allocation.
+
+Byte parity also requires matching inserted values, physical encodings, and 
initial sizes.
+The Bloom opt-in also accepts boolean-filter differences: parquet-java writes 
an all-zero
+filter because it does not insert boolean values, while parquet-rs writes a 
populated filter.
+Dictionary-only chunks can have a native filter where Java omits it. Comet 
follows the loaded

Review Comment:
   Thank you, documented under "Physical file layout only (cosmetic)" in commit 
"docs: explain Iceberg Bloom filter placement".



##########
spark/src/main/scala/org/apache/comet/iceberg/IcebergReflection.scala:
##########
@@ -39,6 +39,13 @@ import org.apache.comet.util.ClassLoaders
  */
 object IcebergReflection extends Logging {
 
+  case class ParquetPathResolution(
+      pathByIcebergColumnName: Map[String, String],
+      pathByFieldId: Map[Int, String],
+      physicalPaths: Set[String],

Review Comment:
   Thank you. After combining #7226 into this PR, both fields are used by 
legacy path resolution (commit "fix: normalize legacy Iceberg Bloom paths").



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to