[
https://issues.apache.org/jira/browse/PARQUET-2361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17775266#comment-17775266
]
ASF GitHub Bot commented on PARQUET-2361:
-----------------------------------------
amousavigourabi commented on PR #1170:
URL: https://github.com/apache/parquet-mr/pull/1170#issuecomment-1763024226
@fengjiajie As the test is now evaluating the false positive rate with
significantly more samples than what we use to build the filter, or are
provided as NDV, might it not be the case that this will increase the false
positive rate of the bloom filter to more than the FPP? Perhaps we could try
increasing the NDV, or maybe an adaptive bloom filter might be more
appropriate? WDYT?
> Reduce failure rate of unit test testParquetFileWithBloomFilterWithFpp
> ----------------------------------------------------------------------
>
> Key: PARQUET-2361
> URL: https://issues.apache.org/jira/browse/PARQUET-2361
> Project: Parquet
> Issue Type: Test
> Components: parquet-mr
> Affects Versions: 1.13.2
> Reporter: Feng Jiajie
> Priority: Major
>
> {code:java}
> [INFO] Results:
> [INFO]
> Error: Failures:
> Error: TestParquetWriter.testParquetFileWithBloomFilterWithFpp:342
> [INFO] {code}
> The unit test utilizes random string generation for test data without using a
> fixed seed. The expectation of a unit test is that the number of false
> positives in the Bloom filter should match the set probability. Therefore, a
> simple fix is to increase the number of tests on the Bloom filter. The reason
> for not using a fixed seed with random numbers is to avoid making the tests
> effective only in specific scenarios. If it is necessary to use a fixed seed,
> I can also modify the PR accordingly.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
