[
https://issues.apache.org/jira/browse/PARQUET-2361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17775272#comment-17775272
]
ASF GitHub Bot commented on PARQUET-2361:
-----------------------------------------
amousavigourabi commented on PR #1170:
URL: https://github.com/apache/parquet-mr/pull/1170#issuecomment-1763038724
> @fengjiajie As the test is now evaluating the false positive rate with
significantly more samples than what we use to build the filter, or are
provided as NDV, might it not be the case that this will increase the false
positive rate of the bloom filter to more than the FPP? Perhaps we could try
increasing the NDV, or maybe an adaptive bloom filter might be more
appropriate? WDYT?
Nevermind, I don't think this should be an issue. I'll be running the test
locally in a loop to see if I can reproduce the flake and check how often it
might occur.
> Reduce failure rate of unit test testParquetFileWithBloomFilterWithFpp
> ----------------------------------------------------------------------
>
> Key: PARQUET-2361
> URL: https://issues.apache.org/jira/browse/PARQUET-2361
> Project: Parquet
> Issue Type: Test
> Components: parquet-mr
> Affects Versions: 1.13.2
> Reporter: Feng Jiajie
> Priority: Major
>
> {code:java}
> [INFO] Results:
> [INFO]
> Error: Failures:
> Error: TestParquetWriter.testParquetFileWithBloomFilterWithFpp:342
> [INFO] {code}
> The unit test utilizes random string generation for test data without using a
> fixed seed. The expectation of a unit test is that the number of false
> positives in the Bloom filter should match the set probability. Therefore, a
> simple fix is to increase the number of tests on the Bloom filter. The reason
> for not using a fixed seed with random numbers is to avoid making the tests
> effective only in specific scenarios. If it is necessary to use a fixed seed,
> I can also modify the PR accordingly.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
