[
https://issues.apache.org/jira/browse/PARQUET-2361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17775115#comment-17775115
]
Feng Jiajie commented on PARQUET-2361:
--------------------------------------
Hi [~amousavigourabi]
I found two issues that may cause the failures to occur easily:
# This may be a bug. The '{*}distinctStrings{*}' is used to generate test
files, but it is cleared after generating the files. It is used for probability
testing, so in the next round of generating 'fpp' files, it uses the data from
the previous round of testing, and the length of the filled strings is also
different (length 10 for test not in bloomfilter) is also different from the
initial one (strings length 12 for build bloomfilter).
# The number of test iterations is insufficient, resulting in an unstable
probability.
I attempted to submit a pull
request：https://github.com/apache/parquet-mr/pull/1170
> Reduce failure rate of unit test testParquetFileWithBloomFilterWithFpp
> ----------------------------------------------------------------------
>
> Key: PARQUET-2361
> URL: https://issues.apache.org/jira/browse/PARQUET-2361
> Project: Parquet
> Issue Type: Test
> Components: parquet-mr
> Affects Versions: 1.13.2
> Reporter: Feng Jiajie
> Priority: Major
>
> {code:java}
> [INFO] Results:
> [INFO]
> Error: Failures:
> Error: TestParquetWriter.testParquetFileWithBloomFilterWithFpp:342
> [INFO] {code}
> The unit test utilizes random string generation for test data without using a
> fixed seed. The expectation of a unit test is that the number of false
> positives in the Bloom filter should match the set probability. Therefore, a
> simple fix is to increase the number of tests on the Bloom filter. The reason
> for not using a fixed seed with random numbers is to avoid making the tests
> effective only in specific scenarios. If it is necessary to use a fixed seed,
> I can also modify the PR accordingly.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
