[
https://issues.apache.org/jira/browse/METRON-728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15880920#comment-15880920
]
ASF GitHub Bot commented on METRON-728:
---------------------------------------
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/463
@mmiklavc The stream can consider itself parallel without it actually
functioning in parallel. If we have one big split, only one thread will be
used in the threadpool, so it'll be de facto serial. Also, we don't actually
implement Stream, we *construct* a stream from a Spliterator.
The issue that @justinleet alluded to was that we were testing
`Spliterator` indirectly by testing `Stream`s in a threadpool. If we just did
the direct test, we'd narrow the scope and remove the variability as well. If
we are, then we should be able to trust `Stream.parallel()` to do the right
thing rather than testing that. What we really want to know is are we
splitting properly.
> ReaderSpliteratorTest fails randomly and extremely rarely
> ---------------------------------------------------------
>
> Key: METRON-728
> URL: https://issues.apache.org/jira/browse/METRON-728
> Project: Metron
> Issue Type: Bug
> Affects Versions: 0.3.1
> Reporter: Justin Leet
> Assignee: Justin Leet
>
> See logs at
> https://travis-ci.org/justinleet/incubator-metron/builds/203298348
> I was able to reproduce this locally by calling
> {{testActuallyParallel_mediumBatch}} in a {{while(true)}} loop. It can also
> occur in {{testActuallyParallel_mediumBatchImplicitlyParallel()}}. I also
> had to add
> {{forkJoinPool.shutdownNow();}} to the end of the test, because otherwise OOM
> errors occur.
> My current assumption is that there's no guarantee you ever actually end up
> running in parallel, so in extremely rare cases you just end up running one
> thread.
> I've had it vary wildly when I hit it, from within a second or two to running
> for over a minute before an assertion failure occurs.
> We could just alter the assertion to be
> {code}Assert.assertTrue(threads.size() <= (int) Math.ceil(9.0 / 2) &&
> threads.size() >= 1);{code}
> This defeats the purpose of testing the parallelism a bit, but if there's no
> guarantee we actually get parallelism there's not a fantastic way to test it.
> Given the extreme rarity, we might want to just live with the fact that
> occasionally {{threads.size() >= 1}} hits the threads.size() == 1 case on the
> two tests.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
