[
https://issues.apache.org/jira/browse/FLINK-18909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17183720#comment-17183720
]
Jingsong Lee edited comment on FLINK-18909 at 8/25/20, 4:20 AM:
----------------------------------------------------------------
Thanks [~aljoscha] for reminding. Hi [~Liquidcoder] , SequenceGenerator can not
accept too huge range.
The {{SequenceGenerator}} and {{StatefulSequenceSource}} have the same
generator strategy. At first, they store all numbers to {{Deque<Long>
valuesToEmit}} . So if the range is to huge, there will be a OOM here... We can
have a better exception message.
I believe we can optimize it to not store all numbers, after the optimization,
it can accept Long.MAX_VALUE.
was (Author: lzljs3620320):
SequenceGenerator can not accept too huge range.
The {{SequenceGenerator}} and {{StatefulSequenceSource}} have the same
generator strategy. At first, they store all numbers to {{Deque<Long>
valuesToEmit}} . So if the range is to huge, there will be a OOM here... We can
have a better exception message.
I believe we can optimize it to not store all numbers, after the optimization,
it can accept Long.MAX_VALUE.
> SequenceGenerator doesn't work as expected
> ------------------------------------------
>
> Key: FLINK-18909
> URL: https://issues.apache.org/jira/browse/FLINK-18909
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
> Affects Versions: 1.11.1
> Reporter: Alan
> Priority: Minor
>
> When using a
> org.apache.flink.streaming.api.functions.source.datagen.SequenceGenerator
> with 0 and Long.MaxValue the following exception is rasied:
> <<snip>>
> new SequenceGenerator(0, Long.MAX_VALUE);
>
> Caused by: java.lang.IllegalArgumentExceptionCaused by:
> java.lang.IllegalArgumentException at
> org.apache.flink.util.Preconditions.checkArgument(Preconditions.java:123) at
> org.apache.flink.streaming.api.functions.source.datagen.SequenceGenerator.safeDivide(SequenceGenerator.java:110)
> at
> org.apache.flink.streaming.api.functions.source.datagen.SequenceGenerator.open(SequenceGenerator.java:83)
> at
> org.apache.flink.streaming.api.functions.source.datagen.DataGeneratorSource.initializeState(DataGeneratorSource.java:64)
>
> This is because the Long overflows. in the check:
> long totalNoOfElements = Math.abs(end - start + 1);
> Which is incorrect. - this code should use BigInteger.
>
> Using
> new SequenceGenerator(0, Long.MAX_VALUE-1);
> Still fails on the following check:
> Caused by: java.lang.IllegalArgumentExceptionCaused by:
> java.lang.IllegalArgumentException at
> org.apache.flink.util.Preconditions.checkArgument(Preconditions.java:123) at
> org.apache.flink.streaming.api.functions.source.datagen.SequenceGenerator.safeDivide(SequenceGenerator.java:111)
> at
> org.apache.flink.streaming.api.functions.source.datagen.SequenceGenerator.open(SequenceGenerator.java:83)
> at
> org.apache.flink.streaming.api.functions.source.datagen.DataGeneratorSource.initializeState(DataGeneratorSource.java:64)
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
