Re: ListState - elements order

[Kostas Kloudas]

Oops,

Sorry but I lost part of the discussion that had already been made.
Please ignore my previous answer.

Kostas

> On Sep 17, 2018, at 4:37 PM, Kostas Kloudas  
> wrote:
> 
> Hi all,
> 
> Flink does not provide any guarantees about the order of the elements in a 
> list and it leaves it to the state-backends.
> 
> This means that semantics between different backends may differ, and even if 
> something holds now for one of them, if 
> RocksDB or a filesystem decides to change its semantics, then any assumptions 
> will not hold anymore.
> 
> So I would recommend to program under the assumption that no order guarantees 
> are provided.
> 
> Cheers,
> Kostas
> 
>> On Sep 14, 2018, at 4:30 AM, vino yang > > wrote:
>> 
>> Hi Aljoscha,
>> 
>> Regarding window merging, as you said, it's not clear, because Flink does 
>> some internal work. 
>> But if it's just for the user, isn't it clear without any internal 
>> operations? I think if the user explicitly uses it, it should conform to the 
>> basic List semantics. Otherwise why define it instead of using SetListState 
>> directly?
>> 
>> Thanks, vino.
>> 
>> Aljoscha Krettek mailto:aljos...@apache.org>> 
>> 于2018年9月13日周四 下午10:42写道：
>> Hi,
>> 
>> this is not clearly defined anywhere, and I was always working under the 
>> assumption that the order is not preserved. This potentially allows more 
>> optimizations by the system, and for example in case of merging windows we 
>> don't know the order of elements in a ListState after a merge.
>> 
>> Best,
>> Aljoscha
>> 
>>> On 6. Sep 2018, at 08:19, vino yang >> > wrote:
>>> 
>>> Hi Alexey,
>>> 
>>> The answer is Yes, which preserves the semantics of the List's order of 
>>> elements.
>>> 
>>> Thank, vino.
>>> 
>>> Alexey Trenikhun mailto:yen...@msn.com>> 于2018年9月6日周四 
>>> 上午10:55写道：
>>> Hello,
>>> Does keyed managed ListState preserve elements order, for example if I 
>>> call listState.add(e1); listState.add(e2); listState.add(e3); , does 
>>> ListState guarantee that listState.get() will return elements in order 
>>> they were added (e1, e2, e3)
>>> 
>>> Alexey
>>> 
>>> 
>> 
> 

Re: ListState - elements order

[Kostas Kloudas]

Hi all,

Flink does not provide any guarantees about the order of the elements in a list 
and it leaves it to the state-backends.

This means that semantics between different backends may differ, and even if 
something holds now for one of them, if 
RocksDB or a filesystem decides to change its semantics, then any assumptions 
will not hold anymore.

So I would recommend to program under the assumption that no order guarantees 
are provided.

Cheers,
Kostas

> On Sep 14, 2018, at 4:30 AM, vino yang  wrote:
> 
> Hi Aljoscha,
> 
> Regarding window merging, as you said, it's not clear, because Flink does 
> some internal work. 
> But if it's just for the user, isn't it clear without any internal 
> operations? I think if the user explicitly uses it, it should conform to the 
> basic List semantics. Otherwise why define it instead of using SetListState 
> directly?
> 
> Thanks, vino.
> 
> Aljoscha Krettek mailto:aljos...@apache.org>> 
> 于2018年9月13日周四 下午10:42写道：
> Hi,
> 
> this is not clearly defined anywhere, and I was always working under the 
> assumption that the order is not preserved. This potentially allows more 
> optimizations by the system, and for example in case of merging windows we 
> don't know the order of elements in a ListState after a merge.
> 
> Best,
> Aljoscha
> 
>> On 6. Sep 2018, at 08:19, vino yang > > wrote:
>> 
>> Hi Alexey,
>> 
>> The answer is Yes, which preserves the semantics of the List's order of 
>> elements.
>> 
>> Thank, vino.
>> 
>> Alexey Trenikhun mailto:yen...@msn.com>> 于2018年9月6日周四 
>> 上午10:55写道：
>> Hello,
>> Does keyed managed ListState preserve elements order, for example if I 
>> call listState.add(e1); listState.add(e2); listState.add(e3); , does 
>> ListState guarantee that listState.get() will return elements in order 
>> they were added (e1, e2, e3)
>> 
>> Alexey
>> 
>> 
> 

Re: ListState - elements order

[Alexey Trenikhun]

“ValueState with values as string separated by the delimiter” - is not 
necessary, it can be ValueState> instead. Drawback of using ValueState 
that it is necessary to serialize whole contained object when at least one 
element is added.

Alexey

Get Outlook for iOS<https://aka.ms/o0ukef>


From: Vijay Bhaskar 
Sent: Friday, September 14, 2018 2:24 AM
To: yanghua1...@gmail.com
Cc: walter...@gmail.com; aljos...@apache.org; yen...@msn.com; 
user@flink.apache.org
Subject: Re: ListState - elements order

How it would be to use ValueState with values as string separated by the 
delimiter. So that order will never be a problem. Only overhead is to separate 
delimiter, read the elements and convert them into primitive types in case 
necessary. It just workaround. In case doesn't suite your requirements pardon me

Regards
Bhaskar

On Fri, Sep 14, 2018 at 12:36 PM vino yang 
mailto:yanghua1...@gmail.com>> wrote:
Hi,

I saw one of ListState's implementations of HeapListState, and its internal 
data store uses the JDK's List.
Of course, from an API point of view, maybe we can't make an absolute order 
guarantee.
But if we look at it from a particular implementation, we can see if it can 
guarantee this, of course, if the user is using the data structure purely.

Thanks, vino.

Rong Rong mailto:walter...@gmail.com>> 于2018年9月14日周五 
下午12:30写道：
I don't think ordering is guaranteed in the internal implementation, to the 
best of my knowledge.
I agreed with Aljoscha, if there is no clear definition of ordering, it is 
assumed to be not preserved by default.

--
Rong

On Thu, Sep 13, 2018 at 7:30 PM vino yang 
mailto:yanghua1...@gmail.com>> wrote:
Hi Aljoscha,

Regarding window merging, as you said, it's not clear, because Flink does some 
internal work.
But if it's just for the user, isn't it clear without any internal operations? 
I think if the user explicitly uses it, it should conform to the basic List 
semantics. Otherwise why define it instead of using SetListState directly?

Thanks, vino.

Aljoscha Krettek mailto:aljos...@apache.org>> 
于2018年9月13日周四 下午10:42写道：
Hi,

this is not clearly defined anywhere, and I was always working under the 
assumption that the order is not preserved. This potentially allows more 
optimizations by the system, and for example in case of merging windows we 
don't know the order of elements in a ListState after a merge.

Best,
Aljoscha

On 6. Sep 2018, at 08:19, vino yang 
mailto:yanghua1...@gmail.com>> wrote:

Hi Alexey,

The answer is Yes, which preserves the semantics of the List's order of 
elements.

Thank, vino.

Alexey Trenikhun mailto:yen...@msn.com>> 于2018年9月6日周四 上午10:55写道：
Hello,
Does keyed managed ListState preserve elements order, for example if I call 
listState.add(e1); listState.add(e2); listState.add(e3); , does ListState 
guarantee that listState.get() will return elements in order they were added 
(e1, e2, e3)

Alexey

Re: ListState - elements order

[vino yang]

Yes, for strings, we can do this, but it is not generic enough. But your
idea reminds me that we decide which data structure to use, if we use
ValueState.

Vijay Bhaskar  于2018年9月14日周五 下午5:24写道：

> How it would be to use ValueState with values as string separated by
> the delimiter. So that order will never be a problem. Only overhead is to
> separate delimiter, read the elements and convert them into primitive types
> in case necessary. It just workaround. In case doesn't suite your
> requirements pardon me
>
> Regards
> Bhaskar
>
> On Fri, Sep 14, 2018 at 12:36 PM vino yang  wrote:
>
>> Hi,
>>
>> I saw one of ListState's implementations of HeapListState, and its
>> internal data store uses the JDK's List.
>> Of course, from an API point of view, maybe we can't make an absolute
>> order guarantee.
>> But if we look at it from a particular implementation, we can see if it
>> can guarantee this, of course, if the user is using the data structure
>> purely.
>>
>> Thanks, vino.
>>
>> Rong Rong  于2018年9月14日周五 下午12:30写道：
>>
>>> I don't think ordering is guaranteed in the internal implementation, to
>>> the best of my knowledge.
>>> I agreed with Aljoscha, if there is no clear definition of ordering, it
>>> is assumed to be not preserved by default.
>>>
>>> --
>>> Rong
>>>
>>> On Thu, Sep 13, 2018 at 7:30 PM vino yang  wrote:
>>>
 Hi Aljoscha,

 Regarding window merging, as you said, it's not clear, because Flink
 does some internal work.
 But if it's just for the user, isn't it clear without any internal
 operations? I think if the user explicitly uses it, it should conform to
 the basic List semantics. Otherwise why define it instead of using
 SetListState directly?

 Thanks, vino.

 Aljoscha Krettek  于2018年9月13日周四 下午10:42写道：

> Hi,
>
> this is not clearly defined anywhere, and I was always working under
> the assumption that the order is not preserved. This potentially allows
> more optimizations by the system, and for example in case of merging
> windows we don't know the order of elements in a ListState after a merge.
>
> Best,
> Aljoscha
>
> On 6. Sep 2018, at 08:19, vino yang  wrote:
>
> Hi Alexey,
>
> The answer is Yes, which preserves the semantics of the List's order
> of elements.
>
> Thank, vino.
>
> Alexey Trenikhun  于2018年9月6日周四 上午10:55写道：
>
>> Hello,
>> Does keyed managed ListState preserve elements order, for example
>> if I call listState.add(e1); listState.add(e2); listState.add(e3); , does
>> ListState guarantee that listState.get() will return elements in order
>> they were added (e1, e2, e3)
>>
>> Alexey
>>
>>
>>
>

Re: ListState - elements order

[Vijay Bhaskar]

How it would be to use ValueState with values as string separated by the
delimiter. So that order will never be a problem. Only overhead is to
separate delimiter, read the elements and convert them into primitive types
in case necessary. It just workaround. In case doesn't suite your
requirements pardon me

Regards
Bhaskar

On Fri, Sep 14, 2018 at 12:36 PM vino yang  wrote:

> Hi,
>
> I saw one of ListState's implementations of HeapListState, and its
> internal data store uses the JDK's List.
> Of course, from an API point of view, maybe we can't make an absolute
> order guarantee.
> But if we look at it from a particular implementation, we can see if it
> can guarantee this, of course, if the user is using the data structure
> purely.
>
> Thanks, vino.
>
> Rong Rong  于2018年9月14日周五 下午12:30写道：
>
>> I don't think ordering is guaranteed in the internal implementation, to
>> the best of my knowledge.
>> I agreed with Aljoscha, if there is no clear definition of ordering, it
>> is assumed to be not preserved by default.
>>
>> --
>> Rong
>>
>> On Thu, Sep 13, 2018 at 7:30 PM vino yang  wrote:
>>
>>> Hi Aljoscha,
>>>
>>> Regarding window merging, as you said, it's not clear, because Flink
>>> does some internal work.
>>> But if it's just for the user, isn't it clear without any internal
>>> operations? I think if the user explicitly uses it, it should conform to
>>> the basic List semantics. Otherwise why define it instead of using
>>> SetListState directly?
>>>
>>> Thanks, vino.
>>>
>>> Aljoscha Krettek  于2018年9月13日周四 下午10:42写道：
>>>
 Hi,

 this is not clearly defined anywhere, and I was always working under
 the assumption that the order is not preserved. This potentially allows
 more optimizations by the system, and for example in case of merging
 windows we don't know the order of elements in a ListState after a merge.

 Best,
 Aljoscha

 On 6. Sep 2018, at 08:19, vino yang  wrote:

 Hi Alexey,

 The answer is Yes, which preserves the semantics of the List's order of
 elements.

 Thank, vino.

 Alexey Trenikhun  于2018年9月6日周四 上午10:55写道：

> Hello,
> Does keyed managed ListState preserve elements order, for example
> if I call listState.add(e1); listState.add(e2); listState.add(e3); , does
> ListState guarantee that listState.get() will return elements in order
> they were added (e1, e2, e3)
>
> Alexey
>
>
>

Re: ListState - elements order

[vino yang]

Hi,

I saw one of ListState's implementations of HeapListState, and its internal
data store uses the JDK's List.
Of course, from an API point of view, maybe we can't make an absolute order
guarantee.
But if we look at it from a particular implementation, we can see if it can
guarantee this, of course, if the user is using the data structure purely.

Thanks, vino.

Rong Rong  于2018年9月14日周五 下午12:30写道：

> I don't think ordering is guaranteed in the internal implementation, to
> the best of my knowledge.
> I agreed with Aljoscha, if there is no clear definition of ordering, it is
> assumed to be not preserved by default.
>
> --
> Rong
>
> On Thu, Sep 13, 2018 at 7:30 PM vino yang  wrote:
>
>> Hi Aljoscha,
>>
>> Regarding window merging, as you said, it's not clear, because Flink does
>> some internal work.
>> But if it's just for the user, isn't it clear without any internal
>> operations? I think if the user explicitly uses it, it should conform to
>> the basic List semantics. Otherwise why define it instead of using
>> SetListState directly?
>>
>> Thanks, vino.
>>
>> Aljoscha Krettek  于2018年9月13日周四 下午10:42写道：
>>
>>> Hi,
>>>
>>> this is not clearly defined anywhere, and I was always working under the
>>> assumption that the order is not preserved. This potentially allows more
>>> optimizations by the system, and for example in case of merging windows we
>>> don't know the order of elements in a ListState after a merge.
>>>
>>> Best,
>>> Aljoscha
>>>
>>> On 6. Sep 2018, at 08:19, vino yang  wrote:
>>>
>>> Hi Alexey,
>>>
>>> The answer is Yes, which preserves the semantics of the List's order of
>>> elements.
>>>
>>> Thank, vino.
>>>
>>> Alexey Trenikhun  于2018年9月6日周四 上午10:55写道：
>>>
 Hello,
 Does keyed managed ListState preserve elements order, for example if
 I call listState.add(e1); listState.add(e2); listState.add(e3); , does
 ListState guarantee that listState.get() will return elements in order
 they were added (e1, e2, e3)

 Alexey



>>>

Re: ListState - elements order

[Rong Rong]

I don't think ordering is guaranteed in the internal implementation, to the
best of my knowledge.
I agreed with Aljoscha, if there is no clear definition of ordering, it is
assumed to be not preserved by default.

--
Rong

On Thu, Sep 13, 2018 at 7:30 PM vino yang  wrote:

> Hi Aljoscha,
>
> Regarding window merging, as you said, it's not clear, because Flink does
> some internal work.
> But if it's just for the user, isn't it clear without any internal
> operations? I think if the user explicitly uses it, it should conform to
> the basic List semantics. Otherwise why define it instead of using
> SetListState directly?
>
> Thanks, vino.
>
> Aljoscha Krettek  于2018年9月13日周四 下午10:42写道：
>
>> Hi,
>>
>> this is not clearly defined anywhere, and I was always working under the
>> assumption that the order is not preserved. This potentially allows more
>> optimizations by the system, and for example in case of merging windows we
>> don't know the order of elements in a ListState after a merge.
>>
>> Best,
>> Aljoscha
>>
>> On 6. Sep 2018, at 08:19, vino yang  wrote:
>>
>> Hi Alexey,
>>
>> The answer is Yes, which preserves the semantics of the List's order of
>> elements.
>>
>> Thank, vino.
>>
>> Alexey Trenikhun  于2018年9月6日周四 上午10:55写道：
>>
>>> Hello,
>>> Does keyed managed ListState preserve elements order, for example if
>>> I call listState.add(e1); listState.add(e2); listState.add(e3); , does
>>> ListState guarantee that listState.get() will return elements in order
>>> they were added (e1, e2, e3)
>>>
>>> Alexey
>>>
>>>
>>>
>>

Re: ListState - elements order

[vino yang]

Hi Aljoscha,

Regarding window merging, as you said, it's not clear, because Flink does
some internal work.
But if it's just for the user, isn't it clear without any internal
operations? I think if the user explicitly uses it, it should conform to
the basic List semantics. Otherwise why define it instead of using
SetListState directly?

Thanks, vino.

Aljoscha Krettek  于2018年9月13日周四 下午10:42写道：

> Hi,
>
> this is not clearly defined anywhere, and I was always working under the
> assumption that the order is not preserved. This potentially allows more
> optimizations by the system, and for example in case of merging windows we
> don't know the order of elements in a ListState after a merge.
>
> Best,
> Aljoscha
>
> On 6. Sep 2018, at 08:19, vino yang  wrote:
>
> Hi Alexey,
>
> The answer is Yes, which preserves the semantics of the List's order of
> elements.
>
> Thank, vino.
>
> Alexey Trenikhun  于2018年9月6日周四 上午10:55写道：
>
>> Hello,
>> Does keyed managed ListState preserve elements order, for example if I
>> call listState.add(e1); listState.add(e2); listState.add(e3); , does
>> ListState guarantee that listState.get() will return elements in order
>> they were added (e1, e2, e3)
>>
>> Alexey
>>
>>
>>
>

Re: ListState - elements order

[Aljoscha Krettek]

Hi,

this is not clearly defined anywhere, and I was always working under the 
assumption that the order is not preserved. This potentially allows more 
optimizations by the system, and for example in case of merging windows we 
don't know the order of elements in a ListState after a merge.

Best,
Aljoscha

> On 6. Sep 2018, at 08:19, vino yang  wrote:
> 
> Hi Alexey,
> 
> The answer is Yes, which preserves the semantics of the List's order of 
> elements.
> 
> Thank, vino.
> 
> Alexey Trenikhun mailto:yen...@msn.com>> 于2018年9月6日周四 
> 上午10:55写道：
> Hello,
> Does keyed managed ListState preserve elements order, for example if I 
> call listState.add(e1); listState.add(e2); listState.add(e3); , does 
> ListState guarantee that listState.get() will return elements in order 
> they were added (e1, e2, e3)
> 
> Alexey
> 
> 

Re: ListState - elements order

[vino yang]

Hi Alexey,

The answer is Yes, which preserves the semantics of the List's order of
elements.

Thank, vino.

Alexey Trenikhun  于2018年9月6日周四 上午10:55写道：

> Hello,
> Does keyed managed ListState preserve elements order, for example if I
> call listState.add(e1); listState.add(e2); listState.add(e3); , does
> ListState guarantee that listState.get() will return elements in order
> they were added (e1, e2, e3)
>
> Alexey
>
>
>

ListState - elements order

[Alexey Trenikhun]

Hello,
Does keyed managed ListState preserve elements order, for example if I call 
listState.add(e1); listState.add(e2); listState.add(e3); , does ListState 
guarantee that listState.get() will return elements in order they were added 
(e1, e2, e3)

Alexey