Re: [DISCUSS] SPIP: Structured Streaming - Arbitrary State API v2

[Jungtaek Lim]

Kindly bump for better reach after the long holiday. Please kindly review
the proposal which opens the chance to address complex use cases of
streaming. Thanks!

On Thu, Nov 23, 2023 at 8:19 AM Jungtaek Lim 
wrote:

> Thanks Anish for proposing SPIP and initiating this thread! I believe this
> SPIP will help a bunch of complex use cases on streaming.
>
> dev@: We are coincidentally initiating this discussion in thanksgiving
> holidays. We understand people in the US may not have time to review the
> SPIP, and we plan to bump this thread in early next week. We are open for
> any feedback from non-US during the holiday. We can either address feedback
> altogether after the holiday (Anish is in the US) or I can answer if the
> feedback is more about the question. Thanks!
>
> On Thu, Nov 23, 2023 at 5:27 AM Anish Shrigondekar <
> anish.shrigonde...@databricks.com> wrote:
>
>> Hi dev,
>>
>> I would like to start a discussion on "Structured Streaming - Arbitrary
>> State API v2". This proposal aims to address a bunch of limitations we see
>> today using mapGroupsWithState/flatMapGroupsWithState operator. The
>> detailed set of limitations is described in the SPIP doc.
>>
>> We propose to support various features such as multiple state variables
>> (flexible data modeling), composite types, enhanced timer functionality,
>> support for chaining operators after new operator, handling initial state
>> along with state data source, schema evolution etc This will allow users to
>> write more powerful streaming state management logic primarily used in
>> operational use-cases. Other built-in stateful operators could also benefit
>> from such changes in the future.
>>
>> JIRA: https://issues.apache.org/jira/browse/SPARK-45939
>> SPIP:
>> https://docs.google.com/document/d/1QtC5qd4WQEia9kl1Qv74WE0TiXYy3x6zeTykygwPWig/edit?usp=sharing
>> Design Doc:
>> https://docs.google.com/document/d/1QjZmNZ-fHBeeCYKninySDIoOEWfX6EmqXs2lK097u9o/edit?usp=sharing
>>
>> cc - @Jungtaek Lim   who has graciously
>> agreed to be the shepherd for this project
>>
>> Looking forward to your feedback !
>>
>> Thanks,
>> Anish
>>
>

Re: [VOTE] SPIP: Testing Framework for Spark UI Javascript files

[Binjie Yang]

+ 1

Thanks,
Binjie Yang

On 2023/11/27 02:27:22 Ruifeng Zheng wrote:
> +1
> 
> On Sun, Nov 26, 2023 at 6:58 AM Gengliang Wang  wrote:
> 
> > +1
> >
> > On Sat, Nov 25, 2023 at 2:50 AM yangjie01 
> > wrote:
> >
> >> +1
> >>
> >>
> >>
> >> *发件人**: *Reynold Xin 
> >> *日期**: *2023年11月25日 星期六 14:35
> >> *收件人**: *Dongjoon Hyun 
> >> *抄送**: *Ye Zhou , Mridul Muralidharan <
> >> mri...@gmail.com>, Kent Yao , dev 
> >> *主题**: *Re: [VOTE] SPIP: Testing Framework for Spark UI Javascript files
> >>
> >>
> >>
> >> +1
> >>
> >> [image: 图像已被发件人删除。]
> >>
> >>
> >>
> >>
> >>
> >> On Fri, Nov 24, 2023 at 10:19 PM, Dongjoon Hyun 
> >> wrote:
> >>
> >> +1
> >>
> >>
> >>
> >> Thanks,
> >>
> >> Dongjoon.
> >>
> >>
> >>
> >> On Fri, Nov 24, 2023 at 7:14 PM Ye Zhou  wrote:
> >>
> >> +1(non-binding)
> >>
> >>
> >>
> >> On Fri, Nov 24, 2023 at 11:16 Mridul Muralidharan 
> >> wrote:
> >>
> >>
> >>
> >> +1
> >>
> >>
> >>
> >> Regards,
> >>
> >> Mridul
> >>
> >>
> >>
> >> On Fri, Nov 24, 2023 at 8:21 AM Kent Yao  wrote:
> >>
> >> Hi Spark Dev,
> >>
> >> Following the discussion [1], I'd like to start the vote for the SPIP [2].
> >>
> >> The SPIP aims to improve the test coverage and develop experience for
> >> Spark UI-related javascript codes.
> >>
> >> This thread will be open for at least the next 72 hours.  Please vote
> >> accordingly,
> >>
> >> [ ] +1: Accept the proposal as an official SPIP
> >> [ ] +0
> >> [ ] -1: I don’t think this is a good idea because …
> >>
> >>
> >> Thank you!
> >> Kent Yao
> >>
> >> [1] https://lists.apache.org/thread/5rqrho4ldgmqlc173y2229pfll5sgkff
> >> 
> >> [2]
> >> https://docs.google.com/document/d/1hWl5Q2CNNOjN5Ubyoa28XmpJtDyD9BtGtiEG2TT94rg/edit?usp=sharing
> >> 
> >>
> >> -
> >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
> >>
> >>
> >>
> >
> 
> -- 
> Ruifeng Zheng
> E-mail: zrfli...@gmail.com
> 

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Join push down in DSv2

[Stefan Hagedorn]

Hi,

At the Spark Summit 2017 Ioana Delaney presented an approach for join
pushdown in Apache Spark [1].

Is there any intent to actually bring this into Spark, especially in the
DSv2 interface? Does anyone know if there's ongoing work or a document
about this?

[1] https://www.youtube.com/watch?v=MDWgPK6XfEo

Thank you,
Stefan