Hi Jungtaek, That's very helpful to have the state source. As a matter of fact I've just this week been working on a similar tool (!) and have been wondering how to recreate the schema of the state key and value. You've helped me a lot. Thanks.
Jacek On Wed, 26 Jun 2019, 23:58 Jungtaek Lim, <[email protected]> wrote: > Hi, > > you could consider state operator's partition numbers as "max > parallelism", as parallelism can be reduced via applying coalesce. It would > be effectively working similar as key groups. > > If you're also considering offline query, there's a tool to manipulate > state which enables reading and writing state in structured streaming, > achieving rescaling and schema evolution. > > https://github.com/HeartSaVioR/spark-state-tools > (DISCLAIMER: I'm an author of this tool.) > > Thanks, > Jungtaek Lim (HeartSaVioR) > > On Thu, Jun 27, 2019 at 4:48 AM Rong, Jialei <[email protected]> > wrote: > >> Thank you for your quick reply! >> >> Is there any plan to improve this? >> >> I asked this question due to some investigation on comparing those state >> of art streaming systems, among which Flink and DataFlow allow changing >> parallelism number, and by my knowledge of Spark Streaming, it seems it is >> also able to do that: if some “key interval” concept is used, then state >> can somehow decoupled from partition number by consistent hashing. >> >> >> >> >> >> Regards >> >> Jialei >> >> >> >> *From: *Jacek Laskowski <[email protected]> >> *Date: *Wednesday, June 26, 2019 at 11:00 AM >> *To: *"Rong, Jialei" <[email protected]> >> *Cc: *"user @spark" <[email protected]> >> *Subject: *Re: Change parallelism number in Spark Streaming >> >> >> >> Hi, >> >> >> >> It's not allowed to change the numer of partitions after your streaming >> query is started. >> >> >> >> The reason is exactly the number of state stores which is exactly the >> number of partitions (perhaps multiplied by the number of stateful >> operators). >> >> >> >> I think you'll even get a warning or an exception when you change it >> after restarting the query. >> >> >> >> The number of partitions is stored in a checkpoint location. >> >> >> >> Jacek >> >> >> >> On Wed, 26 Jun 2019, 19:30 Rong, Jialei, <[email protected]> >> wrote: >> >> Hi Dear Spark Expert >> >> >> >> I’m curious about a question regarding Spark Streaming/Structured >> Streaming: whether it allows to change parallelism number(the default one >> or the one specified in particular operator) in a stream having stateful >> transform/operator? Whether this will cause my checkpointed state get >> messed up? >> >> >> >> >> >> Regards >> >> Jialei >> >> >> >> > > -- > Name : Jungtaek Lim > Blog : http://medium.com/@heartsavior > Twitter : http://twitter.com/heartsavior > LinkedIn : http://www.linkedin.com/in/heartsavior >
