Hi Mich, Yes, you may think of cube rollups. Let me try to give an example: If we have a stream of data like (country,area,count, time), we would be able to get the updated count with different combinations of keys.
> As example - > (country - count) > (country , area - count) We may need to store the state to update the count. So spark structured streaming states will come into picture. As now with batch programming, we can do it with > df.rollup(col1,col2).count But if I try to use it with spark structured streaming state, will it store the state of all the groups as well? I hope I was able to make my point clear. Regards Amit Joshi On Wed, Jun 16, 2021 at 11:36 PM Mich Talebzadeh <mich.talebza...@gmail.com> wrote: > > > Hi, > > Just to clarify > > Are we talking about* rollup* as a subset of a cube that computes > hierarchical subtotals from left to right? > > > > > > view my Linkedin profile > <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> > > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > > > > On Wed, 16 Jun 2021 at 16:37, Amit Joshi <mailtojoshia...@gmail.com> > wrote: > >> Appreciate if someone could give some pointers in the question below. >> >> ---------- Forwarded message --------- >> From: Amit Joshi <mailtojoshia...@gmail.com> >> Date: Tue, Jun 15, 2021 at 12:19 PM >> Subject: [Spark]Does Rollups work with spark structured streaming with >> state. >> To: spark-user <user@spark.apache.org> >> >> >> Hi Spark-Users, >> >> Hope you are all doing well. >> Recently I was looking into rollup operations in spark. >> >> As we know state based aggregation is supported in spark structured >> streaming. >> I was wondering if rollup operations are also supported? >> Like the state of previous aggregation on the rollups are saved. >> >> If rollups are not supported, then what is the standard way to handle >> this? >> >> >> Regards >> Amit Joshi >> >
