Michael,
I have listed used cases above should we proceed with a design doc?
Best,
Stavros
Στις Δευ, 18 Μαρ 2019, 12:21 μ.μ. ο χρήστης Stavros Kontopoulos <
stavros.kontopou...@lightbend.com> έγραψε:
> Not really, if we agree that we want this, I can put together a design
> document and take it
Not really, if we agree that we want this, I can put together a design
document and take it from there. There was also a discussion in another
thread about adding RockDB as a memory storage that is related to this task.
Best,
Stavros
On Sun, Mar 17, 2019 at 4:42 AM kant kodali wrote:
> Any upda
Any update on this?
On Wed, Oct 24, 2018 at 4:26 PM Arun Mahadevan wrote:
> I don't think separate API or RPCs etc might be necessary for queryable
> state if the state can be exposed as just another datasource. Then the sql
> queries can be issued against it just like executing sql queries agai
I don't think separate API or RPCs etc might be necessary for queryable
state if the state can be exposed as just another datasource. Then the sql
queries can be issued against it just like executing sql queries against
any other data source.
For now I think the "memory" sink could be used as a s
It doesn't seem Spark has workarounds other than storing output into
external storages, so +1 on having this.
My major concern on implementing queryable state in structured streaming is
"Are all states available on executors at any time while query is running?"
Querying state shouldn't affect the
Nice I was looking for a jira. So I agree we should justify why we are
building something. Now to that direction here is what I have seen from my
experience.
People quite often use state within their streaming app and may have large
states (TBs). Shortening the pipeline by not having to copy data (
https://issues.apache.org/jira/browse/SPARK-16738
I don't believe anyone is working on it yet. I think the most useful thing
is to start enumerating requirements and use cases and then we can talk
about how to build it.
On Fri, Dec 8, 2017 at 10:47 AM, Stavros Kontopoulos <
st.kontopou...@gmail.
Cool Burak do you have a pointer, should I take the initiative for a first
design document or Databricks is working on it?
Best,
Stavros
On Fri, Dec 8, 2017 at 8:40 PM, Burak Yavuz wrote:
> Hi Stavros,
>
> Queryable state is definitely on the roadmap! We will revamp the
> StateStore API a bit,
Hi Stavros,
Queryable state is definitely on the roadmap! We will revamp the StateStore
API a bit, and a queryable StateStore is definitely one of the things we
are thinking about during that revamp.
Best,
Burak
On Dec 8, 2017 9:57 AM, "Stavros Kontopoulos"
wrote:
> Just to re-phrase my questi
Just to re-phrase my question: Would query-able state make a viable SPIP?
Regards,
Stavros
On Thu, Dec 7, 2017 at 1:34 PM, Stavros Kontopoulos <
st.kontopou...@gmail.com> wrote:
> Hi,
>
> Maybe this has been discussed before. Given the fact that many streaming
> apps out there use state extensiv
Hi,
Maybe this has been discussed before. Given the fact that many streaming
apps out there use state extensively, could be a good idea to make Spark
expose streaming state with an external API like other systems do (Kafka
streams, Flink etc), in order to facilitate interactive queries?
Regards,
11 matches
Mail list logo