Re: [DISCUSS] Time to evaluate "continuous mode" in SS?

It's worth noting that the push-based shuffle SPIP currently in progress addresses a substantial blocker in the area. If you remember when we removed the half-finished stateful query support, the lack of that functionality and the challenge of implementing it is basically why it was half-finished.

Re: [VOTE][SPARK-30602] SPIP: Support push-based shuffle to improve shuffle efficiency

+1 On Mon, Sep 14, 2020 at 6:39 PM angers.zhu wrote: > +1 > > angers.zhu > angers@gmail.com > >

Re: comparable and orderable CalendarInterval

The problem is that there isn't a consistent number of seconds an interval represents - as Wenchen mentioned, a month interval isn't a fixed number of days. If your use case can account for that, maybe you could add the interval to a fixed reference date and then compare the result. On Tue, Feb

Re: Welcoming some new committers and PMC members

congratulations! On Mon, Sep 9, 2019 at 6:27 PM 王 斐 wrote: > congratulations！ > > 获取 Outlook for iOS > > -- > *发件人:* Ye Xianjin > *发送时间:* 星期二, 九月 10, 2019 09:26 > *收件人:* Jeff Zhang > *抄送:* Saisai Shao; dev > *主题:* Re: Welcoming some new

Re: Partitions at DataSource API V2

The reader necessarily knows the number of partitions, since it's responsible for generating its output partitions in the first place. I won't speak for everyone, but it would make sense to me to pass in a Partitioning instance to the writer, since it's already part of the v2 interface through the

Re: [VOTE] Functional DataSourceV2 in Spark 3.0

I'm not worried about rushing. I worry that, without clear parameters for the amount or types of DSv2 delays that are acceptable, we might end up holding back 3.0 indefinitely to meet the deadline when we wouldn't have made that decision de novo. (Or even worse, the PMC eventually feels they must

Re: DSv2 question

I wouldn't be opposed to also documenting that we canonicalize the keys as lowercase, but the case-insensitivity is I think the primary property. It's important to call out that data source developers don't have to worry about a semantic difference between option("mykey", "value") and

Re: [Proposal] New feature: reconfigurable number of partitions on stateful operators in Structured Streaming

key based would > still be worth exploring to support dynamic state rebalancing. May be the > default HDFS based implementation can maintain the state partition wise and > not support it, but there could be implementations based on distributed k-v > store which supports this. > > Th

Re: [Proposal] New feature: reconfigurable number of partitions on stateful operators in Structured Streaming

excluded in query previously. > Moreover, it can’t find existing row from state or store row in wrong > partition if partition id doesn’t match the expected id via hashing > function. > > Could you verify coalesce() meets such requirements? > > On Fri, 3 Aug 2018 at 22:23 Joseph To

Re: [Proposal] New feature: reconfigurable number of partitions on stateful operators in Structured Streaming

Scheduling multiple partitions in the same task is basically what coalesce() does. Is there a reason that doesn't work here? On Fri, Aug 3, 2018 at 5:55 AM, Jungtaek Lim wrote: > Here's a link for Google docs (anyone can comment): >

Re: code freeze and branch cut for Apache Spark 2.4

Full continuous processing aggregation support ran into unanticipated scalability and scheduling problems. We’re planning to overcome those by using some of the barrier execution machinery, but since barrier execution itself is still in progress the full support isn’t going to make it into 2.4.

Re: JDBC Data Source and customSchema option but DataFrameReader.assertNoSpecifiedSchema?

I guess the question is partly about the semantics of DataFrameReader.schema. If it's supposed to mean "the loaded dataframe will definitely have exactly this schema", that doesn't quite match the behavior of the customSchema option. If it's only meant to be an arbitrary schema input which the

Re: TextSocketMicroBatchReader no longer supports nc utility

I tend to agree that this is a bug. It's kinda silly that nc does this, but a socket connector that doesn't work with netcat will surely seem broken to users. It wouldn't be a huge change to defer opening the socket until a read is actually required. On Sun, Jun 3, 2018 at 9:55 PM, Jungtaek Lim

Design proposal for streaming APIs in data source V2

Hi all, https://docs.google.com/document/d/1VzxEuvpLfuHKL6vJO9qJ6ug0x9J_gLoLSH_vJL3-Cho I've finished a full design proposal for streaming APIs in data source V2, following up on my earlier doc with just the writer. Please take a look. (Note that slightly different versions of the APIs already

Design for continuous processing shuffle

Hi all, A few of us have been working on a design for how to do shuffling in continuous processing. Feel free to chip in if you have any comments or questions. doc: https://docs.google.com/document/d/1IL4kJoKrZWeyIhklKUJqsW-yEN7V7aL05MmM65AYOfE continuous processing SPIP:

Re: [Structured streaming, V2] commit on ContinuousReader

In the master branch, we currently call this method in ContinuousExecution.commit(). Note that the ContinuousReader API is experimental and undergoing active design work. We will definitely include some kind of functionality to back-commit data once it's been processed, but the handle we

Re: Datasource API V2 and checkpointing

heckpoint directory. > > rb > > On Mon, Apr 30, 2018 at 12:37 PM, Joseph Torres < > joseph.tor...@databricks.com> wrote: > >> I'd argue that letting bad cases influence the design is an explicit goal >> of DataSourceV2. One of the primary motivations for the pr

Re: Datasource API V2 and checkpointing

a source with state that grows > indefinitely? It sounds like we're letting a bad case influence the > design, when we probably shouldn't. > > On Mon, Apr 30, 2018 at 11:05 AM, Joseph Torres < > joseph.tor...@databricks.com> wrote: > >> Offset is just a type alias for

Re: Datasource API V2 and checkpointing

't > leave unknown state on a single machine's file system. > > rb > > On Fri, Apr 27, 2018 at 9:23 AM, Joseph Torres < > joseph.tor...@databricks.com> wrote: > >> The precise interactions with the DataSourceV2 API haven't yet been >> hammered out in design.

Re: Datasource API V2 and checkpointing

The precise interactions with the DataSourceV2 API haven't yet been hammered out in design. But much of this comes down to the core of Structured Streaming rather than the API details. The execution engine handles checkpointing and recovery. It asks the streaming data source for offsets, and then

Re: [discuss][data source v2] remove type parameter in DataReader/WriterFactory

The fundamental difficulty seems to be that there's a spurious "round-trip" in the API. Spark inspects the source to determine what type it's going to provide, picks an appropriate method according to that type, and then calls that method on the source to finally get what it wants. Pushing this

Re: Spark 2.3 V2 Datasource API questions

Thanks for trying it out! We haven't hooked continuous streaming up to query.status or query.recentProgress yet - commit() should be called under the hood, we just don't yet report that it is. I've filed SPARK-23886 and SPARK-23887 to track the work to add those things. The issue with printing

Re: Beginner searching for guidance with Jira and issues

Hi! I can't speak for the other tasks, but SPARK-23444 I'd expect to be pretty complicated. It's not obvious what the right strategy is, and there's a bunch of minor stuff that needs to be cleaned up (e.g. tasks shouldn't print cancellation warnings when cancellation is expected). If you're

[DISCUSS] Structured Streaming writers in DataSourceV2

Hi all, I've been working for the past few months on figuring out a DataSourceV2 compatible interface for Structured Streaming sinks. I've written a document proposing an overarching design; please take a look and

Re: [VOTE] Spark 2.3.0 (RC2)

SPARK-23221 fixes an issue specific to KafkaContinuousSourceStressForDontFailOnDataLossSuite; I don't think it could cause other suites to deadlock. Do note that the previous hang issues we saw caused by SPARK-23055 were correctly marked as failures. On Thu, Jan 25, 2018 at 3:40 PM,