Re: [VOTE] Spark 2.3.0 (RC4)

FYI. I found two more blockers: https://issues.apache.org/jira/browse/SPARK-23475 https://issues.apache.org/jira/browse/SPARK-23481 On Wed, Feb 21, 2018 at 9:45 AM, Xiao Li wrote: > Hi, Ryan, > > In this release, Data Source V2 is experimental. We are still collecting >

Re: [VOTE] Spark 2.3.0 (RC4)

What does everyone think about getting some of the newer DataSourceV2 improvements in? It should be low risk because it is a new code path, and v2 isn't very usable without things like support for using the output commit coordinator to deconflict writes. The ones I'd like to get in are: * Use the

FINAL REMINDER: CFP for Apache EU Roadshow Closes 25th February

Hello Apache Supporters and Enthusiasts This is your FINAL reminder that the Call for Papers (CFP) for the Apache EU Roadshow is closing soon. Our Apache EU Roadshow will focus on Cloud, IoT, Apache Tomcat, Apache Http and will run from 13-14 June 2018 in Berlin. Note that the CFP deadline

Re: [VOTE] Spark 2.3.0 (RC4)

Greetings, I would also like to ask if the following ticket could make it in to 2.3.0. I’m currently testing the code in production as we were running into issues on non-compacted topics (very occasionally) running into non-consecutive offsets. I imagine other people will encounter similar

Re: [VOTE] Spark 2.3.0 (RC4)

SPARK-23406 fixes a bug of a new feature in Spark 2.3, which is not a regression. I think we have to fix it in 2.3.1, but I'm less sure about 2.3.0. On Thu, Feb 22, 2018 at 1:21 AM, kant kodali wrote: > Hi All, > > +1 for the tickets proposed by Ryan Blue > > Any possible

Re: [VOTE] Spark 2.3.0 (RC4)

Hi, Ryan, In this release, Data Source V2 is experimental. We are still collecting the feedbacks from the community and will improve the related APIs and implementation in the next 2.4 release. Thanks, Xiao 2018-02-21 9:43 GMT-08:00 Xiao Li : > Hi, Justin, > > Based on

Re: [VOTE] Spark 2.3.0 (RC4)

No problem if we can't add them, this is experimental anyway so this release should be more about validating the API and the start of our implementation. I just don't think we can recommend that anyone actually use DataSourceV2 without these patches. On Wed, Feb 21, 2018 at 9:21 AM, Wenchen Fan

Re: [VOTE] Spark 2.3.0 (RC4)

Hi, Ryan, Thank you for bringing it up. Since it is in the RC4 already, we only can accept the regression fixes in the 2.3 branch. This is also the strategy in the previous Spark releases. Data source APIs V2 is newly introduced in this release. In this stage, we are unable to accept any change

Re: [VOTE] Spark 2.3.0 (RC4)

Hi All, +1 for the tickets proposed by Ryan Blue Any possible chance of this one https://issues.apache.org/jira/browse/SPARK-23406 getting into 2.3.0? It's a very important feature for us so if it doesn't make the cut I would have to cherry-pick this commit and compile from the source for our

Re: [VOTE] Spark 2.3.0 (RC4)

SPARK-23323 adds a new API, I'm not sure we can still do it at this stage of the release... Besides users can work around it by calling the spark output coordinator themselves in their data source. SPARK-23203 is non-trivial and didn't fix any known bugs, so it's hard to convince other people

Re: [VOTE] Spark 2.3.0 (RC4)

Hi, Justin, Based on my understanding, SPARK-17147 is also not a regression. Thus, Spark 2.3.0 is unable to contain it. We have to wait for the committers who are familiar with Spark Streaming to make a decision whether we can fix the issue in Spark 2.3.1. Since this is open source, feel free to

Re: [VOTE] Spark 2.3.0 (RC4)

I'm -1 on any changes that aren't fixing major regressions from 2.2 at this point. Also in any cases where its possible we should be flipping new features off if they are still regressing, rather than continuing to attempt to fix them. Since its experimental, I would support backporting the

Re: [VOTE] Spark 2.3.0 (RC4)

Ah gotcha thanks for letting me know. We’ve been using the patch in production for a couple weeks now and it’s been working great. If anyone else runs into the issue (non-compacted topics have “gaps” in offsets) feel free to have them e-mail me and I can try to help them get going with patching