Hi, Justin, Based on my understanding, SPARK-17147 is also not a regression. Thus, Spark 2.3.0 is unable to contain it. We have to wait for the committers who are familiar with Spark Streaming to make a decision whether we can fix the issue in Spark 2.3.1.
Since this is open source, feel free to add the patch in your local build. Thanks for using Spark! Xiao 2018-02-21 9:36 GMT-08:00 Ryan Blue <rb...@netflix.com.invalid>: > No problem if we can't add them, this is experimental anyway so this > release should be more about validating the API and the start of our > implementation. I just don't think we can recommend that anyone actually > use DataSourceV2 without these patches. > > On Wed, Feb 21, 2018 at 9:21 AM, Wenchen Fan <cloud0...@gmail.com> wrote: > >> SPARK-23323 adds a new API, I'm not sure we can still do it at this stage >> of the release... Besides users can work around it by calling the spark >> output coordinator themselves in their data source. >> >> SPARK-23203 is non-trivial and didn't fix any known bugs, so it's hard to >> convince other people that it's safe to add it to the release during the RC >> phase. >> >> SPARK-23418 depends on the above one. >> >> Generally they are good to have in Spark 2.3, if they were merged before >> the RC. I think this is a lesson we should learn from, that we should work >> on stuff we want in the release before the RC, instead of after. >> >> On Thu, Feb 22, 2018 at 1:01 AM, Ryan Blue <rb...@netflix.com.invalid> >> wrote: >> >>> What does everyone think about getting some of the newer DataSourceV2 >>> improvements in? It should be low risk because it is a new code path, and >>> v2 isn't very usable without things like support for using the output >>> commit coordinator to deconflict writes. >>> >>> The ones I'd like to get in are: >>> * Use the output commit coordinator: https://issues.ap >>> ache.org/jira/browse/SPARK-23323 >>> * Use immutable trees and the same push-down logic as other read paths: >>> https://issues.apache.org/jira/browse/SPARK-23203 >>> * Don't allow users to supply schemas when they aren't supported: >>> https://issues.apache.org/jira/browse/SPARK-23418 >>> >>> I think it would make the 2.3.0 release more usable for anyone >>> interested in the v2 read and write paths. >>> >>> Thanks! >>> >>> On Tue, Feb 20, 2018 at 7:07 PM, Weichen Xu <weichen...@databricks.com> >>> wrote: >>> >>>> +1 >>>> >>>> On Wed, Feb 21, 2018 at 10:07 AM, Marcelo Vanzin <van...@cloudera.com> >>>> wrote: >>>> >>>>> Done, thanks! >>>>> >>>>> On Tue, Feb 20, 2018 at 6:05 PM, Sameer Agarwal <samee...@apache.org> >>>>> wrote: >>>>> > Sure, please feel free to backport. >>>>> > >>>>> > On 20 February 2018 at 18:02, Marcelo Vanzin <van...@cloudera.com> >>>>> wrote: >>>>> >> >>>>> >> Hey Sameer, >>>>> >> >>>>> >> Mind including https://github.com/apache/spark/pull/20643 >>>>> >> (SPARK-23468) in the new RC? It's a minor bug since I've only hit >>>>> it >>>>> >> with older shuffle services, but it's pretty safe. >>>>> >> >>>>> >> On Tue, Feb 20, 2018 at 5:58 PM, Sameer Agarwal < >>>>> samee...@apache.org> >>>>> >> wrote: >>>>> >> > This RC has failed due to >>>>> >> > https://issues.apache.org/jira/browse/SPARK-23470. >>>>> >> > Now that the fix has been merged in 2.3 (thanks Marcelo!), I'll >>>>> follow >>>>> >> > up >>>>> >> > with an RC5 soon. >>>>> >> > >>>>> >> > On 20 February 2018 at 16:49, Ryan Blue <rb...@netflix.com> >>>>> wrote: >>>>> >> >> >>>>> >> >> +1 >>>>> >> >> >>>>> >> >> Build & tests look fine, checked signature and checksums for src >>>>> >> >> tarball. >>>>> >> >> >>>>> >> >> On Tue, Feb 20, 2018 at 12:54 PM, Shixiong(Ryan) Zhu >>>>> >> >> <shixi...@databricks.com> wrote: >>>>> >> >>> >>>>> >> >>> I'm -1 because of the UI regression >>>>> >> >>> https://issues.apache.org/jira/browse/SPARK-23470 : the All >>>>> Jobs page >>>>> >> >>> may be >>>>> >> >>> too slow and cause "read timeout" when there are lots of jobs >>>>> and >>>>> >> >>> stages. >>>>> >> >>> This is one of the most important pages because when it's >>>>> broken, it's >>>>> >> >>> pretty hard to use Spark Web UI. >>>>> >> >>> >>>>> >> >>> >>>>> >> >>> On Tue, Feb 20, 2018 at 4:37 AM, Marco Gaido < >>>>> marcogaid...@gmail.com> >>>>> >> >>> wrote: >>>>> >> >>>> >>>>> >> >>>> +1 >>>>> >> >>>> >>>>> >> >>>> 2018-02-20 12:30 GMT+01:00 Hyukjin Kwon <gurwls...@gmail.com>: >>>>> >> >>>>> >>>>> >> >>>>> +1 too >>>>> >> >>>>> >>>>> >> >>>>> 2018-02-20 14:41 GMT+09:00 Takuya UESHIN < >>>>> ues...@happy-camper.st>: >>>>> >> >>>>>> >>>>> >> >>>>>> +1 >>>>> >> >>>>>> >>>>> >> >>>>>> >>>>> >> >>>>>> On Tue, Feb 20, 2018 at 2:14 PM, Xingbo Jiang >>>>> >> >>>>>> <jiangxb1...@gmail.com> >>>>> >> >>>>>> wrote: >>>>> >> >>>>>>> >>>>> >> >>>>>>> +1 >>>>> >> >>>>>>> >>>>> >> >>>>>>> >>>>> >> >>>>>>> Wenchen Fan <cloud0...@gmail.com>于2018年2月20日 周二下午1:09写道： >>>>> >> >>>>>>>> >>>>> >> >>>>>>>> +1 >>>>> >> >>>>>>>> >>>>> >> >>>>>>>> On Tue, Feb 20, 2018 at 12:53 PM, Reynold Xin >>>>> >> >>>>>>>> <r...@databricks.com> >>>>> >> >>>>>>>> wrote: >>>>> >> >>>>>>>>> >>>>> >> >>>>>>>>> +1 >>>>> >> >>>>>>>>> >>>>> >> >>>>>>>>> On Feb 20, 2018, 5:51 PM +1300, Sameer Agarwal >>>>> >> >>>>>>>>> <sameer.a...@gmail.com>, wrote: >>>>> >> >>>>>>>>>> >>>>> >> >>>>>>>>>> this file shouldn't be included? >>>>> >> >>>>>>>>>> >>>>> >> >>>>>>>>>> https://dist.apache.org/repos/ >>>>> dist/dev/spark/v2.3.0-rc4-bin/spark-parent_2.11.iml >>>>> >> >>>>>>>>> >>>>> >> >>>>>>>>> >>>>> >> >>>>>>>>> I've now deleted this file >>>>> >> >>>>>>>>> >>>>> >> >>>>>>>>>> From: Sameer Agarwal <sameer.a...@gmail.com> >>>>> >> >>>>>>>>>> Sent: Saturday, February 17, 2018 1:43:39 PM >>>>> >> >>>>>>>>>> To: Sameer Agarwal >>>>> >> >>>>>>>>>> Cc: dev >>>>> >> >>>>>>>>>> Subject: Re: [VOTE] Spark 2.3.0 (RC4) >>>>> >> >>>>>>>>>> >>>>> >> >>>>>>>>>> I'll start with a +1 once again. >>>>> >> >>>>>>>>>> >>>>> >> >>>>>>>>>> All blockers reported against RC3 have been resolved and >>>>> the >>>>> >> >>>>>>>>>> builds are healthy. >>>>> >> >>>>>>>>>> >>>>> >> >>>>>>>>>> On 17 February 2018 at 13:41, Sameer Agarwal >>>>> >> >>>>>>>>>> <samee...@apache.org> >>>>> >> >>>>>>>>>> wrote: >>>>> >> >>>>>>>>>>> >>>>> >> >>>>>>>>>>> Please vote on releasing the following candidate as >>>>> Apache >>>>> >> >>>>>>>>>>> Spark >>>>> >> >>>>>>>>>>> version 2.3.0. The vote is open until Thursday February >>>>> 22, >>>>> >> >>>>>>>>>>> 2018 at 8:00:00 >>>>> >> >>>>>>>>>>> am UTC and passes if a majority of at least 3 PMC +1 >>>>> votes are >>>>> >> >>>>>>>>>>> cast. >>>>> >> >>>>>>>>>>> >>>>> >> >>>>>>>>>>> >>>>> >> >>>>>>>>>>> [ ] +1 Release this package as Apache Spark 2.3.0 >>>>> >> >>>>>>>>>>> >>>>> >> >>>>>>>>>>> [ ] -1 Do not release this package because ... >>>>> >> >>>>>>>>>>> >>>>> >> >>>>>>>>>>> >>>>> >> >>>>>>>>>>> To learn more about Apache Spark, please see >>>>> >> >>>>>>>>>>> https://spark.apache.org/ >>>>> >> >>>>>>>>>>> >>>>> >> >>>>>>>>>>> The tag to be voted on is v2.3.0-rc4: >>>>> >> >>>>>>>>>>> https://github.com/apache/spark/tree/v2.3.0-rc4 >>>>> >> >>>>>>>>>>> (44095cb65500739695b0324c177c19dfa1471472) >>>>> >> >>>>>>>>>>> >>>>> >> >>>>>>>>>>> List of JIRA tickets resolved in this release can be >>>>> found >>>>> >> >>>>>>>>>>> here: >>>>> >> >>>>>>>>>>> >>>>> >> >>>>>>>>>>> https://issues.apache.org/jira >>>>> /projects/SPARK/versions/12339551 >>>>> >> >>>>>>>>>>> >>>>> >> >>>>>>>>>>> The release files, including signatures, digests, etc. >>>>> can be >>>>> >> >>>>>>>>>>> found at: >>>>> >> >>>>>>>>>>> https://dist.apache.org/repos/ >>>>> dist/dev/spark/v2.3.0-rc4-bin/ >>>>> >> >>>>>>>>>>> >>>>> >> >>>>>>>>>>> Release artifacts are signed with the following key: >>>>> >> >>>>>>>>>>> https://dist.apache.org/repos/dist/dev/spark/KEYS >>>>> >> >>>>>>>>>>> >>>>> >> >>>>>>>>>>> The staging repository for this release can be found at: >>>>> >> >>>>>>>>>>> >>>>> >> >>>>>>>>>>> >>>>> >> >>>>>>>>>>> https://repository.apache.org/ >>>>> content/repositories/orgapachespark-1265/ >>>>> >> >>>>>>>>>>> >>>>> >> >>>>>>>>>>> The documentation corresponding to this release can be >>>>> found >>>>> >> >>>>>>>>>>> at: >>>>> >> >>>>>>>>>>> >>>>> >> >>>>>>>>>>> >>>>> >> >>>>>>>>>>> https://dist.apache.org/repos/ >>>>> dist/dev/spark/v2.3.0-rc4-docs/_site/index.html >>>>> >> >>>>>>>>>>> >>>>> >> >>>>>>>>>>> >>>>> >> >>>>>>>>>>> FAQ >>>>> >> >>>>>>>>>>> >>>>> >> >>>>>>>>>>> ======================================= >>>>> >> >>>>>>>>>>> What are the unresolved issues targeted for 2.3.0? >>>>> >> >>>>>>>>>>> ======================================= >>>>> >> >>>>>>>>>>> >>>>> >> >>>>>>>>>>> Please see https://s.apache.org/oXKi. At the time of >>>>> writing, >>>>> >> >>>>>>>>>>> there are currently no known release blockers. >>>>> >> >>>>>>>>>>> >>>>> >> >>>>>>>>>>> ========================= >>>>> >> >>>>>>>>>>> How can I help test this release? >>>>> >> >>>>>>>>>>> ========================= >>>>> >> >>>>>>>>>>> >>>>> >> >>>>>>>>>>> If you are a Spark user, you can help us test this >>>>> release by >>>>> >> >>>>>>>>>>> taking an existing Spark workload and running on this >>>>> release >>>>> >> >>>>>>>>>>> candidate, >>>>> >> >>>>>>>>>>> then reporting any regressions. >>>>> >> >>>>>>>>>>> >>>>> >> >>>>>>>>>>> If you're working in PySpark you can set up a virtual >>>>> env and >>>>> >> >>>>>>>>>>> install the current RC and see if anything important >>>>> breaks, >>>>> >> >>>>>>>>>>> in the >>>>> >> >>>>>>>>>>> Java/Scala you can add the staging repository to your >>>>> projects >>>>> >> >>>>>>>>>>> resolvers and >>>>> >> >>>>>>>>>>> test with the RC (make sure to clean up the artifact >>>>> cache >>>>> >> >>>>>>>>>>> before/after so >>>>> >> >>>>>>>>>>> you don't end up building with a out of date RC going >>>>> >> >>>>>>>>>>> forward). >>>>> >> >>>>>>>>>>> >>>>> >> >>>>>>>>>>> =========================================== >>>>> >> >>>>>>>>>>> What should happen to JIRA tickets still targeting >>>>> 2.3.0? >>>>> >> >>>>>>>>>>> =========================================== >>>>> >> >>>>>>>>>>> >>>>> >> >>>>>>>>>>> Committers should look at those and triage. Extremely >>>>> >> >>>>>>>>>>> important >>>>> >> >>>>>>>>>>> bug fixes, documentation, and API tweaks that impact >>>>> >> >>>>>>>>>>> compatibility should be >>>>> >> >>>>>>>>>>> worked on immediately. Everything else please retarget >>>>> to >>>>> >> >>>>>>>>>>> 2.3.1 or 2.4.0 as >>>>> >> >>>>>>>>>>> appropriate. >>>>> >> >>>>>>>>>>> >>>>> >> >>>>>>>>>>> =================== >>>>> >> >>>>>>>>>>> Why is my bug not fixed? >>>>> >> >>>>>>>>>>> =================== >>>>> >> >>>>>>>>>>> >>>>> >> >>>>>>>>>>> In order to make timely releases, we will typically not >>>>> hold >>>>> >> >>>>>>>>>>> the >>>>> >> >>>>>>>>>>> release unless the bug in question is a regression from >>>>> 2.2.0. >>>>> >> >>>>>>>>>>> That being >>>>> >> >>>>>>>>>>> said, if there is something which is a regression from >>>>> 2.2.0 >>>>> >> >>>>>>>>>>> and has not >>>>> >> >>>>>>>>>>> been correctly targeted please ping me or a committer >>>>> to help >>>>> >> >>>>>>>>>>> target the >>>>> >> >>>>>>>>>>> issue (you can see the open issues listed as impacting >>>>> Spark >>>>> >> >>>>>>>>>>> 2.3.0 at >>>>> >> >>>>>>>>>>> https://s.apache.org/WmoI). >>>>> >> >>>>>>>>>> >>>>> >> >>>>>>>>>> >>>>> >> >>>>>>>>>> >>>>> >> >>>>>>>>>> >>>>> >> >>>>>>>>>> -- >>>>> >> >>>>>>>>>> Sameer Agarwal >>>>> >> >>>>>>>>>> Computer Science | UC Berkeley >>>>> >> >>>>>>>>>> http://cs.berkeley.edu/~sameerag >>>>> >> >>>>>>>>> >>>>> >> >>>>>>>>> >>>>> >> >>>>>>>>> >>>>> >> >>>>>>>>> >>>>> >> >>>>>>>>> -- >>>>> >> >>>>>>>>> Sameer Agarwal >>>>> >> >>>>>>>>> Computer Science | UC Berkeley >>>>> >> >>>>>>>>> http://cs.berkeley.edu/~sameerag >>>>> >> >>>>>>>> >>>>> >> >>>>>>>> >>>>> >> >>>>>> >>>>> >> >>>>>> >>>>> >> >>>>>> >>>>> >> >>>>>> -- >>>>> >> >>>>>> Takuya UESHIN >>>>> >> >>>>>> Tokyo, Japan >>>>> >> >>>>>> >>>>> >> >>>>>> http://twitter.com/ueshin >>>>> >> >>>>> >>>>> >> >>>>> >>>>> >> >>>> >>>>> >> >>> >>>>> >> >> >>>>> >> >> >>>>> >> >> >>>>> >> >> -- >>>>> >> >> Ryan Blue >>>>> >> >> Software Engineer >>>>> >> >> Netflix >>>>> >> > >>>>> >> > >>>>> >> >>>>> >> >>>>> >> >>>>> >> -- >>>>> >> Marcelo >>>>> > >>>>> > >>>>> >>>>> >>>>> >>>>> -- >>>>> Marcelo >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>>>> >>>>> >>>> >>> >>> >>> -- >>> Ryan Blue >>> Software Engineer >>> Netflix >>> >> >> > > > -- > Ryan Blue > Software Engineer > Netflix >
