Migration onto error classes and their testing
Hi All, As you probably know, we introduced error classes ( https://github.com/apache/spark/pull/32850) as a part of the error messages framework (https://issues.apache.org/jira/browse/SPARK-33539). I would like to propose to migrate all exceptions from QueryExecutionErrors, QueryCompilationErrors and QueryParsingErrors on the error classes using instances of SparkThrowable, and carefully test every error class by writing tests in dedicated test suites: - QueryExecutionErrorsSuite for the errors that are occurred during query execution - QueryCompilationErrorsSuite ... query compilation or eagerly executing commands - QueryParsingErrorsSuite ... parsing errors Here is an example https://github.com/apache/spark/pull/35157 of how an existing Java exception can be replaced, and testing of related error classes. If there will be no objections to the approach, I would create an umbrella JIRA for this activity, and sub-tasks per every error class (or a group of related classes). At the end, we should migrate all exceptions from the files Query.*Errors.scala and cover all error classes from the error-classes.json file by tests. WDYT, any thoughts are welcome. Yours faithfully, Max Gekk
Re: [VOTE][SPIP] Support Customized Kubernetes Schedulers Proposal
+1 (non-binding) On Wed, Jan 12, 2022 at 10:29 AM Mridul Muralidharan wrote: > > +1 (binding) > This should be a great improvement ! > > Regards, > Mridul > > On Wed, Jan 12, 2022 at 4:04 AM Kent Yao wrote: > >> +1 (non-binding) >> >> Thomas Graves 于2022年1月12日周三 11:52写道： >> >>> +1 (binding). >>> >>> One minor note since I haven't had time to look at the implementation >>> details is please make sure resource aware scheduling and the stage >>> level scheduling still work or any caveats are documented. Feel free >>> to ping me if questions in these areas. >>> >>> Tom >>> >>> On Wed, Jan 5, 2022 at 7:07 PM Yikun Jiang wrote: >>> > >>> > Hi all, >>> > >>> > I’d like to start a vote for SPIP: "Support Customized Kubernetes >>> Schedulers Proposal" >>> > >>> > The SPIP is to support customized Kubernetes schedulers in Spark on >>> Kubernetes. >>> > >>> > Please also refer to: >>> > >>> > - Previous discussion in dev mailing list: [DISCUSSION] SPIP: Support >>> Volcano/Alternative Schedulers Proposal >>> > - Design doc: [SPIP] Spark-36057 Support Customized Kubernetes >>> Schedulers Proposal >>> > - JIRA: SPARK-36057 >>> > >>> > Please vote on the SPIP: >>> > >>> > [ ] +1: Accept the proposal as an official SPIP >>> > [ ] +0 >>> > [ ] -1: I don’t think this is a good idea because … >>> > >>> > Regards, >>> > Yikun >>> >>> - >>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>> >>> -- Ryan Blue Tabular
Re: [VOTE][SPIP] Support Customized Kubernetes Schedulers Proposal
+1 (binding) This should be a great improvement ! Regards, Mridul On Wed, Jan 12, 2022 at 4:04 AM Kent Yao wrote: > +1 (non-binding) > > Thomas Graves 于2022年1月12日周三 11:52写道： > >> +1 (binding). >> >> One minor note since I haven't had time to look at the implementation >> details is please make sure resource aware scheduling and the stage >> level scheduling still work or any caveats are documented. Feel free >> to ping me if questions in these areas. >> >> Tom >> >> On Wed, Jan 5, 2022 at 7:07 PM Yikun Jiang wrote: >> > >> > Hi all, >> > >> > I’d like to start a vote for SPIP: "Support Customized Kubernetes >> Schedulers Proposal" >> > >> > The SPIP is to support customized Kubernetes schedulers in Spark on >> Kubernetes. >> > >> > Please also refer to: >> > >> > - Previous discussion in dev mailing list: [DISCUSSION] SPIP: Support >> Volcano/Alternative Schedulers Proposal >> > - Design doc: [SPIP] Spark-36057 Support Customized Kubernetes >> Schedulers Proposal >> > - JIRA: SPARK-36057 >> > >> > Please vote on the SPIP: >> > >> > [ ] +1: Accept the proposal as an official SPIP >> > [ ] +0 >> > [ ] -1: I don’t think this is a good idea because … >> > >> > Regards, >> > Yikun >> >> - >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> >>
Re: [VOTE] Release Spark 3.2.1 (RC1)
Thank you all for testing and voting! I will -1 this RC because https://issues.apache.org/jira/browse/SPARK-37855 and https://issues.apache.org/jira/browse/SPARK-37859 are regressions. These are not blockers but I think it's better to fix them in 3.2.1. I will prepare for RC2. Thanks, Huaxin On Wed, Jan 12, 2022 at 2:03 AM Kent Yao wrote: > +1 (non-binding). > > Chao Sun 于2022年1月12日周三 16:10写道： > >> +1 (non-binding). Thanks Huaxin for driving the release! >> >> On Tue, Jan 11, 2022 at 11:56 PM Ruifeng Zheng >> wrote: >> >>> +1 (non-binding) >>> >>> Thanks, ruifeng zheng >>> >>> -- Original -- >>> *From:* "Cheng Su" ; >>> *Date:* Wed, Jan 12, 2022 02:54 PM >>> *To:* "Qian Sun";"huaxin gao"< >>> huaxin.ga...@gmail.com>; >>> *Cc:* "dev"; >>> *Subject:* Re: [VOTE] Release Spark 3.2.1 (RC1) >>> >>> +1 (non-binding). Checked commit history and ran some local tests. >>> >>> >>> >>> Thanks, >>> >>> Cheng Su >>> >>> >>> >>> *From: *Qian Sun >>> *Date: *Tuesday, January 11, 2022 at 7:55 PM >>> *To: *huaxin gao >>> *Cc: *dev >>> *Subject: *Re: [VOTE] Release Spark 3.2.1 (RC1) >>> >>> +1 >>> >>> >>> >>> Looks good. All integration tests passed. >>> >>> >>> >>> Qian >>> >>> >>> >>> 2022年1月11日 上午2:09，huaxin gao 写道： >>> >>> >>> >>> Please vote on releasing the following candidate as Apache Spark version >>> 3.2.1. >>> >>> >>> The vote is open until Jan. 13th at 12 PM PST (8 PM UTC) and passes if a >>> majority >>> >>> +1 PMC votes are cast, with a minimum of 3 + 1 votes. >>> >>> >>> [ ] +1 Release this package as Apache Spark 3.2.1 >>> [ ] -1 Do not release this package because ... >>> >>> To learn more about Apache Spark, please see http://spark.apache.org/ >>> >>> There are currently no issues targeting 3.2.1 (try project = SPARK AND >>> "Target Version/s" = "3.2.1" AND status in (Open, Reopened, "In >>> Progress")) >>> >>> The tag to be voted on is v3.2.1-rc1 (commit >>> 2b0ee226f8dd17b278ad11139e62464433191653): >>> >>> https://github.com/apache/spark/tree/v3.2.1-rc1 >>> >>> The release files, including signatures, digests, etc. can be found at: >>> https://dist.apache.org/repos/dist/dev/spark/v3.2.1-rc1-bin/ >>> >>> Signatures used for Spark RCs can be found in this file: >>> https://dist.apache.org/repos/dist/dev/spark/KEYS >>> >>> The staging repository for this release can be found at: >>> https://repository.apache.org/content/repositories/orgapachespark-1395/ >>> >>> The documentation corresponding to this release can be found at: >>> https://dist.apache.org/repos/dist/dev/spark/v3.2.1-rc1-docs/ >>> >>> The list of bug fixes going into 3.2.1 can be found at the following URL: >>> https://s.apache.org/7tzik >>> >>> This release is using the release script of the tag v3.2.1-rc1. >>> >>> FAQ >>> >>> >>> = >>> How can I help test this release? >>> = >>> >>> If you are a Spark user, you can help us test this release by taking >>> an existing Spark workload and running on this release candidate, then >>> reporting any regressions. >>> >>> If you're working in PySpark you can set up a virtual env and install >>> the current RC and see if anything important breaks, in the Java/Scala >>> you can add the staging repository to your projects resolvers and test >>> with the RC (make sure to clean up the artifact cache before/after so >>> you don't end up building with an out of date RC going forward). >>> >>> === >>> What should happen to JIRA tickets still targeting 3.2.1? >>> === >>> >>> The current list of open tickets targeted at 3.2.1 can be found at: >>> https://issues.apache.org/jira/projects/SPARK and search for "Target >>> Version/s" = 3.2.1 >>> >>> Committers should look at those and triage. Extremely important bug >>> fixes, documentation, and API tweaks that impact compatibility should >>> be worked on immediately. Everything else please retarget to an >>> appropriate release. >>> >>> == >>> But my bug isn't fixed? >>> == >>> >>> In order to make timely releases, we will typically not hold the >>> release unless the bug in question is a regression from the previous >>> release. That being said, if there is something which is a regression >>> that has not been correctly targeted please ping me or a committer to >>> help target the issue. >>> >>> >>> >>
Re: [VOTE][SPIP] Support Customized Kubernetes Schedulers Proposal
+1 (non-binding) Thomas Graves 于2022年1月12日周三 11:52写道： > +1 (binding). > > One minor note since I haven't had time to look at the implementation > details is please make sure resource aware scheduling and the stage > level scheduling still work or any caveats are documented. Feel free > to ping me if questions in these areas. > > Tom > > On Wed, Jan 5, 2022 at 7:07 PM Yikun Jiang wrote: > > > > Hi all, > > > > I’d like to start a vote for SPIP: "Support Customized Kubernetes > Schedulers Proposal" > > > > The SPIP is to support customized Kubernetes schedulers in Spark on > Kubernetes. > > > > Please also refer to: > > > > - Previous discussion in dev mailing list: [DISCUSSION] SPIP: Support > Volcano/Alternative Schedulers Proposal > > - Design doc: [SPIP] Spark-36057 Support Customized Kubernetes > Schedulers Proposal > > - JIRA: SPARK-36057 > > > > Please vote on the SPIP: > > > > [ ] +1: Accept the proposal as an official SPIP > > [ ] +0 > > [ ] -1: I don’t think this is a good idea because … > > > > Regards, > > Yikun > > - > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > >
Re: [VOTE] Release Spark 3.2.1 (RC1)
+1 (non-binding). Chao Sun 于2022年1月12日周三 16:10写道： > +1 (non-binding). Thanks Huaxin for driving the release! > > On Tue, Jan 11, 2022 at 11:56 PM Ruifeng Zheng > wrote: > >> +1 (non-binding) >> >> Thanks, ruifeng zheng >> >> -- Original -- >> *From:* "Cheng Su" ; >> *Date:* Wed, Jan 12, 2022 02:54 PM >> *To:* "Qian Sun";"huaxin gao"< >> huaxin.ga...@gmail.com>; >> *Cc:* "dev"; >> *Subject:* Re: [VOTE] Release Spark 3.2.1 (RC1) >> >> +1 (non-binding). Checked commit history and ran some local tests. >> >> >> >> Thanks, >> >> Cheng Su >> >> >> >> *From: *Qian Sun >> *Date: *Tuesday, January 11, 2022 at 7:55 PM >> *To: *huaxin gao >> *Cc: *dev >> *Subject: *Re: [VOTE] Release Spark 3.2.1 (RC1) >> >> +1 >> >> >> >> Looks good. All integration tests passed. >> >> >> >> Qian >> >> >> >> 2022年1月11日 上午2:09，huaxin gao 写道： >> >> >> >> Please vote on releasing the following candidate as Apache Spark version >> 3.2.1. >> >> >> The vote is open until Jan. 13th at 12 PM PST (8 PM UTC) and passes if a >> majority >> >> +1 PMC votes are cast, with a minimum of 3 + 1 votes. >> >> >> [ ] +1 Release this package as Apache Spark 3.2.1 >> [ ] -1 Do not release this package because ... >> >> To learn more about Apache Spark, please see http://spark.apache.org/ >> >> There are currently no issues targeting 3.2.1 (try project = SPARK AND >> "Target Version/s" = "3.2.1" AND status in (Open, Reopened, "In >> Progress")) >> >> The tag to be voted on is v3.2.1-rc1 (commit >> 2b0ee226f8dd17b278ad11139e62464433191653): >> >> https://github.com/apache/spark/tree/v3.2.1-rc1 >> >> The release files, including signatures, digests, etc. can be found at: >> https://dist.apache.org/repos/dist/dev/spark/v3.2.1-rc1-bin/ >> >> Signatures used for Spark RCs can be found in this file: >> https://dist.apache.org/repos/dist/dev/spark/KEYS >> >> The staging repository for this release can be found at: >> https://repository.apache.org/content/repositories/orgapachespark-1395/ >> >> The documentation corresponding to this release can be found at: >> https://dist.apache.org/repos/dist/dev/spark/v3.2.1-rc1-docs/ >> >> The list of bug fixes going into 3.2.1 can be found at the following URL: >> https://s.apache.org/7tzik >> >> This release is using the release script of the tag v3.2.1-rc1. >> >> FAQ >> >> >> = >> How can I help test this release? >> = >> >> If you are a Spark user, you can help us test this release by taking >> an existing Spark workload and running on this release candidate, then >> reporting any regressions. >> >> If you're working in PySpark you can set up a virtual env and install >> the current RC and see if anything important breaks, in the Java/Scala >> you can add the staging repository to your projects resolvers and test >> with the RC (make sure to clean up the artifact cache before/after so >> you don't end up building with an out of date RC going forward). >> >> === >> What should happen to JIRA tickets still targeting 3.2.1? >> === >> >> The current list of open tickets targeted at 3.2.1 can be found at: >> https://issues.apache.org/jira/projects/SPARK and search for "Target >> Version/s" = 3.2.1 >> >> Committers should look at those and triage. Extremely important bug >> fixes, documentation, and API tweaks that impact compatibility should >> be worked on immediately. Everything else please retarget to an >> appropriate release. >> >> == >> But my bug isn't fixed? >> == >> >> In order to make timely releases, we will typically not hold the >> release unless the bug in question is a regression from the previous >> release. That being said, if there is something which is a regression >> that has not been correctly targeted please ping me or a committer to >> help target the issue. >> >> >> >
Re: [VOTE] Release Spark 3.2.1 (RC1)
+1 (non-binding). Thanks Huaxin for driving the release! On Tue, Jan 11, 2022 at 11:56 PM Ruifeng Zheng wrote: > +1 (non-binding) > > Thanks, ruifeng zheng > > -- Original -- > *From:* "Cheng Su" ; > *Date:* Wed, Jan 12, 2022 02:54 PM > *To:* "Qian Sun";"huaxin gao"< > huaxin.ga...@gmail.com>; > *Cc:* "dev"; > *Subject:* Re: [VOTE] Release Spark 3.2.1 (RC1) > > +1 (non-binding). Checked commit history and ran some local tests. > > > > Thanks, > > Cheng Su > > > > *From: *Qian Sun > *Date: *Tuesday, January 11, 2022 at 7:55 PM > *To: *huaxin gao > *Cc: *dev > *Subject: *Re: [VOTE] Release Spark 3.2.1 (RC1) > > +1 > > > > Looks good. All integration tests passed. > > > > Qian > > > > 2022年1月11日 上午2:09，huaxin gao 写道： > > > > Please vote on releasing the following candidate as Apache Spark version > 3.2.1. > > > The vote is open until Jan. 13th at 12 PM PST (8 PM UTC) and passes if a > majority > > +1 PMC votes are cast, with a minimum of 3 + 1 votes. > > > [ ] +1 Release this package as Apache Spark 3.2.1 > [ ] -1 Do not release this package because ... > > To learn more about Apache Spark, please see http://spark.apache.org/ > > There are currently no issues targeting 3.2.1 (try project = SPARK AND > "Target Version/s" = "3.2.1" AND status in (Open, Reopened, "In Progress")) > > The tag to be voted on is v3.2.1-rc1 (commit > 2b0ee226f8dd17b278ad11139e62464433191653): > > https://github.com/apache/spark/tree/v3.2.1-rc1 > > The release files, including signatures, digests, etc. can be found at: > https://dist.apache.org/repos/dist/dev/spark/v3.2.1-rc1-bin/ > > Signatures used for Spark RCs can be found in this file: > https://dist.apache.org/repos/dist/dev/spark/KEYS > > The staging repository for this release can be found at: > https://repository.apache.org/content/repositories/orgapachespark-1395/ > > The documentation corresponding to this release can be found at: > https://dist.apache.org/repos/dist/dev/spark/v3.2.1-rc1-docs/ > > The list of bug fixes going into 3.2.1 can be found at the following URL: > https://s.apache.org/7tzik > > This release is using the release script of the tag v3.2.1-rc1. > > FAQ > > > = > How can I help test this release? > = > > If you are a Spark user, you can help us test this release by taking > an existing Spark workload and running on this release candidate, then > reporting any regressions. > > If you're working in PySpark you can set up a virtual env and install > the current RC and see if anything important breaks, in the Java/Scala > you can add the staging repository to your projects resolvers and test > with the RC (make sure to clean up the artifact cache before/after so > you don't end up building with an out of date RC going forward). > > === > What should happen to JIRA tickets still targeting 3.2.1? > === > > The current list of open tickets targeted at 3.2.1 can be found at: > https://issues.apache.org/jira/projects/SPARK and search for "Target > Version/s" = 3.2.1 > > Committers should look at those and triage. Extremely important bug > fixes, documentation, and API tweaks that impact compatibility should > be worked on immediately. Everything else please retarget to an > appropriate release. > > == > But my bug isn't fixed? > == > > In order to make timely releases, we will typically not hold the > release unless the bug in question is a regression from the previous > release. That being said, if there is something which is a regression > that has not been correctly targeted please ping me or a committer to > help target the issue. > > >
