> > What's the process to propose a feature to be included in the final Spark > 3.0 release? >
I don't know whether there exists any specific process here, normally you just merge the feature into Spark master before release code freeze, and then the feature would probably be included in the release. The code freeze date for Spark 3.0 has not been decided yet, though. Li Jin <ice.xell...@gmail.com> 于2019年10月8日周二 下午2:14写道： > Thanks for summary! > > I have a question that is semi-related - What's the process to propose a > feature to be included in the final Spark 3.0 release? > > In particular, I am interested in > https://issues.apache.org/jira/browse/SPARK-28006. I am happy to do the > work so want to make sure I don't miss the "cut" date. > > On Tue, Oct 8, 2019 at 4:53 PM Xingbo Jiang <jiangxb1...@gmail.com> wrote: > >> Hi all, >> >> Thanks for all the feedbacks, here is the updated feature list: >> >> SPARK-11215 <https://issues.apache.org/jira/browse/SPARK-11215> Multiple >> columns support added to various Transformers: StringIndexer >> >> SPARK-11150 <https://issues.apache.org/jira/browse/SPARK-11150> >> Implement Dynamic Partition Pruning >> >> SPARK-13677 <https://issues.apache.org/jira/browse/SPARK-13677> Support >> Tree-Based Feature Transformation >> >> SPARK-16692 <https://issues.apache.org/jira/browse/SPARK-16692> Add >> MultilabelClassificationEvaluator >> >> SPARK-19591 <https://issues.apache.org/jira/browse/SPARK-19591> Add >> sample weights to decision trees >> >> SPARK-19712 <https://issues.apache.org/jira/browse/SPARK-19712> Pushing >> Left Semi and Left Anti joins through Project, Aggregate, Window, Union etc. >> >> SPARK-19827 <https://issues.apache.org/jira/browse/SPARK-19827> R API >> for Power Iteration Clustering >> >> SPARK-20286 <https://issues.apache.org/jira/browse/SPARK-20286> Improve >> logic for timing out executors in dynamic allocation >> >> SPARK-20636 <https://issues.apache.org/jira/browse/SPARK-20636> >> Eliminate unnecessary shuffle with adjacent Window expressions >> >> SPARK-22148 <https://issues.apache.org/jira/browse/SPARK-22148> Acquire >> new executors to avoid hang because of blacklisting >> >> SPARK-22796 <https://issues.apache.org/jira/browse/SPARK-22796> Multiple >> columns support added to various Transformers: PySpark QuantileDiscretizer >> >> SPARK-23128 <https://issues.apache.org/jira/browse/SPARK-23128> A new >> approach to do adaptive execution in Spark SQL >> >> SPARK-23155 <https://issues.apache.org/jira/browse/SPARK-23155> Apply >> custom log URL pattern for executor log URLs in SHS >> >> SPARK-23539 <https://issues.apache.org/jira/browse/SPARK-23539> Add >> support for Kafka headers >> >> SPARK-23674 <https://issues.apache.org/jira/browse/SPARK-23674> Add >> Spark ML Listener for Tracking ML Pipeline Status >> >> SPARK-23710 <https://issues.apache.org/jira/browse/SPARK-23710> Upgrade >> the built-in Hive to 2.3.5 for hadoop-3.2 >> >> SPARK-24333 <https://issues.apache.org/jira/browse/SPARK-24333> Add fit >> with validation set to Gradient Boosted Trees: Python API >> >> SPARK-24417 <https://issues.apache.org/jira/browse/SPARK-24417> Build >> and Run Spark on JDK11 >> >> SPARK-24615 <https://issues.apache.org/jira/browse/SPARK-24615> >> Accelerator-aware task scheduling for Spark >> >> SPARK-24920 <https://issues.apache.org/jira/browse/SPARK-24920> Allow >> sharing Netty's memory pool allocators >> >> SPARK-25250 <https://issues.apache.org/jira/browse/SPARK-25250> Fix race >> condition with tasks running when new attempt for same stage is created >> leads to other task in the next attempt running on the same partition id >> retry multiple times >> >> SPARK-25341 <https://issues.apache.org/jira/browse/SPARK-25341> Support >> rolling back a shuffle map stage and re-generate the shuffle files >> >> SPARK-25348 <https://issues.apache.org/jira/browse/SPARK-25348> Data >> source for binary files >> >> SPARK-25501 <https://issues.apache.org/jira/browse/SPARK-25501> Add >> kafka delegation token support >> >> SPARK-25603 <https://issues.apache.org/jira/browse/SPARK-25603> >> Generalize Nested Column Pruning >> >> SPARK-26132 <https://issues.apache.org/jira/browse/SPARK-26132> Remove >> support for Scala 2.11 in Spark 3.0.0 >> >> SPARK-26215 <https://issues.apache.org/jira/browse/SPARK-26215> define >> reserved keywords after SQL standard >> >> SPARK-26412 <https://issues.apache.org/jira/browse/SPARK-26412> Allow >> Pandas UDF to take an iterator of pd.DataFrames >> >> SPARK-26759 <https://issues.apache.org/jira/browse/SPARK-26759> Arrow >> optimization in SparkR's interoperability >> >> SPARK-26785 <https://issues.apache.org/jira/browse/SPARK-26785> data >> source v2 API refactor: streaming write >> >> SPARK-26848 <https://issues.apache.org/jira/browse/SPARK-26848> >> Introduce new option to Kafka source: offset by timestamp (starting/ending) >> >> SPARK-26956 <https://issues.apache.org/jira/browse/SPARK-26956> remove >> streaming output mode from data source v2 APIs >> >> SPARK-27064 <https://issues.apache.org/jira/browse/SPARK-27064> create >> StreamingWrite at the beginning of streaming execution >> >> SPARK-27119 <https://issues.apache.org/jira/browse/SPARK-27119> Do not >> infer schema when reading Hive serde table with native data source >> >> SPARK-27225 <https://issues.apache.org/jira/browse/SPARK-27225> >> Implement join strategy hints >> >> SPARK-27240 <https://issues.apache.org/jira/browse/SPARK-27240> Use >> pandas DataFrame for struct type argument in Scalar Pandas UDF >> >> SPARK-27338 <https://issues.apache.org/jira/browse/SPARK-27338> Fix >> deadlock between TaskMemoryManager and >> UnsafeExternalSorter$SpillableIterator >> >> SPARK-27396 <https://issues.apache.org/jira/browse/SPARK-27396> Public >> APIs for extended Columnar Processing Support >> >> SPARK-27463 <https://issues.apache.org/jira/browse/SPARK-27463> Support >> Dataframe Cogroup via Pandas UDFs >> >> SPARK-27589 <https://issues.apache.org/jira/browse/SPARK-27589> >> Re-implement file sources with data source V2 API >> >> SPARK-27677 <https://issues.apache.org/jira/browse/SPARK-27677> >> Disk-persisted RDD blocks served by shuffle service, and ignored for >> Dynamic Allocation >> >> SPARK-27699 <https://issues.apache.org/jira/browse/SPARK-27699> >> Partially push down disjunctive predicated in Parquet/ORC >> >> SPARK-27763 <https://issues.apache.org/jira/browse/SPARK-27763> Port >> test cases from PostgreSQL to Spark SQL >> >> SPARK-27884 <https://issues.apache.org/jira/browse/SPARK-27884> >> Deprecate Python 2 support >> >> SPARK-27921 <https://issues.apache.org/jira/browse/SPARK-27921> Convert >> applicable *.sql tests into UDF integrated test base >> >> SPARK-27963 <https://issues.apache.org/jira/browse/SPARK-27963> Allow >> dynamic allocation without an external shuffle service >> >> SPARK-28177 <https://issues.apache.org/jira/browse/SPARK-28177> Adjust >> post shuffle partition number in adaptive execution >> >> SPARK-28199 <https://issues.apache.org/jira/browse/SPARK-28199> Move >> Trigger implementations to Triggers.scala and avoid exposing these to the >> end users >> >> SPARK-28372 <https://issues.apache.org/jira/browse/SPARK-28372> Document >> Spark WEB UI >> >> SPARK-28399 <https://issues.apache.org/jira/browse/SPARK-28399> >> RobustScaler feature transformer >> >> SPARK-28426 <https://issues.apache.org/jira/browse/SPARK-28426> Metadata >> Handling in Thrift Server >> >> SPARK-28588 <https://issues.apache.org/jira/browse/SPARK-28588> Build a >> SQL reference doc >> >> SPARK-28608 <https://issues.apache.org/jira/browse/SPARK-28608> Improve >> test coverage of ThriftServer >> >> SPARK-28753 <https://issues.apache.org/jira/browse/SPARK-28753> >> Dynamically reuse subqueries in AQE >> >> SPARK-28855 <https://issues.apache.org/jira/browse/SPARK-28855> Remove >> outdated Experimental, Evolving annotations >> SPARK-25908 <https://issues.apache.org/jira/browse/SPARK-25908> >> SPARK-28980 <https://issues.apache.org/jira/browse/SPARK-28980> Remove >> deprecated items since <= 2.2.0 >> >> Cheers, >> >> Xingbo >> >> Hyukjin Kwon <gurwls...@gmail.com> 于2019年10月7日周一 下午9:29写道： >> >>> Cogroup Pandas UDF missing: >>> >>> SPARK-27463 <https://issues.apache.org/jira/browse/SPARK-27463> Support >>> Dataframe Cogroup via Pandas UDFs >>> Vectorized R execution: >>> >>> SPARK-26759 <https://issues.apache.org/jira/browse/SPARK-26759> Arrow >>> optimization in SparkR's interoperability >>> >>> >>> 2019년 10월 8일 (화) 오전 7:50, Jungtaek Lim <kabhwan.opensou...@gmail.com>님이 >>> 작성: >>> >>>> Thanks for bringing the nice summary of Spark 3.0 improvements! >>>> >>>> I'd like to add some items from structured streaming side, >>>> >>>> SPARK-28199 <https://issues.apache.org/jira/browse/SPARK-28199> Move >>>> Trigger implementations to Triggers.scala and avoid exposing these to the >>>> end users (removal of deprecated) >>>> SPARK-23539 <https://issues.apache.org/jira/browse/SPARK-23539> Add >>>> support for Kafka headers in Structured Streaming >>>> SPARK-25501 <https://issues.apache.org/jira/browse/SPARK-25501> Add >>>> kafka delegation token support (there were follow-up issues to add >>>> functionalities like support multi clusters, etc.) >>>> SPARK-26848 <https://issues.apache.org/jira/browse/SPARK-26848> >>>> Introduce new option to Kafka source: offset by timestamp (starting/ending) >>>> SPARK-28074 <https://issues.apache.org/jira/browse/SPARK-28074> Log >>>> warn message on possible correctness issue for multiple stateful operations >>>> in single query >>>> >>>> and core side, >>>> >>>> SPARK-23155 <https://issues.apache.org/jira/browse/SPARK-23155> New >>>> feature: apply custom log URL pattern for executor log URLs in SHS >>>> (follow-up issue expanded the functionality to Spark UI as well) >>>> >>>> FYI if we count on current work in progress, there's ongoing umbrella >>>> issue regarding rolling event log & snapshot (SPARK-28594 >>>> <https://issues.apache.org/jira/browse/SPARK-28594>) which we struggle >>>> to get things done in Spark 3.0. >>>> >>>> Thanks, >>>> Jungtaek Lim (HeartSaVioR) >>>> >>>> >>>> On Tue, Oct 8, 2019 at 7:02 AM Xingbo Jiang <jiangxb1...@gmail.com> >>>> wrote: >>>> >>>>> Hi all, >>>>> >>>>> I went over all the finished JIRA tickets targeted to Spark 3.0.0, >>>>> here I'm listing all the notable features and major changes that are ready >>>>> to test/deliver, please don't hesitate to add more to the list: >>>>> >>>>> SPARK-11215 <https://issues.apache.org/jira/browse/SPARK-11215> >>>>> Multiple columns support added to various Transformers: StringIndexer >>>>> >>>>> SPARK-11150 <https://issues.apache.org/jira/browse/SPARK-11150> >>>>> Implement Dynamic Partition Pruning >>>>> >>>>> SPARK-13677 <https://issues.apache.org/jira/browse/SPARK-13677> >>>>> Support Tree-Based Feature Transformation >>>>> >>>>> SPARK-16692 <https://issues.apache.org/jira/browse/SPARK-16692> Add >>>>> MultilabelClassificationEvaluator >>>>> >>>>> SPARK-19591 <https://issues.apache.org/jira/browse/SPARK-19591> Add >>>>> sample weights to decision trees >>>>> >>>>> SPARK-19712 <https://issues.apache.org/jira/browse/SPARK-19712> >>>>> Pushing Left Semi and Left Anti joins through Project, Aggregate, Window, >>>>> Union etc. >>>>> >>>>> SPARK-19827 <https://issues.apache.org/jira/browse/SPARK-19827> R API >>>>> for Power Iteration Clustering >>>>> >>>>> SPARK-20286 <https://issues.apache.org/jira/browse/SPARK-20286> >>>>> Improve logic for timing out executors in dynamic allocation >>>>> >>>>> SPARK-20636 <https://issues.apache.org/jira/browse/SPARK-20636> >>>>> Eliminate unnecessary shuffle with adjacent Window expressions >>>>> >>>>> SPARK-22148 <https://issues.apache.org/jira/browse/SPARK-22148> >>>>> Acquire new executors to avoid hang because of blacklisting >>>>> >>>>> SPARK-22796 <https://issues.apache.org/jira/browse/SPARK-22796> >>>>> Multiple columns support added to various Transformers: PySpark >>>>> QuantileDiscretizer >>>>> >>>>> SPARK-23128 <https://issues.apache.org/jira/browse/SPARK-23128> A new >>>>> approach to do adaptive execution in Spark SQL >>>>> >>>>> SPARK-23674 <https://issues.apache.org/jira/browse/SPARK-23674> Add >>>>> Spark ML Listener for Tracking ML Pipeline Status >>>>> >>>>> SPARK-23710 <https://issues.apache.org/jira/browse/SPARK-23710> >>>>> Upgrade the built-in Hive to 2.3.5 for hadoop-3.2 >>>>> >>>>> SPARK-24333 <https://issues.apache.org/jira/browse/SPARK-24333> Add >>>>> fit with validation set to Gradient Boosted Trees: Python API >>>>> >>>>> SPARK-24417 <https://issues.apache.org/jira/browse/SPARK-24417> Build >>>>> and Run Spark on JDK11 >>>>> >>>>> SPARK-24615 <https://issues.apache.org/jira/browse/SPARK-24615> >>>>> Accelerator-aware task scheduling for Spark >>>>> >>>>> SPARK-24920 <https://issues.apache.org/jira/browse/SPARK-24920> Allow >>>>> sharing Netty's memory pool allocators >>>>> >>>>> SPARK-25250 <https://issues.apache.org/jira/browse/SPARK-25250> Fix >>>>> race condition with tasks running when new attempt for same stage is >>>>> created leads to other task in the next attempt running on the same >>>>> partition id retry multiple times >>>>> >>>>> SPARK-25341 <https://issues.apache.org/jira/browse/SPARK-25341> >>>>> Support rolling back a shuffle map stage and re-generate the shuffle files >>>>> >>>>> SPARK-25348 <https://issues.apache.org/jira/browse/SPARK-25348> Data >>>>> source for binary files >>>>> >>>>> SPARK-25603 <https://issues.apache.org/jira/browse/SPARK-25603> >>>>> Generalize Nested Column Pruning >>>>> >>>>> SPARK-26132 <https://issues.apache.org/jira/browse/SPARK-26132> >>>>> Remove support for Scala 2.11 in Spark 3.0.0 >>>>> >>>>> SPARK-26215 <https://issues.apache.org/jira/browse/SPARK-26215> >>>>> define reserved keywords after SQL standard >>>>> >>>>> SPARK-26412 <https://issues.apache.org/jira/browse/SPARK-26412> Allow >>>>> Pandas UDF to take an iterator of pd.DataFrames >>>>> >>>>> SPARK-26785 <https://issues.apache.org/jira/browse/SPARK-26785> data >>>>> source v2 API refactor: streaming write >>>>> >>>>> SPARK-26956 <https://issues.apache.org/jira/browse/SPARK-26956> >>>>> remove streaming output mode from data source v2 APIs >>>>> >>>>> SPARK-27064 <https://issues.apache.org/jira/browse/SPARK-27064> >>>>> create StreamingWrite at the beginning of streaming execution >>>>> >>>>> SPARK-27119 <https://issues.apache.org/jira/browse/SPARK-27119> Do >>>>> not infer schema when reading Hive serde table with native data source >>>>> >>>>> SPARK-27225 <https://issues.apache.org/jira/browse/SPARK-27225> >>>>> Implement join strategy hints >>>>> >>>>> SPARK-27240 <https://issues.apache.org/jira/browse/SPARK-27240> Use >>>>> pandas DataFrame for struct type argument in Scalar Pandas UDF >>>>> >>>>> SPARK-27338 <https://issues.apache.org/jira/browse/SPARK-27338> Fix >>>>> deadlock between TaskMemoryManager and >>>>> UnsafeExternalSorter$SpillableIterator >>>>> >>>>> SPARK-27396 <https://issues.apache.org/jira/browse/SPARK-27396> >>>>> Public APIs for extended Columnar Processing Support >>>>> >>>>> SPARK-27589 <https://issues.apache.org/jira/browse/SPARK-27589> >>>>> Re-implement file sources with data source V2 API >>>>> >>>>> SPARK-27677 <https://issues.apache.org/jira/browse/SPARK-27677> >>>>> Disk-persisted RDD blocks served by shuffle service, and ignored for >>>>> Dynamic Allocation >>>>> >>>>> SPARK-27699 <https://issues.apache.org/jira/browse/SPARK-27699> >>>>> Partially push down disjunctive predicated in Parquet/ORC >>>>> >>>>> SPARK-27763 <https://issues.apache.org/jira/browse/SPARK-27763> Port >>>>> test cases from PostgreSQL to Spark SQL (ongoing) >>>>> >>>>> SPARK-27884 <https://issues.apache.org/jira/browse/SPARK-27884> >>>>> Deprecate Python 2 support >>>>> >>>>> SPARK-27921 <https://issues.apache.org/jira/browse/SPARK-27921> >>>>> Convert applicable *.sql tests into UDF integrated test base >>>>> >>>>> SPARK-27963 <https://issues.apache.org/jira/browse/SPARK-27963> Allow >>>>> dynamic allocation without an external shuffle service >>>>> >>>>> SPARK-28177 <https://issues.apache.org/jira/browse/SPARK-28177> >>>>> Adjust post shuffle partition number in adaptive execution >>>>> >>>>> SPARK-28372 <https://issues.apache.org/jira/browse/SPARK-28372> >>>>> Document Spark WEB UI >>>>> >>>>> SPARK-28399 <https://issues.apache.org/jira/browse/SPARK-28399> >>>>> RobustScaler feature transformer >>>>> >>>>> SPARK-28426 <https://issues.apache.org/jira/browse/SPARK-28426> >>>>> Metadata Handling in Thrift Server >>>>> >>>>> SPARK-28588 <https://issues.apache.org/jira/browse/SPARK-28588> Build >>>>> a SQL reference doc (ongoing) >>>>> >>>>> SPARK-28608 <https://issues.apache.org/jira/browse/SPARK-28608> >>>>> Improve test coverage of ThriftServer >>>>> >>>>> SPARK-28753 <https://issues.apache.org/jira/browse/SPARK-28753> >>>>> Dynamically reuse subqueries in AQE >>>>> >>>>> SPARK-28855 <https://issues.apache.org/jira/browse/SPARK-28855> >>>>> Remove outdated Experimental, Evolving annotations >>>>> SPARK-25908 <https://issues.apache.org/jira/browse/SPARK-25908> >>>>> SPARK-28980 <https://issues.apache.org/jira/browse/SPARK-28980> >>>>> Remove deprecated items since <= 2.2.0 >>>>> >>>>> Cheers, >>>>> >>>>> Xingbo >>>>> >>>>
