[jira] [Commented] (SPARK-29446) Upgrade netty-all to 4.1.42 and fix vulnerabilities.
[ https://issues.apache.org/jira/browse/SPARK-29446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949958#comment-16949958 ] Iskender Unlu commented on SPARK-29446: --- I will have a look around this issue. > Upgrade netty-all to 4.1.42 and fix vulnerabilities. > > > Key: SPARK-29446 > URL: https://issues.apache.org/jira/browse/SPARK-29446 > Project: Spark > Issue Type: Dependency upgrade > Components: Build >Affects Versions: 3.0.0 >Reporter: jiaan.geng >Priority: Major > > The current code uses io.netty:netty-all:jar:4.1.17 and it will cause a > security vulnerabilities. We could get some security info from > [https://www.tenable.com/cve/CVE-2019-16869]. > This reference remind to upgrate the version of netty-all to 4.1.42 or later. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28015) Invalid date formats should throw an exception
[
https://issues.apache.org/jira/browse/SPARK-28015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16881887#comment-16881887
]
Iskender Unlu commented on SPARK-28015:
---
Maxim Gekk, you might prepare a PR.
> Invalid date formats should throw an exception
> --
>
> Key: SPARK-28015
> URL: https://issues.apache.org/jira/browse/SPARK-28015
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.4.3
>Reporter: Yuming Wang
>Priority: Major
>
> Invalid date formats should throw an exception:
> {code:sql}
> SELECT date '1999 08 01'
> 1999-01-01
> {code}
> Supported date formats:
> https://github.com/apache/spark/blob/ab8710b57916a129fcb89464209361120d224535/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala#L365-L374
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28015) Invalid date formats should throw an exception
[
https://issues.apache.org/jira/browse/SPARK-28015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877275#comment-16877275
]
Iskender Unlu commented on SPARK-28015:
---
I will try to work on this issue as my first contribution trial.
> Invalid date formats should throw an exception
> --
>
> Key: SPARK-28015
> URL: https://issues.apache.org/jira/browse/SPARK-28015
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.4.3
>Reporter: Yuming Wang
>Priority: Major
>
> Invalid date formats should throw an exception:
> {code:sql}
> SELECT date '1999 08 01'
> 1999-01-01
> {code}
> Supported date formats:
> https://github.com/apache/spark/blob/ab8710b57916a129fcb89464209361120d224535/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala#L365-L374
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28090) Spark hangs when an execution plan has many projections on nested structs
[
https://issues.apache.org/jira/browse/SPARK-28090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876150#comment-16876150
]
Iskender Unlu commented on SPARK-28090:
---
I will try to work on this issue as my first contribution trial.
> Spark hangs when an execution plan has many projections on nested structs
> -
>
> Key: SPARK-28090
> URL: https://issues.apache.org/jira/browse/SPARK-28090
> Project: Spark
> Issue Type: Bug
> Components: Optimizer
>Affects Versions: 2.4.3
> Environment: Tried in
> * Spark 2.2.1, Spark 2.4.3 in local mode on Linux, MasOS and Windows
> * Spark 2.4.3 / Yarn on a Linux cluster
>Reporter: Ruslan Yushchenko
>Priority: Major
>
> This was already posted (#28016), but the provided example didn't always
> reproduce the error. This example consistently reproduces the issue.
> Spark applications freeze on execution plan optimization stage (Catalyst)
> when a logical execution plan contains a lot of projections that operate on
> nested struct fields.
> The code listed below demonstrates the issue.
> To reproduce the Spark App does the following:
> * A small dataframe is created from a JSON example.
> * Several nested transformations (negation of a number) are applied on
> struct fields and each time a new struct field is created.
> * Once more than 9 such transformations are applied the Catalyst optimizer
> freezes on optimizing the execution plan.
> * You can control the freezing by choosing different upper bound for the
> Range. E.g. it will work file if the upper bound is 5, but will hang is the
> bound is 10.
> {code:java}
> package com.example
> import org.apache.spark.sql._
> import org.apache.spark.sql.functions._
> import org.apache.spark.sql.types.{StructField, StructType}
> import scala.collection.mutable.ListBuffer
> object SparkApp1IssueSelfContained {
> // A sample data for a dataframe with nested structs
> val sample: List[String] =
> """ { "numerics": {"num1": 101, "num2": 102, "num3": 103, "num4": 104,
> "num5": 105, "num6": 106, "num7": 107, "num8": 108, "num9": 109, "num10":
> 110, "num11": 111, "num12": 112, "num13": 113, "num14": 114, "num15": 115} }
> """ ::
> """ { "numerics": {"num1": 201, "num2": 202, "num3": 203, "num4": 204,
> "num5": 205, "num6": 206, "num7": 207, "num8": 208, "num9": 209, "num10":
> 210, "num11": 211, "num12": 212, "num13": 213, "num14": 214, "num15": 215} }
> """ ::
> """ { "numerics": {"num1": 301, "num2": 302, "num3": 303, "num4": 304,
> "num5": 305, "num6": 306, "num7": 307, "num8": 308, "num9": 309, "num10":
> 310, "num11": 311, "num12": 312, "num13": 313, "num14": 314, "num15": 315} }
> """ ::
> Nil
> /**
> * Transforms a column inside a nested struct. The transformed value will
> be put into a new field of that nested struct
> *
> * The output column name can omit the full path as the field will be
> created at the same level of nesting as the input column.
> *
> * @param inputColumnName A column name for which to apply the
> transformation, e.g. `company.employee.firstName`.
> * @param outputColumnName The output column name. The path is optional,
> e.g. you can use `transformedName` instead of
> `company.employee.transformedName`.
> * @param expression A function that applies a transformation to a
> column as a Spark expression.
> * @return A dataframe with a new field that contains transformed values.
> */
> def transformInsideNestedStruct(df: DataFrame,
> inputColumnName: String,
> outputColumnName: String,
> expression: Column => Column): DataFrame = {
> def mapStruct(schema: StructType, path: Seq[String], parentColumn:
> Option[Column] = None): Seq[Column] = {
> val mappedFields = new ListBuffer[Column]()
> def handleMatchedLeaf(field: StructField, curColumn: Column):
> Seq[Column] = {
> val newColumn = expression(curColumn).as(outputColumnName)
> mappedFields += newColumn
> Seq(curColumn)
> }
> def handleMatchedNonLeaf(field: StructField, curColumn: Column):
> Seq[Column] = {
> // Non-leaf columns need to be further processed recursively
> field.dataType match {
> case dt: StructType => Seq(struct(mapStruct(dt, path.tail,
> Some(curColumn)): _*).as(field.name))
> case _ => throw new IllegalArgumentException(s"Field
> '${field.name}' is not a struct type.")
> }
> }
> val fieldName = path.head
> val isLeaf = path.lengthCompare(2) < 0
> val newColumns = schema.fields.flatMap(field => {
> // This is the original column (struct field) we want to process
> val
[jira] [Issue Comment Deleted] (SPARK-28090) Spark hangs when an execution plan has many projections on nested structs
[
https://issues.apache.org/jira/browse/SPARK-28090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Iskender Unlu updated SPARK-28090:
--
Comment: was deleted
(was: I will work on this issue.)
> Spark hangs when an execution plan has many projections on nested structs
> -
>
> Key: SPARK-28090
> URL: https://issues.apache.org/jira/browse/SPARK-28090
> Project: Spark
> Issue Type: Bug
> Components: Optimizer
>Affects Versions: 2.4.3
> Environment: Tried in
> * Spark 2.2.1, Spark 2.4.3 in local mode on Linux, MasOS and Windows
> * Spark 2.4.3 / Yarn on a Linux cluster
>Reporter: Ruslan Yushchenko
>Priority: Major
>
> This was already posted (#28016), but the provided example didn't always
> reproduce the error. This example consistently reproduces the issue.
> Spark applications freeze on execution plan optimization stage (Catalyst)
> when a logical execution plan contains a lot of projections that operate on
> nested struct fields.
> The code listed below demonstrates the issue.
> To reproduce the Spark App does the following:
> * A small dataframe is created from a JSON example.
> * Several nested transformations (negation of a number) are applied on
> struct fields and each time a new struct field is created.
> * Once more than 9 such transformations are applied the Catalyst optimizer
> freezes on optimizing the execution plan.
> * You can control the freezing by choosing different upper bound for the
> Range. E.g. it will work file if the upper bound is 5, but will hang is the
> bound is 10.
> {code:java}
> package com.example
> import org.apache.spark.sql._
> import org.apache.spark.sql.functions._
> import org.apache.spark.sql.types.{StructField, StructType}
> import scala.collection.mutable.ListBuffer
> object SparkApp1IssueSelfContained {
> // A sample data for a dataframe with nested structs
> val sample: List[String] =
> """ { "numerics": {"num1": 101, "num2": 102, "num3": 103, "num4": 104,
> "num5": 105, "num6": 106, "num7": 107, "num8": 108, "num9": 109, "num10":
> 110, "num11": 111, "num12": 112, "num13": 113, "num14": 114, "num15": 115} }
> """ ::
> """ { "numerics": {"num1": 201, "num2": 202, "num3": 203, "num4": 204,
> "num5": 205, "num6": 206, "num7": 207, "num8": 208, "num9": 209, "num10":
> 210, "num11": 211, "num12": 212, "num13": 213, "num14": 214, "num15": 215} }
> """ ::
> """ { "numerics": {"num1": 301, "num2": 302, "num3": 303, "num4": 304,
> "num5": 305, "num6": 306, "num7": 307, "num8": 308, "num9": 309, "num10":
> 310, "num11": 311, "num12": 312, "num13": 313, "num14": 314, "num15": 315} }
> """ ::
> Nil
> /**
> * Transforms a column inside a nested struct. The transformed value will
> be put into a new field of that nested struct
> *
> * The output column name can omit the full path as the field will be
> created at the same level of nesting as the input column.
> *
> * @param inputColumnName A column name for which to apply the
> transformation, e.g. `company.employee.firstName`.
> * @param outputColumnName The output column name. The path is optional,
> e.g. you can use `transformedName` instead of
> `company.employee.transformedName`.
> * @param expression A function that applies a transformation to a
> column as a Spark expression.
> * @return A dataframe with a new field that contains transformed values.
> */
> def transformInsideNestedStruct(df: DataFrame,
> inputColumnName: String,
> outputColumnName: String,
> expression: Column => Column): DataFrame = {
> def mapStruct(schema: StructType, path: Seq[String], parentColumn:
> Option[Column] = None): Seq[Column] = {
> val mappedFields = new ListBuffer[Column]()
> def handleMatchedLeaf(field: StructField, curColumn: Column):
> Seq[Column] = {
> val newColumn = expression(curColumn).as(outputColumnName)
> mappedFields += newColumn
> Seq(curColumn)
> }
> def handleMatchedNonLeaf(field: StructField, curColumn: Column):
> Seq[Column] = {
> // Non-leaf columns need to be further processed recursively
> field.dataType match {
> case dt: StructType => Seq(struct(mapStruct(dt, path.tail,
> Some(curColumn)): _*).as(field.name))
> case _ => throw new IllegalArgumentException(s"Field
> '${field.name}' is not a struct type.")
> }
> }
> val fieldName = path.head
> val isLeaf = path.lengthCompare(2) < 0
> val newColumns = schema.fields.flatMap(field => {
> // This is the original column (struct field) we want to process
> val curColumn = parentColumn match {
>
[jira] [Commented] (SPARK-28090) Spark hangs when an execution plan has many projections on nested structs
[
https://issues.apache.org/jira/browse/SPARK-28090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876145#comment-16876145
]
Iskender Unlu commented on SPARK-28090:
---
I will work on this issue.
> Spark hangs when an execution plan has many projections on nested structs
> -
>
> Key: SPARK-28090
> URL: https://issues.apache.org/jira/browse/SPARK-28090
> Project: Spark
> Issue Type: Bug
> Components: Optimizer
>Affects Versions: 2.4.3
> Environment: Tried in
> * Spark 2.2.1, Spark 2.4.3 in local mode on Linux, MasOS and Windows
> * Spark 2.4.3 / Yarn on a Linux cluster
>Reporter: Ruslan Yushchenko
>Priority: Major
>
> This was already posted (#28016), but the provided example didn't always
> reproduce the error. This example consistently reproduces the issue.
> Spark applications freeze on execution plan optimization stage (Catalyst)
> when a logical execution plan contains a lot of projections that operate on
> nested struct fields.
> The code listed below demonstrates the issue.
> To reproduce the Spark App does the following:
> * A small dataframe is created from a JSON example.
> * Several nested transformations (negation of a number) are applied on
> struct fields and each time a new struct field is created.
> * Once more than 9 such transformations are applied the Catalyst optimizer
> freezes on optimizing the execution plan.
> * You can control the freezing by choosing different upper bound for the
> Range. E.g. it will work file if the upper bound is 5, but will hang is the
> bound is 10.
> {code:java}
> package com.example
> import org.apache.spark.sql._
> import org.apache.spark.sql.functions._
> import org.apache.spark.sql.types.{StructField, StructType}
> import scala.collection.mutable.ListBuffer
> object SparkApp1IssueSelfContained {
> // A sample data for a dataframe with nested structs
> val sample: List[String] =
> """ { "numerics": {"num1": 101, "num2": 102, "num3": 103, "num4": 104,
> "num5": 105, "num6": 106, "num7": 107, "num8": 108, "num9": 109, "num10":
> 110, "num11": 111, "num12": 112, "num13": 113, "num14": 114, "num15": 115} }
> """ ::
> """ { "numerics": {"num1": 201, "num2": 202, "num3": 203, "num4": 204,
> "num5": 205, "num6": 206, "num7": 207, "num8": 208, "num9": 209, "num10":
> 210, "num11": 211, "num12": 212, "num13": 213, "num14": 214, "num15": 215} }
> """ ::
> """ { "numerics": {"num1": 301, "num2": 302, "num3": 303, "num4": 304,
> "num5": 305, "num6": 306, "num7": 307, "num8": 308, "num9": 309, "num10":
> 310, "num11": 311, "num12": 312, "num13": 313, "num14": 314, "num15": 315} }
> """ ::
> Nil
> /**
> * Transforms a column inside a nested struct. The transformed value will
> be put into a new field of that nested struct
> *
> * The output column name can omit the full path as the field will be
> created at the same level of nesting as the input column.
> *
> * @param inputColumnName A column name for which to apply the
> transformation, e.g. `company.employee.firstName`.
> * @param outputColumnName The output column name. The path is optional,
> e.g. you can use `transformedName` instead of
> `company.employee.transformedName`.
> * @param expression A function that applies a transformation to a
> column as a Spark expression.
> * @return A dataframe with a new field that contains transformed values.
> */
> def transformInsideNestedStruct(df: DataFrame,
> inputColumnName: String,
> outputColumnName: String,
> expression: Column => Column): DataFrame = {
> def mapStruct(schema: StructType, path: Seq[String], parentColumn:
> Option[Column] = None): Seq[Column] = {
> val mappedFields = new ListBuffer[Column]()
> def handleMatchedLeaf(field: StructField, curColumn: Column):
> Seq[Column] = {
> val newColumn = expression(curColumn).as(outputColumnName)
> mappedFields += newColumn
> Seq(curColumn)
> }
> def handleMatchedNonLeaf(field: StructField, curColumn: Column):
> Seq[Column] = {
> // Non-leaf columns need to be further processed recursively
> field.dataType match {
> case dt: StructType => Seq(struct(mapStruct(dt, path.tail,
> Some(curColumn)): _*).as(field.name))
> case _ => throw new IllegalArgumentException(s"Field
> '${field.name}' is not a struct type.")
> }
> }
> val fieldName = path.head
> val isLeaf = path.lengthCompare(2) < 0
> val newColumns = schema.fields.flatMap(field => {
> // This is the original column (struct field) we want to process
> val curColumn = parentColumn match {
>
[jira] [Created] (SPARK-18314) CLONE - Support model save/load in Python's LDA
Iskender Unlu created SPARK-18314: - Summary: CLONE - Support model save/load in Python's LDA Key: SPARK-18314 URL: https://issues.apache.org/jira/browse/SPARK-18314 Project: Spark Issue Type: Sub-task Components: MLlib, PySpark Reporter: Iskender Unlu Assignee: Manoj Kumar Priority: Minor Fix For: 1.5.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-18312) CLONE - Support model save/load in Python's GaussianMixture
Iskender Unlu created SPARK-18312: - Summary: CLONE - Support model save/load in Python's GaussianMixture Key: SPARK-18312 URL: https://issues.apache.org/jira/browse/SPARK-18312 Project: Spark Issue Type: Sub-task Components: MLlib, PySpark Reporter: Iskender Unlu Assignee: Manoj Kumar Priority: Minor Fix For: 1.5.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-18313) CLONE - Support model save/load in Python's FPGrowth
Iskender Unlu created SPARK-18313: - Summary: CLONE - Support model save/load in Python's FPGrowth Key: SPARK-18313 URL: https://issues.apache.org/jira/browse/SPARK-18313 Project: Spark Issue Type: Sub-task Components: MLlib, PySpark Reporter: Iskender Unlu Assignee: Kai Jiang Priority: Minor Fix For: 2.0.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-18305) CLONE - Python API for ML model import/export
Iskender Unlu created SPARK-18305: - Summary: CLONE - Python API for ML model import/export Key: SPARK-18305 URL: https://issues.apache.org/jira/browse/SPARK-18305 Project: Spark Issue Type: Umbrella Components: MLlib, PySpark Affects Versions: 1.3.0 Reporter: Iskender Unlu Assignee: Joseph K. Bradley Priority: Critical Many ML models support save/load in Scala and Java. The Python API needs this. It should mostly be a simple matter of calling the JVM methods for save/load, except for models which are stored in Python (e.g., linear models). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-18310) CLONE - Support model save/load in Python's KMeans
Iskender Unlu created SPARK-18310: - Summary: CLONE - Support model save/load in Python's KMeans Key: SPARK-18310 URL: https://issues.apache.org/jira/browse/SPARK-18310 Project: Spark Issue Type: Sub-task Components: MLlib, PySpark Affects Versions: 1.3.0 Reporter: Iskender Unlu Assignee: Xusen Yin Fix For: 1.4.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-18309) CLONE - Support model save/load in Python's tree models
Iskender Unlu created SPARK-18309: - Summary: CLONE - Support model save/load in Python's tree models Key: SPARK-18309 URL: https://issues.apache.org/jira/browse/SPARK-18309 Project: Spark Issue Type: Sub-task Components: MLlib, PySpark Reporter: Iskender Unlu Assignee: Xiangrui Meng Fix For: 1.3.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-18307) CLONE - Support model save/load in Python's linear models
Iskender Unlu created SPARK-18307: - Summary: CLONE - Support model save/load in Python's linear models Key: SPARK-18307 URL: https://issues.apache.org/jira/browse/SPARK-18307 Project: Spark Issue Type: Sub-task Components: MLlib, PySpark Reporter: Iskender Unlu Assignee: Yanbo Liang Fix For: 1.4.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-18308) CLONE - Support model save/load in Python's naive Bayes
Iskender Unlu created SPARK-18308: - Summary: CLONE - Support model save/load in Python's naive Bayes Key: SPARK-18308 URL: https://issues.apache.org/jira/browse/SPARK-18308 Project: Spark Issue Type: Sub-task Components: MLlib, PySpark Reporter: Iskender Unlu Assignee: Xusen Yin Fix For: 1.4.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-18306) CLONE - Support model save/load in Python's ALS.
Iskender Unlu created SPARK-18306: - Summary: CLONE - Support model save/load in Python's ALS. Key: SPARK-18306 URL: https://issues.apache.org/jira/browse/SPARK-18306 Project: Spark Issue Type: Sub-task Components: MLlib, PySpark Reporter: Iskender Unlu Assignee: Xiangrui Meng Priority: Minor Fix For: 1.3.0 It should be a simple wrapper of the Scala's implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-18311) CLONE - Support model save/load in Python's Word2Vec
Iskender Unlu created SPARK-18311: - Summary: CLONE - Support model save/load in Python's Word2Vec Key: SPARK-18311 URL: https://issues.apache.org/jira/browse/SPARK-18311 Project: Spark Issue Type: Sub-task Components: MLlib, PySpark Reporter: Iskender Unlu Assignee: Yu Ishikawa Priority: Minor Fix For: 1.5.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-18304) [SPARK-5256] Improving MLlib optimization APIs
Iskender Unlu created SPARK-18304: - Summary: [SPARK-5256] Improving MLlib optimization APIs Key: SPARK-18304 URL: https://issues.apache.org/jira/browse/SPARK-18304 Project: Spark Issue Type: Bug Reporter: Iskender Unlu -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-18303) CLONE - Improving MLlib optimization APIs
Iskender Unlu created SPARK-18303: - Summary: CLONE - Improving MLlib optimization APIs Key: SPARK-18303 URL: https://issues.apache.org/jira/browse/SPARK-18303 Project: Spark Issue Type: Umbrella Components: MLlib Reporter: Iskender Unlu *Goal*: Improve APIs for optimization *Motivation*: There have been several disjoint mentions of improving the optimization APIs to make them more pluggable, extensible, etc. This JIRA is a place to discuss what API changes are necessary for the long term, and to provide links to other relevant JIRAs. Eventually, I hope this leads to a design doc outlining: * current issues * requirements such as supporting many types of objective functions, optimization algorithms, and parameters to those algorithms * ideal API * breakdown of smaller JIRAs needed to achieve that API I will soon create an initial design doc, and I will try to watch this JIRA and include ideas from JIRA comments. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
