[jira] [Commented] (SPARK-37421) Inline type hints for python/pyspark/mllib/evaluation.py
[ https://issues.apache.org/jira/browse/SPARK-37421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17447223#comment-17447223 ] Apache Spark commented on SPARK-37421: -- User 'dchvn' has created a pull request for this issue: https://github.com/apache/spark/pull/34680 > Inline type hints for python/pyspark/mllib/evaluation.py > > > Key: SPARK-37421 > URL: https://issues.apache.org/jira/browse/SPARK-37421 > Project: Spark > Issue Type: Sub-task > Components: MLlib, PySpark >Affects Versions: 3.3.0 >Reporter: Maciej Szymkiewicz >Priority: Major > > Inline type hints from python/pyspark/mlib/evaluation.pyi to > python/pyspark/mllib/evaluation.py -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-37421) Inline type hints for python/pyspark/mllib/evaluation.py
[ https://issues.apache.org/jira/browse/SPARK-37421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37421: Assignee: (was: Apache Spark) > Inline type hints for python/pyspark/mllib/evaluation.py > > > Key: SPARK-37421 > URL: https://issues.apache.org/jira/browse/SPARK-37421 > Project: Spark > Issue Type: Sub-task > Components: MLlib, PySpark >Affects Versions: 3.3.0 >Reporter: Maciej Szymkiewicz >Priority: Major > > Inline type hints from python/pyspark/mlib/evaluation.pyi to > python/pyspark/mllib/evaluation.py -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-37421) Inline type hints for python/pyspark/mllib/evaluation.py
[ https://issues.apache.org/jira/browse/SPARK-37421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37421: Assignee: Apache Spark > Inline type hints for python/pyspark/mllib/evaluation.py > > > Key: SPARK-37421 > URL: https://issues.apache.org/jira/browse/SPARK-37421 > Project: Spark > Issue Type: Sub-task > Components: MLlib, PySpark >Affects Versions: 3.3.0 >Reporter: Maciej Szymkiewicz >Assignee: Apache Spark >Priority: Major > > Inline type hints from python/pyspark/mlib/evaluation.pyi to > python/pyspark/mllib/evaluation.py -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-37421) Inline type hints for python/pyspark/mllib/evaluation.py
[ https://issues.apache.org/jira/browse/SPARK-37421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17447221#comment-17447221 ] Apache Spark commented on SPARK-37421: -- User 'dchvn' has created a pull request for this issue: https://github.com/apache/spark/pull/34680 > Inline type hints for python/pyspark/mllib/evaluation.py > > > Key: SPARK-37421 > URL: https://issues.apache.org/jira/browse/SPARK-37421 > Project: Spark > Issue Type: Sub-task > Components: MLlib, PySpark >Affects Versions: 3.3.0 >Reporter: Maciej Szymkiewicz >Priority: Major > > Inline type hints from python/pyspark/mlib/evaluation.pyi to > python/pyspark/mllib/evaluation.py -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-37437) Remove unused profile
[ https://issues.apache.org/jira/browse/SPARK-37437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37437: Assignee: Apache Spark > Remove unused profile > - > > Key: SPARK-37437 > URL: https://issues.apache.org/jira/browse/SPARK-37437 > Project: Spark > Issue Type: Task > Components: SQL >Affects Versions: 3.2.0 >Reporter: angerszhu >Assignee: Apache Spark >Priority: Major > > remove > > hive-2.3 > > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-37437) Remove unused profile
[ https://issues.apache.org/jira/browse/SPARK-37437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37437: Assignee: (was: Apache Spark) > Remove unused profile > - > > Key: SPARK-37437 > URL: https://issues.apache.org/jira/browse/SPARK-37437 > Project: Spark > Issue Type: Task > Components: SQL >Affects Versions: 3.2.0 >Reporter: angerszhu >Priority: Major > > remove > > hive-2.3 > > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-37437) Remove unused profile
[ https://issues.apache.org/jira/browse/SPARK-37437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17447218#comment-17447218 ] Apache Spark commented on SPARK-37437: -- User 'AngersZh' has created a pull request for this issue: https://github.com/apache/spark/pull/34679 > Remove unused profile > - > > Key: SPARK-37437 > URL: https://issues.apache.org/jira/browse/SPARK-37437 > Project: Spark > Issue Type: Task > Components: SQL >Affects Versions: 3.2.0 >Reporter: angerszhu >Priority: Major > > remove > > hive-2.3 > > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-37437) Remove unused profile
[ https://issues.apache.org/jira/browse/SPARK-37437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17447219#comment-17447219 ] Apache Spark commented on SPARK-37437: -- User 'AngersZh' has created a pull request for this issue: https://github.com/apache/spark/pull/34679 > Remove unused profile > - > > Key: SPARK-37437 > URL: https://issues.apache.org/jira/browse/SPARK-37437 > Project: Spark > Issue Type: Task > Components: SQL >Affects Versions: 3.2.0 >Reporter: angerszhu >Priority: Major > > remove > > hive-2.3 > > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-37437) Remove unused profile
angerszhu created SPARK-37437: - Summary: Remove unused profile Key: SPARK-37437 URL: https://issues.apache.org/jira/browse/SPARK-37437 Project: Spark Issue Type: Task Components: SQL Affects Versions: 3.2.0 Reporter: angerszhu remove hive-2.3 -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-37407) Inline type hints for python/pyspark/ml/functions.py
[ https://issues.apache.org/jira/browse/SPARK-37407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37407: Assignee: Apache Spark > Inline type hints for python/pyspark/ml/functions.py > > > Key: SPARK-37407 > URL: https://issues.apache.org/jira/browse/SPARK-37407 > Project: Spark > Issue Type: Sub-task > Components: ML, PySpark >Affects Versions: 3.3.0 >Reporter: Maciej Szymkiewicz >Assignee: Apache Spark >Priority: Major > > Inline type hints from python/pyspark/ml/functions.pyi to > python/pyspark/ml/functions.py. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-37407) Inline type hints for python/pyspark/ml/functions.py
[ https://issues.apache.org/jira/browse/SPARK-37407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37407: Assignee: (was: Apache Spark) > Inline type hints for python/pyspark/ml/functions.py > > > Key: SPARK-37407 > URL: https://issues.apache.org/jira/browse/SPARK-37407 > Project: Spark > Issue Type: Sub-task > Components: ML, PySpark >Affects Versions: 3.3.0 >Reporter: Maciej Szymkiewicz >Priority: Major > > Inline type hints from python/pyspark/ml/functions.pyi to > python/pyspark/ml/functions.py. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-37407) Inline type hints for python/pyspark/ml/functions.py
[ https://issues.apache.org/jira/browse/SPARK-37407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17447210#comment-17447210 ] Apache Spark commented on SPARK-37407: -- User 'dchvn' has created a pull request for this issue: https://github.com/apache/spark/pull/34678 > Inline type hints for python/pyspark/ml/functions.py > > > Key: SPARK-37407 > URL: https://issues.apache.org/jira/browse/SPARK-37407 > Project: Spark > Issue Type: Sub-task > Components: ML, PySpark >Affects Versions: 3.3.0 >Reporter: Maciej Szymkiewicz >Priority: Major > > Inline type hints from python/pyspark/ml/functions.pyi to > python/pyspark/ml/functions.py. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-37436) Uses Python's standard string formatter for SQL API in pandas API on Spark
[
https://issues.apache.org/jira/browse/SPARK-37436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17447208#comment-17447208
]
Apache Spark commented on SPARK-37436:
--
User 'HyukjinKwon' has created a pull request for this issue:
https://github.com/apache/spark/pull/34677
> Uses Python's standard string formatter for SQL API in pandas API on Spark
> --
>
> Key: SPARK-37436
> URL: https://issues.apache.org/jira/browse/SPARK-37436
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
>Affects Versions: 3.3.0
>Reporter: Hyukjin Kwon
>Priority: Major
>
> Currently {{pyspark.pandas.sql}} uses its own hacky parser:
> https://github.com/apache/spark/blob/master/python/pyspark/pandas/sql_processor.py
> We should ideally switch it to the standard Python formatter
> https://docs.python.org/3/library/string.html#custom-string-formatting
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-37436) Uses Python's standard string formatter for SQL API in pandas API on Spark
[
https://issues.apache.org/jira/browse/SPARK-37436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17447207#comment-17447207
]
Apache Spark commented on SPARK-37436:
--
User 'HyukjinKwon' has created a pull request for this issue:
https://github.com/apache/spark/pull/34677
> Uses Python's standard string formatter for SQL API in pandas API on Spark
> --
>
> Key: SPARK-37436
> URL: https://issues.apache.org/jira/browse/SPARK-37436
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
>Affects Versions: 3.3.0
>Reporter: Hyukjin Kwon
>Priority: Major
>
> Currently {{pyspark.pandas.sql}} uses its own hacky parser:
> https://github.com/apache/spark/blob/master/python/pyspark/pandas/sql_processor.py
> We should ideally switch it to the standard Python formatter
> https://docs.python.org/3/library/string.html#custom-string-formatting
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-37436) Uses Python's standard string formatter for SQL API in pandas API on Spark
[
https://issues.apache.org/jira/browse/SPARK-37436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-37436:
Assignee: Apache Spark
> Uses Python's standard string formatter for SQL API in pandas API on Spark
> --
>
> Key: SPARK-37436
> URL: https://issues.apache.org/jira/browse/SPARK-37436
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
>Affects Versions: 3.3.0
>Reporter: Hyukjin Kwon
>Assignee: Apache Spark
>Priority: Major
>
> Currently {{pyspark.pandas.sql}} uses its own hacky parser:
> https://github.com/apache/spark/blob/master/python/pyspark/pandas/sql_processor.py
> We should ideally switch it to the standard Python formatter
> https://docs.python.org/3/library/string.html#custom-string-formatting
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-37436) Uses Python's standard string formatter for SQL API in pandas API on Spark
[
https://issues.apache.org/jira/browse/SPARK-37436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-37436:
Assignee: (was: Apache Spark)
> Uses Python's standard string formatter for SQL API in pandas API on Spark
> --
>
> Key: SPARK-37436
> URL: https://issues.apache.org/jira/browse/SPARK-37436
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
>Affects Versions: 3.3.0
>Reporter: Hyukjin Kwon
>Priority: Major
>
> Currently {{pyspark.pandas.sql}} uses its own hacky parser:
> https://github.com/apache/spark/blob/master/python/pyspark/pandas/sql_processor.py
> We should ideally switch it to the standard Python formatter
> https://docs.python.org/3/library/string.html#custom-string-formatting
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-37436) Uses Python's standard string formatter for SQL API in pandas API on Spark
[
https://issues.apache.org/jira/browse/SPARK-37436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-37436:
-
Issue Type: Improvement (was: Test)
> Uses Python's standard string formatter for SQL API in pandas API on Spark
> --
>
> Key: SPARK-37436
> URL: https://issues.apache.org/jira/browse/SPARK-37436
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
>Affects Versions: 3.3.0
>Reporter: Hyukjin Kwon
>Priority: Major
>
> Currently {{pyspark.pandas.sql}} uses its own hacky parser:
> https://github.com/apache/spark/blob/master/python/pyspark/pandas/sql_processor.py
> We should ideally switch it to the standard Python formatter
> https://docs.python.org/3/library/string.html#custom-string-formatting
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-37436) Uses Python's standard string formatter for SQL API in pandas API on Spark
Hyukjin Kwon created SPARK-37436:
Summary: Uses Python's standard string formatter for SQL API in
pandas API on Spark
Key: SPARK-37436
URL: https://issues.apache.org/jira/browse/SPARK-37436
Project: Spark
Issue Type: Test
Components: PySpark
Affects Versions: 3.3.0
Reporter: Hyukjin Kwon
Currently {{pyspark.pandas.sql}} uses its own hacky parser:
https://github.com/apache/spark/blob/master/python/pyspark/pandas/sql_processor.py
We should ideally switch it to the standard Python formatter
https://docs.python.org/3/library/string.html#custom-string-formatting
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-36231) Support arithmetic operations of Series containing Decimal(np.nan)
[
https://issues.apache.org/jira/browse/SPARK-36231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-36231:
Assignee: Yikun Jiang (was: Apache Spark)
> Support arithmetic operations of Series containing Decimal(np.nan)
> ---
>
> Key: SPARK-36231
> URL: https://issues.apache.org/jira/browse/SPARK-36231
> Project: Spark
> Issue Type: Sub-task
> Components: PySpark
>Affects Versions: 3.2.0
>Reporter: Xinrong Meng
>Assignee: Yikun Jiang
>Priority: Major
>
> Arithmetic operations of Series containing Decimal(np.nan) raise
> java.lang.NullPointerException in driver. An example is shown as below:
> {code:java}
> >>> pser = pd.Series([decimal.Decimal(1.0), decimal.Decimal(2.0),
> >>> decimal.Decimal(np.nan)])
> >>> psser = ps.from_pandas(pser)
> >>> pser + 1
> 0 2
> 1 3
> 2 NaN
> >>> psser + 1
> Driver stacktrace:
> at
> org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2259)
> at
> org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2208)
> at
> org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2207)
> at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
> at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
> at
> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2207)
> at
> org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1084)
> at
> org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1084)
> at scala.Option.foreach(Option.scala:407)
> at
> org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1084)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2446)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2388)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2377)
> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
> at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:873)
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:2208)
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:2303)
> at
> org.apache.spark.sql.Dataset.$anonfun$collectAsArrowToPython$5(Dataset.scala:3648)
> at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
> at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1437)
> at
> org.apache.spark.sql.Dataset.$anonfun$collectAsArrowToPython$2(Dataset.scala:3652)
> at
> org.apache.spark.sql.Dataset.$anonfun$collectAsArrowToPython$2$adapted(Dataset.scala:3629)
> at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3706)
> at
> org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)
> at
> org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
> at
> org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)
> at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:774)
> at
> org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
> at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3704)
> at
> org.apache.spark.sql.Dataset.$anonfun$collectAsArrowToPython$1(Dataset.scala:3629)
> at
> org.apache.spark.sql.Dataset.$anonfun$collectAsArrowToPython$1$adapted(Dataset.scala:3628)
> at
> org.apache.spark.security.SocketAuthServer$.$anonfun$serveToStream$2(SocketAuthServer.scala:139)
> at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
> at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1437)
> at
> org.apache.spark.security.SocketAuthServer$.$anonfun$serveToStream$1(SocketAuthServer.scala:141)
> at
> org.apache.spark.security.SocketAuthServer$.$anonfun$serveToStream$1$adapted(SocketAuthServer.scala:136)
> at
> org.apache.spark.security.SocketFuncServer.handleConnection(SocketAuthServer.scala:113)
> at
> org.apache.spark.security.SocketFuncServer.handleConnection(SocketAuthServer.scala:107)
> at
> org.apache.spark.security.SocketAuthServer$$anon$1.$anonfun$run$4(SocketAuthServer.scala:68)
> at scala.util.Try$.apply(Try.scala:213)
> at
> org.apache.spark.security.SocketAuthServer$$anon$1.run(SocketAuthServer.scala:68)
> Caused by: java.lang.NullPointerException
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown
> Source)
> at
>
[jira] [Assigned] (SPARK-36231) Support arithmetic operations of Series containing Decimal(np.nan)
[
https://issues.apache.org/jira/browse/SPARK-36231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-36231:
Assignee: Apache Spark (was: Yikun Jiang)
> Support arithmetic operations of Series containing Decimal(np.nan)
> ---
>
> Key: SPARK-36231
> URL: https://issues.apache.org/jira/browse/SPARK-36231
> Project: Spark
> Issue Type: Sub-task
> Components: PySpark
>Affects Versions: 3.2.0
>Reporter: Xinrong Meng
>Assignee: Apache Spark
>Priority: Major
>
> Arithmetic operations of Series containing Decimal(np.nan) raise
> java.lang.NullPointerException in driver. An example is shown as below:
> {code:java}
> >>> pser = pd.Series([decimal.Decimal(1.0), decimal.Decimal(2.0),
> >>> decimal.Decimal(np.nan)])
> >>> psser = ps.from_pandas(pser)
> >>> pser + 1
> 0 2
> 1 3
> 2 NaN
> >>> psser + 1
> Driver stacktrace:
> at
> org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2259)
> at
> org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2208)
> at
> org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2207)
> at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
> at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
> at
> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2207)
> at
> org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1084)
> at
> org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1084)
> at scala.Option.foreach(Option.scala:407)
> at
> org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1084)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2446)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2388)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2377)
> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
> at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:873)
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:2208)
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:2303)
> at
> org.apache.spark.sql.Dataset.$anonfun$collectAsArrowToPython$5(Dataset.scala:3648)
> at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
> at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1437)
> at
> org.apache.spark.sql.Dataset.$anonfun$collectAsArrowToPython$2(Dataset.scala:3652)
> at
> org.apache.spark.sql.Dataset.$anonfun$collectAsArrowToPython$2$adapted(Dataset.scala:3629)
> at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3706)
> at
> org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)
> at
> org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
> at
> org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)
> at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:774)
> at
> org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
> at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3704)
> at
> org.apache.spark.sql.Dataset.$anonfun$collectAsArrowToPython$1(Dataset.scala:3629)
> at
> org.apache.spark.sql.Dataset.$anonfun$collectAsArrowToPython$1$adapted(Dataset.scala:3628)
> at
> org.apache.spark.security.SocketAuthServer$.$anonfun$serveToStream$2(SocketAuthServer.scala:139)
> at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
> at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1437)
> at
> org.apache.spark.security.SocketAuthServer$.$anonfun$serveToStream$1(SocketAuthServer.scala:141)
> at
> org.apache.spark.security.SocketAuthServer$.$anonfun$serveToStream$1$adapted(SocketAuthServer.scala:136)
> at
> org.apache.spark.security.SocketFuncServer.handleConnection(SocketAuthServer.scala:113)
> at
> org.apache.spark.security.SocketFuncServer.handleConnection(SocketAuthServer.scala:107)
> at
> org.apache.spark.security.SocketAuthServer$$anon$1.$anonfun$run$4(SocketAuthServer.scala:68)
> at scala.util.Try$.apply(Try.scala:213)
> at
> org.apache.spark.security.SocketAuthServer$$anon$1.run(SocketAuthServer.scala:68)
> Caused by: java.lang.NullPointerException
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown
> Source)
> at
>
[jira] [Created] (SPARK-37435) Did not find value which can be converted into java.lang.String
Ziqun Ye created SPARK-37435: Summary: Did not find value which can be converted into java.lang.String Key: SPARK-37435 URL: https://issues.apache.org/jira/browse/SPARK-37435 Project: Spark Issue Type: Bug Components: PySpark Affects Versions: 3.0.2, 2.4.4 Reporter: Ziqun Ye Got this following error when loading the saved model. ``` ERROR:ADS Exception Traceback (most recent call last): File "/home/datascience/conda/pyspark30_p37_cpu_v2/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3441, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "/tmp/ipykernel_12307/1140552986.py", line 15, in LogisticRegressionModel.load(spark, "./lrmodelv2") File "/home/datascience/conda/pyspark30_p37_cpu_v2/lib/python3.7/site-packages/pyspark/mllib/classification.py", line 249, in load sc._jsc.sc(), path) File "/home/datascience/conda/pyspark30_p37_cpu_v2/lib/python3.7/site-packages/py4j/java_gateway.py", line 1305, in __call__ answer, self.gateway_client, self.target_id, self.name) File "/home/datascience/conda/pyspark30_p37_cpu_v2/lib/python3.7/site-packages/pyspark/sql/utils.py", line 128, in deco return f(*a, **kw) File "/home/datascience/conda/pyspark30_p37_cpu_v2/lib/python3.7/site-packages/py4j/protocol.py", line 328, in get_return_value format(target_id, ".", name), value) py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.mllib.classification.LogisticRegressionModel.load. : org.json4s.package$MappingException: Did not find value which can be converted into java.lang.String at org.json4s.reflect.package$.fail(package.scala:95) at org.json4s.Extraction$.$anonfun$convert$2(Extraction.scala:756) at scala.Option.getOrElse(Option.scala:189) at org.json4s.Extraction$.convert(Extraction.scala:756) at org.json4s.Extraction$.$anonfun$extract$10(Extraction.scala:404) at org.json4s.Extraction$.$anonfun$customOrElse$1(Extraction.scala:658) at scala.PartialFunction.applyOrElse(PartialFunction.scala:127) at scala.PartialFunction.applyOrElse$(PartialFunction.scala:126) at scala.PartialFunction$$anon$1.applyOrElse(PartialFunction.scala:257) at org.json4s.Extraction$.customOrElse(Extraction.scala:658) at org.json4s.Extraction$.extract(Extraction.scala:402) at org.json4s.Extraction$.extract(Extraction.scala:40) at org.json4s.ExtractableJsonAstNode.extract(ExtractableJsonAstNode.scala:21) at org.apache.spark.mllib.util.Loader$.loadMetadata(modelSaveLoad.scala:122) at org.apache.spark.mllib.classification.LogisticRegressionModel$.load(LogisticRegression.scala:176) at org.apache.spark.mllib.classification.LogisticRegressionModel.load(LogisticRegression.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.lang.Thread.run(Thread.java:748) Py4JJavaError: An error occurred while calling z:org.apache.spark.mllib.classification.LogisticRegressionModel.load. : org.json4s.package$MappingException: Did not find value which can be converted into java.lang.String at org.json4s.reflect.package$.fail(package.scala:95) at org.json4s.Extraction$.$anonfun$convert$2(Extraction.scala:756) at scala.Option.getOrElse(Option.scala:189) at org.json4s.Extraction$.convert(Extraction.scala:756) at org.json4s.Extraction$.$anonfun$extract$10(Extraction.scala:404) at org.json4s.Extraction$.$anonfun$customOrElse$1(Extraction.scala:658) at scala.PartialFunction.applyOrElse(PartialFunction.scala:127) at scala.PartialFunction.applyOrElse$(PartialFunction.scala:126) at scala.PartialFunction$$anon$1.applyOrElse(PartialFunction.scala:257) at org.json4s.Extraction$.customOrElse(Extraction.scala:658) at org.json4s.Extraction$.extract(Extraction.scala:402) at org.json4s.Extraction$.extract(Extraction.scala:40) at org.json4s.ExtractableJsonAstNode.extract(ExtractableJsonAstNode.scala:21) at org.apache.spark.mllib.util.Loader$.loadMetadata(modelSaveLoad.scala:122) at org.apache.spark.mllib.classification.LogisticRegressionModel$.load(LogisticRegression.scala:176) at org.apache.spark.mllib.classification.LogisticRegressionModel.load(LogisticRegression.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at
[jira] [Assigned] (SPARK-37434) Add a new profile to auto disable unsupported UTs on Macos using Apple Silicon
[
https://issues.apache.org/jira/browse/SPARK-37434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-37434:
Assignee: (was: Apache Spark)
> Add a new profile to auto disable unsupported UTs on Macos using Apple Silicon
> --
>
> Key: SPARK-37434
> URL: https://issues.apache.org/jira/browse/SPARK-37434
> Project: Spark
> Issue Type: Improvement
> Components: Build
>Affects Versions: 3.3.0
>Reporter: Yang Jie
>Priority: Major
>
> After SPARK-37272 and SPARK-37282, we can manually add
> {code:java}
> -Dtest.exclude.tags=org.apache.spark.tags.ExtendedLevelDBTest,org.apache.spark.tags.ExtendedRocksDBTest
> {code}
> when run mvn test or sbt test to disable unsupported UTs on Macos using Apple
> Silicon.
>
> We can add a profile to and activate this property automatically when run
> UTs on Macos using Apple Silicon.
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-37434) Add a new profile to auto disable unsupported UTs on Macos using Apple Silicon
[
https://issues.apache.org/jira/browse/SPARK-37434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-37434:
Assignee: Apache Spark
> Add a new profile to auto disable unsupported UTs on Macos using Apple Silicon
> --
>
> Key: SPARK-37434
> URL: https://issues.apache.org/jira/browse/SPARK-37434
> Project: Spark
> Issue Type: Improvement
> Components: Build
>Affects Versions: 3.3.0
>Reporter: Yang Jie
>Assignee: Apache Spark
>Priority: Major
>
> After SPARK-37272 and SPARK-37282, we can manually add
> {code:java}
> -Dtest.exclude.tags=org.apache.spark.tags.ExtendedLevelDBTest,org.apache.spark.tags.ExtendedRocksDBTest
> {code}
> when run mvn test or sbt test to disable unsupported UTs on Macos using Apple
> Silicon.
>
> We can add a profile to and activate this property automatically when run
> UTs on Macos using Apple Silicon.
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-37434) Add a new profile to auto disable unsupported UTs on Macos using Apple Silicon
[
https://issues.apache.org/jira/browse/SPARK-37434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17447185#comment-17447185
]
Apache Spark commented on SPARK-37434:
--
User 'LuciferYang' has created a pull request for this issue:
https://github.com/apache/spark/pull/34676
> Add a new profile to auto disable unsupported UTs on Macos using Apple Silicon
> --
>
> Key: SPARK-37434
> URL: https://issues.apache.org/jira/browse/SPARK-37434
> Project: Spark
> Issue Type: Improvement
> Components: Build
>Affects Versions: 3.3.0
>Reporter: Yang Jie
>Priority: Major
>
> After SPARK-37272 and SPARK-37282, we can manually add
> {code:java}
> -Dtest.exclude.tags=org.apache.spark.tags.ExtendedLevelDBTest,org.apache.spark.tags.ExtendedRocksDBTest
> {code}
> when run mvn test or sbt test to disable unsupported UTs on Macos using Apple
> Silicon.
>
> We can add a profile to and activate this property automatically when run
> UTs on Macos using Apple Silicon.
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-37434) Add a new profile to auto disable unsupported UTs on Macos using Apple Silicon
Yang Jie created SPARK-37434:
Summary: Add a new profile to auto disable unsupported UTs on
Macos using Apple Silicon
Key: SPARK-37434
URL: https://issues.apache.org/jira/browse/SPARK-37434
Project: Spark
Issue Type: Improvement
Components: Build
Affects Versions: 3.3.0
Reporter: Yang Jie
After SPARK-37272 and SPARK-37282, we can manually add
{code:java}
-Dtest.exclude.tags=org.apache.spark.tags.ExtendedLevelDBTest,org.apache.spark.tags.ExtendedRocksDBTest
{code}
when run mvn test or sbt test to disable unsupported UTs on Macos using Apple
Silicon.
We can add a profile to and activate this property automatically when run UTs
on Macos using Apple Silicon.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-37354) Make the Java version installed on the container image used by the K8s integration tests with SBT configurable
[ https://issues.apache.org/jira/browse/SPARK-37354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta resolved SPARK-37354. Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved in https://github.com/apache/spark/pull/34628 > Make the Java version installed on the container image used by the K8s > integration tests with SBT configurable > -- > > Key: SPARK-37354 > URL: https://issues.apache.org/jira/browse/SPARK-37354 > Project: Spark > Issue Type: Bug > Components: Kubernetes, Tests >Affects Versions: 3.2.0 >Reporter: Kousuke Saruta >Assignee: Kousuke Saruta >Priority: Major > Fix For: 3.3.0 > > > I noticed that the default Java version installed on the container image used > by the K8s integration tests are different depending on the way to run the > tests. > If the tests are launched by Maven, the Java version is 8 is installed. > On the other hand, if the tests are launched by SBT, the Java version is 11. > Further, we have no way to change the version. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-36231) Support arithmetic operations of Series containing Decimal(np.nan)
[
https://issues.apache.org/jira/browse/SPARK-36231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-36231:
-
Fix Version/s: (was: 3.3.0)
> Support arithmetic operations of Series containing Decimal(np.nan)
> ---
>
> Key: SPARK-36231
> URL: https://issues.apache.org/jira/browse/SPARK-36231
> Project: Spark
> Issue Type: Sub-task
> Components: PySpark
>Affects Versions: 3.2.0
>Reporter: Xinrong Meng
>Assignee: Yikun Jiang
>Priority: Major
>
> Arithmetic operations of Series containing Decimal(np.nan) raise
> java.lang.NullPointerException in driver. An example is shown as below:
> {code:java}
> >>> pser = pd.Series([decimal.Decimal(1.0), decimal.Decimal(2.0),
> >>> decimal.Decimal(np.nan)])
> >>> psser = ps.from_pandas(pser)
> >>> pser + 1
> 0 2
> 1 3
> 2 NaN
> >>> psser + 1
> Driver stacktrace:
> at
> org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2259)
> at
> org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2208)
> at
> org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2207)
> at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
> at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
> at
> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2207)
> at
> org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1084)
> at
> org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1084)
> at scala.Option.foreach(Option.scala:407)
> at
> org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1084)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2446)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2388)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2377)
> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
> at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:873)
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:2208)
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:2303)
> at
> org.apache.spark.sql.Dataset.$anonfun$collectAsArrowToPython$5(Dataset.scala:3648)
> at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
> at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1437)
> at
> org.apache.spark.sql.Dataset.$anonfun$collectAsArrowToPython$2(Dataset.scala:3652)
> at
> org.apache.spark.sql.Dataset.$anonfun$collectAsArrowToPython$2$adapted(Dataset.scala:3629)
> at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3706)
> at
> org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)
> at
> org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
> at
> org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)
> at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:774)
> at
> org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
> at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3704)
> at
> org.apache.spark.sql.Dataset.$anonfun$collectAsArrowToPython$1(Dataset.scala:3629)
> at
> org.apache.spark.sql.Dataset.$anonfun$collectAsArrowToPython$1$adapted(Dataset.scala:3628)
> at
> org.apache.spark.security.SocketAuthServer$.$anonfun$serveToStream$2(SocketAuthServer.scala:139)
> at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
> at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1437)
> at
> org.apache.spark.security.SocketAuthServer$.$anonfun$serveToStream$1(SocketAuthServer.scala:141)
> at
> org.apache.spark.security.SocketAuthServer$.$anonfun$serveToStream$1$adapted(SocketAuthServer.scala:136)
> at
> org.apache.spark.security.SocketFuncServer.handleConnection(SocketAuthServer.scala:113)
> at
> org.apache.spark.security.SocketFuncServer.handleConnection(SocketAuthServer.scala:107)
> at
> org.apache.spark.security.SocketAuthServer$$anon$1.$anonfun$run$4(SocketAuthServer.scala:68)
> at scala.util.Try$.apply(Try.scala:213)
> at
> org.apache.spark.security.SocketAuthServer$$anon$1.run(SocketAuthServer.scala:68)
> Caused by: java.lang.NullPointerException
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown
> Source)
> at
>
[jira] [Reopened] (SPARK-36231) Support arithmetic operations of Series containing Decimal(np.nan)
[
https://issues.apache.org/jira/browse/SPARK-36231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reopened SPARK-36231:
--
Reverted at
https://github.com/apache/spark/commit/406455d79f787486f9e6fab1dce0d9a2645b8d14
> Support arithmetic operations of Series containing Decimal(np.nan)
> ---
>
> Key: SPARK-36231
> URL: https://issues.apache.org/jira/browse/SPARK-36231
> Project: Spark
> Issue Type: Sub-task
> Components: PySpark
>Affects Versions: 3.2.0
>Reporter: Xinrong Meng
>Assignee: Yikun Jiang
>Priority: Major
> Fix For: 3.3.0
>
>
> Arithmetic operations of Series containing Decimal(np.nan) raise
> java.lang.NullPointerException in driver. An example is shown as below:
> {code:java}
> >>> pser = pd.Series([decimal.Decimal(1.0), decimal.Decimal(2.0),
> >>> decimal.Decimal(np.nan)])
> >>> psser = ps.from_pandas(pser)
> >>> pser + 1
> 0 2
> 1 3
> 2 NaN
> >>> psser + 1
> Driver stacktrace:
> at
> org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2259)
> at
> org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2208)
> at
> org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2207)
> at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
> at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
> at
> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2207)
> at
> org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1084)
> at
> org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1084)
> at scala.Option.foreach(Option.scala:407)
> at
> org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1084)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2446)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2388)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2377)
> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
> at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:873)
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:2208)
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:2303)
> at
> org.apache.spark.sql.Dataset.$anonfun$collectAsArrowToPython$5(Dataset.scala:3648)
> at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
> at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1437)
> at
> org.apache.spark.sql.Dataset.$anonfun$collectAsArrowToPython$2(Dataset.scala:3652)
> at
> org.apache.spark.sql.Dataset.$anonfun$collectAsArrowToPython$2$adapted(Dataset.scala:3629)
> at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3706)
> at
> org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)
> at
> org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
> at
> org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)
> at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:774)
> at
> org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
> at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3704)
> at
> org.apache.spark.sql.Dataset.$anonfun$collectAsArrowToPython$1(Dataset.scala:3629)
> at
> org.apache.spark.sql.Dataset.$anonfun$collectAsArrowToPython$1$adapted(Dataset.scala:3628)
> at
> org.apache.spark.security.SocketAuthServer$.$anonfun$serveToStream$2(SocketAuthServer.scala:139)
> at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
> at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1437)
> at
> org.apache.spark.security.SocketAuthServer$.$anonfun$serveToStream$1(SocketAuthServer.scala:141)
> at
> org.apache.spark.security.SocketAuthServer$.$anonfun$serveToStream$1$adapted(SocketAuthServer.scala:136)
> at
> org.apache.spark.security.SocketFuncServer.handleConnection(SocketAuthServer.scala:113)
> at
> org.apache.spark.security.SocketFuncServer.handleConnection(SocketAuthServer.scala:107)
> at
> org.apache.spark.security.SocketAuthServer$$anon$1.$anonfun$run$4(SocketAuthServer.scala:68)
> at scala.util.Try$.apply(Try.scala:213)
> at
> org.apache.spark.security.SocketAuthServer$$anon$1.run(SocketAuthServer.scala:68)
> Caused by: java.lang.NullPointerException
> at
>
[jira] [Resolved] (SPARK-37209) YarnShuffleIntegrationSuite and other two similar cases in `resource-managers` test failed
[
https://issues.apache.org/jira/browse/SPARK-37209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean R. Owen resolved SPARK-37209.
--
Fix Version/s: 3.3.0
3.0.4
3.2.1
3.1.3
Resolution: Fixed
Issue resolved by pull request 34620
[https://github.com/apache/spark/pull/34620]
> YarnShuffleIntegrationSuite and other two similar cases in
> `resource-managers` test failed
> ---
>
> Key: SPARK-37209
> URL: https://issues.apache.org/jira/browse/SPARK-37209
> Project: Spark
> Issue Type: Bug
> Components: Tests, YARN
>Affects Versions: 3.3.0
>Reporter: Yang Jie
>Assignee: Yang Jie
>Priority: Minor
> Fix For: 3.3.0, 3.0.4, 3.2.1, 3.1.3
>
> Attachments: failed-unit-tests.log, success-unit-tests.log
>
>
> Execute :
> # build/mvn clean package -DskipTests -Phadoop-3.2 -Phive-2.3 -Phadoop-cloud
> -Pmesos -Pyarn -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl
> -Pkubernetes -Phive
> # build/mvn test -Phadoop-3.2 -Phive-2.3 -Phadoop-cloud -Pmesos -Pyarn
> -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive
> -Pscala-2.13 -pl resource-managers/yarn
> The test will successful.
>
> Execute :
> # build/mvn clean -Phadoop-3.2 -Phive-2.3 -Phadoop-cloud -Pmesos -Pyarn
> -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive
> # build/mvn clean test -Phadoop-3.2 -Phive-2.3 -Phadoop-cloud -Pmesos -Pyarn
> -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive
> -Pscala-2.13 -pl resource-managers/yarn
> The test will failed.
>
> Execute :
> # build/mvn clean package -DskipTests -Phadoop-3.2 -Phive-2.3 -Phadoop-cloud
> -Pmesos -Pyarn -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl
> -Pkubernetes -Phive
> # Delete assembly/target/scala-2.12/jars manually
> # build/mvn test -Phadoop-3.2 -Phive-2.3 -Phadoop-cloud -Pmesos -Pyarn
> -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive
> -Pscala-2.13 -pl resource-managers/yarn
> The test will failed.
>
> The error stack is :
> {code:java}
> 21/11/04 19:48:52.159 main ERROR Client: Application diagnostics message:
> User class threw exception: org.apache.spark.SparkException: Job aborted due
> to stage failure: Task 0 in stage 0.0 failed 4 times,
> most recent failure: Lost task 0.3 in stage 0.0 (TID 6) (localhost executor
> 1): java.lang.NoClassDefFoundError: breeze/linalg/Matrix
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at org.apache.spark.util.Utils$.classForName(Utils.scala:216)
> at
> org.apache.spark.serializer.KryoSerializer$.$anonfun$loadableSparkClasses$1(KryoSerializer.scala:537)
> at scala.collection.immutable.List.flatMap(List.scala:293)
> at scala.collection.immutable.List.flatMap(List.scala:79)
> at
> org.apache.spark.serializer.KryoSerializer$.loadableSparkClasses$lzycompute(KryoSerializer.scala:535)
> at
> org.apache.spark.serializer.KryoSerializer$.org$apache$spark$serializer$KryoSerializer$$loadableSparkClasses(KryoSerializer.scala:502)
> at
> org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:226)
> at
> org.apache.spark.serializer.KryoSerializer$$anon$1.create(KryoSerializer.scala:102)
> at
> com.esotericsoftware.kryo.pool.KryoPoolQueueImpl.borrow(KryoPoolQueueImpl.java:48)
> at
> org.apache.spark.serializer.KryoSerializer$PoolWrapper.borrow(KryoSerializer.scala:109)
> at
> org.apache.spark.serializer.KryoSerializerInstance.borrowKryo(KryoSerializer.scala:346)
> at
> org.apache.spark.serializer.KryoSerializationStream.(KryoSerializer.scala:266)
> at
> org.apache.spark.serializer.KryoSerializerInstance.serializeStream(KryoSerializer.scala:432)
> at
> org.apache.spark.shuffle.ShufflePartitionPairsWriter.open(ShufflePartitionPairsWriter.scala:76)
> at
> org.apache.spark.shuffle.ShufflePartitionPairsWriter.write(ShufflePartitionPairsWriter.scala:59)
> at
> org.apache.spark.util.collection.WritablePartitionedIterator.writeNext(WritablePartitionedPairCollection.scala:83)
> at
> org.apache.spark.util.collection.ExternalSorter.$anonfun$writePartitionedMapOutput$1(ExternalSorter.scala:772)
> at
> scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
> at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1468)
> at
> org.apache.spark.util.collection.ExternalSorter.writePartitionedMapOutput(ExternalSorter.scala:775)
> at
> org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:70)
> at
>
[jira] [Assigned] (SPARK-37209) YarnShuffleIntegrationSuite and other two similar cases in `resource-managers` test failed
[
https://issues.apache.org/jira/browse/SPARK-37209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean R. Owen reassigned SPARK-37209:
Assignee: Yang Jie
> YarnShuffleIntegrationSuite and other two similar cases in
> `resource-managers` test failed
> ---
>
> Key: SPARK-37209
> URL: https://issues.apache.org/jira/browse/SPARK-37209
> Project: Spark
> Issue Type: Bug
> Components: Tests, YARN
>Affects Versions: 3.3.0
>Reporter: Yang Jie
>Assignee: Yang Jie
>Priority: Minor
> Attachments: failed-unit-tests.log, success-unit-tests.log
>
>
> Execute :
> # build/mvn clean package -DskipTests -Phadoop-3.2 -Phive-2.3 -Phadoop-cloud
> -Pmesos -Pyarn -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl
> -Pkubernetes -Phive
> # build/mvn test -Phadoop-3.2 -Phive-2.3 -Phadoop-cloud -Pmesos -Pyarn
> -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive
> -Pscala-2.13 -pl resource-managers/yarn
> The test will successful.
>
> Execute :
> # build/mvn clean -Phadoop-3.2 -Phive-2.3 -Phadoop-cloud -Pmesos -Pyarn
> -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive
> # build/mvn clean test -Phadoop-3.2 -Phive-2.3 -Phadoop-cloud -Pmesos -Pyarn
> -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive
> -Pscala-2.13 -pl resource-managers/yarn
> The test will failed.
>
> Execute :
> # build/mvn clean package -DskipTests -Phadoop-3.2 -Phive-2.3 -Phadoop-cloud
> -Pmesos -Pyarn -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl
> -Pkubernetes -Phive
> # Delete assembly/target/scala-2.12/jars manually
> # build/mvn test -Phadoop-3.2 -Phive-2.3 -Phadoop-cloud -Pmesos -Pyarn
> -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive
> -Pscala-2.13 -pl resource-managers/yarn
> The test will failed.
>
> The error stack is :
> {code:java}
> 21/11/04 19:48:52.159 main ERROR Client: Application diagnostics message:
> User class threw exception: org.apache.spark.SparkException: Job aborted due
> to stage failure: Task 0 in stage 0.0 failed 4 times,
> most recent failure: Lost task 0.3 in stage 0.0 (TID 6) (localhost executor
> 1): java.lang.NoClassDefFoundError: breeze/linalg/Matrix
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at org.apache.spark.util.Utils$.classForName(Utils.scala:216)
> at
> org.apache.spark.serializer.KryoSerializer$.$anonfun$loadableSparkClasses$1(KryoSerializer.scala:537)
> at scala.collection.immutable.List.flatMap(List.scala:293)
> at scala.collection.immutable.List.flatMap(List.scala:79)
> at
> org.apache.spark.serializer.KryoSerializer$.loadableSparkClasses$lzycompute(KryoSerializer.scala:535)
> at
> org.apache.spark.serializer.KryoSerializer$.org$apache$spark$serializer$KryoSerializer$$loadableSparkClasses(KryoSerializer.scala:502)
> at
> org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:226)
> at
> org.apache.spark.serializer.KryoSerializer$$anon$1.create(KryoSerializer.scala:102)
> at
> com.esotericsoftware.kryo.pool.KryoPoolQueueImpl.borrow(KryoPoolQueueImpl.java:48)
> at
> org.apache.spark.serializer.KryoSerializer$PoolWrapper.borrow(KryoSerializer.scala:109)
> at
> org.apache.spark.serializer.KryoSerializerInstance.borrowKryo(KryoSerializer.scala:346)
> at
> org.apache.spark.serializer.KryoSerializationStream.(KryoSerializer.scala:266)
> at
> org.apache.spark.serializer.KryoSerializerInstance.serializeStream(KryoSerializer.scala:432)
> at
> org.apache.spark.shuffle.ShufflePartitionPairsWriter.open(ShufflePartitionPairsWriter.scala:76)
> at
> org.apache.spark.shuffle.ShufflePartitionPairsWriter.write(ShufflePartitionPairsWriter.scala:59)
> at
> org.apache.spark.util.collection.WritablePartitionedIterator.writeNext(WritablePartitionedPairCollection.scala:83)
> at
> org.apache.spark.util.collection.ExternalSorter.$anonfun$writePartitionedMapOutput$1(ExternalSorter.scala:772)
> at
> scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
> at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1468)
> at
> org.apache.spark.util.collection.ExternalSorter.writePartitionedMapOutput(ExternalSorter.scala:775)
> at
> org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:70)
> at
> org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
> at
>
[jira] [Commented] (SPARK-37391) SIGNIFICANT bottleneck introduced by fix for SPARK-34497
[ https://issues.apache.org/jira/browse/SPARK-37391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17447162#comment-17447162 ] Hyukjin Kwon commented on SPARK-37391: -- [~danny-seismic], it would be great to assess this issue futher with problem description and preferably self-contained reproducer > SIGNIFICANT bottleneck introduced by fix for SPARK-34497 > > > Key: SPARK-37391 > URL: https://issues.apache.org/jira/browse/SPARK-37391 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.1.0, 3.1.1, 3.1.2, 3.2.0 > Environment: N/A >Reporter: Danny Guinther >Priority: Major > > The fix for SPARK-34497 ( [https://github.com/apache/spark/pull/31622] ) does > not seem to have consider the reality that some apps may rely on being able > to establish many JDBC connections simultaneously for performance reasons. > The fix forces concurrency to 1 when establishing database connections and > that strikes me as a *significant* user impacting change and a *significant* > bottleneck. > Can anyone propose a workaround for this? I have an app that makes > connections to thousands of databases and I can't upgrade to any version > >3.1.x because of this significant bottleneck. > > Thanks in advance for your help! -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-37433) TimeZoneAwareExpression throws NoSuchElementException: None.get on expr.eval()
[
https://issues.apache.org/jira/browse/SPARK-37433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-37433:
Assignee: (was: Apache Spark)
> TimeZoneAwareExpression throws NoSuchElementException: None.get on expr.eval()
> --
>
> Key: SPARK-37433
> URL: https://issues.apache.org/jira/browse/SPARK-37433
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.2.0
>Reporter: Sathiya Kumar
>Priority: Minor
>
> TimeZoneAwareExpression like hour, date_format etc. throws
> NoSuchElementException: None.get on expr.eval()
> *hour(current_timestamp).expr.eval()*
> *date_format(current_timestamp, "dd").expr.eval()*
>
> {code:java}
> java.util.NoSuchElementException: None.get
> at scala.None$.get(Option.scala:529)
> at scala.None$.get(Option.scala:527)
> at
> org.apache.spark.sql.catalyst.expressions.TimeZoneAwareExpression.zoneId(datetimeExpressions.scala:53)
> at
> org.apache.spark.sql.catalyst.expressions.TimeZoneAwareExpression.zoneId$(datetimeExpressions.scala:53)
> at
> org.apache.spark.sql.catalyst.expressions.DateFormatClass.zoneId$lzycompute(datetimeExpressions.scala:772)
> at
> org.apache.spark.sql.catalyst.expressions.DateFormatClass.zoneId(datetimeExpressions.scala:772)
> at
> org.apache.spark.sql.catalyst.expressions.TimestampFormatterHelper.getFormatter(datetimeExpressions.scala:70)
> at
> org.apache.spark.sql.catalyst.expressions.TimestampFormatterHelper.getFormatter$(datetimeExpressions.scala:67)
> at
> org.apache.spark.sql.catalyst.expressions.DateFormatClass.getFormatter(datetimeExpressions.scala:772)
> at
> org.apache.spark.sql.catalyst.expressions.TimestampFormatterHelper.$anonfun$formatterOption$1(datetimeExpressions.scala:64)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-37433) TimeZoneAwareExpression throws NoSuchElementException: None.get on expr.eval()
[
https://issues.apache.org/jira/browse/SPARK-37433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17447159#comment-17447159
]
Apache Spark commented on SPARK-37433:
--
User 'sathiyapk' has created a pull request for this issue:
https://github.com/apache/spark/pull/34675
> TimeZoneAwareExpression throws NoSuchElementException: None.get on expr.eval()
> --
>
> Key: SPARK-37433
> URL: https://issues.apache.org/jira/browse/SPARK-37433
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.2.0
>Reporter: Sathiya Kumar
>Priority: Minor
>
> TimeZoneAwareExpression like hour, date_format etc. throws
> NoSuchElementException: None.get on expr.eval()
> *hour(current_timestamp).expr.eval()*
> *date_format(current_timestamp, "dd").expr.eval()*
>
> {code:java}
> java.util.NoSuchElementException: None.get
> at scala.None$.get(Option.scala:529)
> at scala.None$.get(Option.scala:527)
> at
> org.apache.spark.sql.catalyst.expressions.TimeZoneAwareExpression.zoneId(datetimeExpressions.scala:53)
> at
> org.apache.spark.sql.catalyst.expressions.TimeZoneAwareExpression.zoneId$(datetimeExpressions.scala:53)
> at
> org.apache.spark.sql.catalyst.expressions.DateFormatClass.zoneId$lzycompute(datetimeExpressions.scala:772)
> at
> org.apache.spark.sql.catalyst.expressions.DateFormatClass.zoneId(datetimeExpressions.scala:772)
> at
> org.apache.spark.sql.catalyst.expressions.TimestampFormatterHelper.getFormatter(datetimeExpressions.scala:70)
> at
> org.apache.spark.sql.catalyst.expressions.TimestampFormatterHelper.getFormatter$(datetimeExpressions.scala:67)
> at
> org.apache.spark.sql.catalyst.expressions.DateFormatClass.getFormatter(datetimeExpressions.scala:772)
> at
> org.apache.spark.sql.catalyst.expressions.TimestampFormatterHelper.$anonfun$formatterOption$1(datetimeExpressions.scala:64)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-37433) TimeZoneAwareExpression throws NoSuchElementException: None.get on expr.eval()
[
https://issues.apache.org/jira/browse/SPARK-37433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-37433:
Assignee: Apache Spark
> TimeZoneAwareExpression throws NoSuchElementException: None.get on expr.eval()
> --
>
> Key: SPARK-37433
> URL: https://issues.apache.org/jira/browse/SPARK-37433
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.2.0
>Reporter: Sathiya Kumar
>Assignee: Apache Spark
>Priority: Minor
>
> TimeZoneAwareExpression like hour, date_format etc. throws
> NoSuchElementException: None.get on expr.eval()
> *hour(current_timestamp).expr.eval()*
> *date_format(current_timestamp, "dd").expr.eval()*
>
> {code:java}
> java.util.NoSuchElementException: None.get
> at scala.None$.get(Option.scala:529)
> at scala.None$.get(Option.scala:527)
> at
> org.apache.spark.sql.catalyst.expressions.TimeZoneAwareExpression.zoneId(datetimeExpressions.scala:53)
> at
> org.apache.spark.sql.catalyst.expressions.TimeZoneAwareExpression.zoneId$(datetimeExpressions.scala:53)
> at
> org.apache.spark.sql.catalyst.expressions.DateFormatClass.zoneId$lzycompute(datetimeExpressions.scala:772)
> at
> org.apache.spark.sql.catalyst.expressions.DateFormatClass.zoneId(datetimeExpressions.scala:772)
> at
> org.apache.spark.sql.catalyst.expressions.TimestampFormatterHelper.getFormatter(datetimeExpressions.scala:70)
> at
> org.apache.spark.sql.catalyst.expressions.TimestampFormatterHelper.getFormatter$(datetimeExpressions.scala:67)
> at
> org.apache.spark.sql.catalyst.expressions.DateFormatClass.getFormatter(datetimeExpressions.scala:772)
> at
> org.apache.spark.sql.catalyst.expressions.TimestampFormatterHelper.$anonfun$formatterOption$1(datetimeExpressions.scala:64)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-37433) TimeZoneAwareExpression throws NoSuchElementException: None.get on expr.eval()
Sathiya Kumar created SPARK-37433:
-
Summary: TimeZoneAwareExpression throws NoSuchElementException:
None.get on expr.eval()
Key: SPARK-37433
URL: https://issues.apache.org/jira/browse/SPARK-37433
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.2.0
Reporter: Sathiya Kumar
TimeZoneAwareExpression like hour, date_format etc. throws
NoSuchElementException: None.get on expr.eval()
*hour(current_timestamp).expr.eval()*
*date_format(current_timestamp, "dd").expr.eval()*
{code:java}
java.util.NoSuchElementException: None.get
at scala.None$.get(Option.scala:529)
at scala.None$.get(Option.scala:527)
at
org.apache.spark.sql.catalyst.expressions.TimeZoneAwareExpression.zoneId(datetimeExpressions.scala:53)
at
org.apache.spark.sql.catalyst.expressions.TimeZoneAwareExpression.zoneId$(datetimeExpressions.scala:53)
at
org.apache.spark.sql.catalyst.expressions.DateFormatClass.zoneId$lzycompute(datetimeExpressions.scala:772)
at
org.apache.spark.sql.catalyst.expressions.DateFormatClass.zoneId(datetimeExpressions.scala:772)
at
org.apache.spark.sql.catalyst.expressions.TimestampFormatterHelper.getFormatter(datetimeExpressions.scala:70)
at
org.apache.spark.sql.catalyst.expressions.TimestampFormatterHelper.getFormatter$(datetimeExpressions.scala:67)
at
org.apache.spark.sql.catalyst.expressions.DateFormatClass.getFormatter(datetimeExpressions.scala:772)
at
org.apache.spark.sql.catalyst.expressions.TimestampFormatterHelper.$anonfun$formatterOption$1(datetimeExpressions.scala:64)
{code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-37432) Driver keep a record of decommission executor
[
https://issues.apache.org/jira/browse/SPARK-37432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hoa Le updated SPARK-37432:
---
Attachment: master_ui_executor_tab.png
> Driver keep a record of decommission executor
> -
>
> Key: SPARK-37432
> URL: https://issues.apache.org/jira/browse/SPARK-37432
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
>Affects Versions: 3.1.1
>Reporter: Hoa Le
>Priority: Minor
> Attachments: master_ui_executor_tab.png
>
>
> Hi,
> We are running spark on k8s with version 3.1.1. After the spark application
> running for a while, we are getting the exception below:
> On driver:
>
> {code:java}
> 2021-11-21 18:25:21,859 ERROR Failed to send RPC RPC 6827167497981418905 to
> /10.1.201.113:58354: java.nio.channels.ClosedChannelException
> (org.apache.spark.network.client.TransportClient) [rpc-server-4-1]
> java.nio.channels.ClosedChannelException
> at
> io.netty.channel.AbstractChannel$AbstractUnsafe.newClosedChannelException(AbstractChannel.java:957)
> at
> io.netty.channel.AbstractChannel$AbstractUnsafe.write(AbstractChannel.java:865)
> at
> io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1367)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:764)
> at
> io.netty.channel.AbstractChannelHandlerContext$WriteTask.run(AbstractChannelHandlerContext.java:1071)
> at
> io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
> at
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Unknown Source)
> 2021-11-21 18:25:21,864 ERROR Failed to send RPC RPC 7618635518207296341 to
> /10.1.201.113:58354: java.nio.channels.ClosedChannelException
> (org.apache.spark.network.client.TransportClient) [rpc-server-4-1]
> java.nio.channels.ClosedChannelException
> at
> io.netty.channel.AbstractChannel$AbstractUnsafe.newClosedChannelException(AbstractChannel.java:957)
> at
> io.netty.channel.AbstractChannel$AbstractUnsafe.write(AbstractChannel.java:865)
> at
> io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1367)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:764)
> at
> io.netty.channel.AbstractChannelHandlerContext$WriteTask.run(AbstractChannelHandlerContext.java:1071)
> at
> io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
> at
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Unknown Source)
> 2021-11-21 18:25:21,868 ERROR Failed to send RPC RPC 5040314884474308699 to
> /10.1.201.113:58354: java.nio.channels.ClosedChannelException
> (org.apache.spark.network.client.TransportClient) [rpc-server-4-1]
> java.nio.channels.ClosedChannelException
> at
> io.netty.channel.AbstractChannel$AbstractUnsafe.newClosedChannelException(AbstractChannel.java:957)
> at
> io.netty.channel.AbstractChannel$AbstractUnsafe.write(AbstractChannel.java:865)
> at
> io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1367)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:764)
> at
> io.netty.channel.AbstractChannelHandlerContext$WriteTask.run(AbstractChannelHandlerContext.java:1071)
> at
> io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
[jira] [Created] (SPARK-37432) Driver keep a record of decommission executor
Hoa Le created SPARK-37432:
--
Summary: Driver keep a record of decommission executor
Key: SPARK-37432
URL: https://issues.apache.org/jira/browse/SPARK-37432
Project: Spark
Issue Type: Bug
Components: Kubernetes
Affects Versions: 3.1.1
Reporter: Hoa Le
Hi,
We are running spark on k8s with version 3.1.1. After the spark application
running for a while, we are getting the exception below:
On driver:
{code:java}
2021-11-21 18:25:21,859 ERROR Failed to send RPC RPC 6827167497981418905 to
/10.1.201.113:58354: java.nio.channels.ClosedChannelException
(org.apache.spark.network.client.TransportClient) [rpc-server-4-1]
java.nio.channels.ClosedChannelException
at
io.netty.channel.AbstractChannel$AbstractUnsafe.newClosedChannelException(AbstractChannel.java:957)
at
io.netty.channel.AbstractChannel$AbstractUnsafe.write(AbstractChannel.java:865)
at
io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1367)
at
io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717)
at
io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:764)
at
io.netty.channel.AbstractChannelHandlerContext$WriteTask.run(AbstractChannelHandlerContext.java:1071)
at
io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
at
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Unknown Source)
2021-11-21 18:25:21,864 ERROR Failed to send RPC RPC 7618635518207296341 to
/10.1.201.113:58354: java.nio.channels.ClosedChannelException
(org.apache.spark.network.client.TransportClient) [rpc-server-4-1]
java.nio.channels.ClosedChannelException
at
io.netty.channel.AbstractChannel$AbstractUnsafe.newClosedChannelException(AbstractChannel.java:957)
at
io.netty.channel.AbstractChannel$AbstractUnsafe.write(AbstractChannel.java:865)
at
io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1367)
at
io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717)
at
io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:764)
at
io.netty.channel.AbstractChannelHandlerContext$WriteTask.run(AbstractChannelHandlerContext.java:1071)
at
io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
at
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Unknown Source)
2021-11-21 18:25:21,868 ERROR Failed to send RPC RPC 5040314884474308699 to
/10.1.201.113:58354: java.nio.channels.ClosedChannelException
(org.apache.spark.network.client.TransportClient) [rpc-server-4-1]
java.nio.channels.ClosedChannelException
at
io.netty.channel.AbstractChannel$AbstractUnsafe.newClosedChannelException(AbstractChannel.java:957)
at
io.netty.channel.AbstractChannel$AbstractUnsafe.write(AbstractChannel.java:865)
at
io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1367)
at
io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717)
at
io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:764)
at
io.netty.channel.AbstractChannelHandlerContext$WriteTask.run(AbstractChannelHandlerContext.java:1071)
at
io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
at
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
[jira] [Assigned] (SPARK-37104) RDD and DStream should be covariant
[
https://issues.apache.org/jira/browse/SPARK-37104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maciej Szymkiewicz reassigned SPARK-37104:
--
Assignee: Maciej Szymkiewicz
> RDD and DStream should be covariant
> ---
>
> Key: SPARK-37104
> URL: https://issues.apache.org/jira/browse/SPARK-37104
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
>Affects Versions: 3.1.0, 3.2.0, 3.3.0
>Reporter: Maciej Szymkiewicz
>Assignee: Maciej Szymkiewicz
>Priority: Major
>
> At the moment {{RDD}} and {{DStream}} are defined as invariant.
>
> However, there are immutable and could be marked as covariant.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-37104) RDD and DStream should be covariant
[
https://issues.apache.org/jira/browse/SPARK-37104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maciej Szymkiewicz resolved SPARK-37104.
Fix Version/s: 3.3.0
Resolution: Fixed
Issue resolved by pull request 34374
[https://github.com/apache/spark/pull/34374]
> RDD and DStream should be covariant
> ---
>
> Key: SPARK-37104
> URL: https://issues.apache.org/jira/browse/SPARK-37104
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
>Affects Versions: 3.1.0, 3.2.0, 3.3.0
>Reporter: Maciej Szymkiewicz
>Assignee: Maciej Szymkiewicz
>Priority: Major
> Fix For: 3.3.0
>
>
> At the moment {{RDD}} and {{DStream}} are defined as invariant.
>
> However, there are immutable and could be marked as covariant.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-37431) pyspark.{ml, mllib}.linalg.Vector API
Maciej Szymkiewicz created SPARK-37431:
--
Summary: pyspark.{ml, mllib}.linalg.Vector API
Key: SPARK-37431
URL: https://issues.apache.org/jira/browse/SPARK-37431
Project: Spark
Issue Type: Improvement
Components: ML, MLlib, PySpark
Affects Versions: 3.3.0
Reporter: Maciej Szymkiewicz
At the moment both {{Vector}} implementations have minimal API. {{mllib}}
{code:python}
class Vector(object):
__UDT__ =
def toArray(self): ...
def asML(self): ...
{code}
{{ml}}
{code:python}
class Vector(object):
__UDT__ = ...
def toArray(self): ...
{code}
which doesn't cover actual API being used.
This causes typing issues, when we expect any {{Vector}}.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-37430) Inline type hints for python/pyspark/mllib/linalg/distributed.py
Maciej Szymkiewicz created SPARK-37430: -- Summary: Inline type hints for python/pyspark/mllib/linalg/distributed.py Key: SPARK-37430 URL: https://issues.apache.org/jira/browse/SPARK-37430 Project: Spark Issue Type: Sub-task Components: MLlib, PySpark Affects Versions: 3.3.0 Reporter: Maciej Szymkiewicz Inline type hints from python/pyspark/mlib/linalg/distributed.pyi to python/pyspark/mllib/linalg/distributed.py -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-37429) Inline type hints for python/pyspark/mllib/linalg/__init__.py
Maciej Szymkiewicz created SPARK-37429: -- Summary: Inline type hints for python/pyspark/mllib/linalg/__init__.py Key: SPARK-37429 URL: https://issues.apache.org/jira/browse/SPARK-37429 Project: Spark Issue Type: Sub-task Components: MLlib, PySpark Affects Versions: 3.3.0 Reporter: Maciej Szymkiewicz Inline type hints from python/pyspark/mlib/linalg/__init__.pyi to python/pyspark/mllib/linalg/__init__.py -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-37423) Inline type hints for python/pyspark/mllib/fpm.py
[ https://issues.apache.org/jira/browse/SPARK-37423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz updated SPARK-37423: --- Summary: Inline type hints for python/pyspark/mllib/fpm.py (was: Inline type hints for python/pyspark/mllib/fpm.pypy) > Inline type hints for python/pyspark/mllib/fpm.py > - > > Key: SPARK-37423 > URL: https://issues.apache.org/jira/browse/SPARK-37423 > Project: Spark > Issue Type: Sub-task > Components: MLlib, PySpark >Affects Versions: 3.3.0 >Reporter: Maciej Szymkiewicz >Priority: Major > > Inline type hints from python/pyspark/mlib/fpm.pyi to > python/pyspark/mllib/fpm.py -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-37427) Inline type hints for python/pyspark/mllib/tree.py
[ https://issues.apache.org/jira/browse/SPARK-37427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz updated SPARK-37427: --- Summary: Inline type hints for python/pyspark/mllib/tree.py (was: Inline type hints for python/pyspark/mllib/tree.pypy) > Inline type hints for python/pyspark/mllib/tree.py > -- > > Key: SPARK-37427 > URL: https://issues.apache.org/jira/browse/SPARK-37427 > Project: Spark > Issue Type: Sub-task > Components: MLlib, PySpark >Affects Versions: 3.3.0 >Reporter: Maciej Szymkiewicz >Priority: Major > > Inline type hints from python/pyspark/mlib/tree.pyi to > python/pyspark/mllib/tree.py -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-37428) Inline type hints for python/pyspark/mllib/util.py
[ https://issues.apache.org/jira/browse/SPARK-37428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz updated SPARK-37428: --- Summary: Inline type hints for python/pyspark/mllib/util.py (was: Inline type hints for python/pyspark/mllib/util.pypy) > Inline type hints for python/pyspark/mllib/util.py > -- > > Key: SPARK-37428 > URL: https://issues.apache.org/jira/browse/SPARK-37428 > Project: Spark > Issue Type: Sub-task > Components: PySpark >Affects Versions: 3.3.0 >Reporter: Maciej Szymkiewicz >Priority: Major > > Inline type hints from python/pyspark/mlib/util.pyi to > python/pyspark/mllib/util.py -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-37424) Inline type hints for python/pyspark/mllib/random.py
[ https://issues.apache.org/jira/browse/SPARK-37424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz updated SPARK-37424: --- Summary: Inline type hints for python/pyspark/mllib/random.py (was: Inline type hints for python/pyspark/mllib/random.pypy) > Inline type hints for python/pyspark/mllib/random.py > > > Key: SPARK-37424 > URL: https://issues.apache.org/jira/browse/SPARK-37424 > Project: Spark > Issue Type: Sub-task > Components: MLlib, PySpark >Affects Versions: 3.3.0 >Reporter: Maciej Szymkiewicz >Priority: Major > > Inline type hints from python/pyspark/mlib/random.pyi to > python/pyspark/mllib/random.py -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-37426) Inline type hints for python/pyspark/mllib/regression.py
[ https://issues.apache.org/jira/browse/SPARK-37426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz updated SPARK-37426: --- Summary: Inline type hints for python/pyspark/mllib/regression.py (was: Inline type hints for python/pyspark/mllib/regression.pypy) > Inline type hints for python/pyspark/mllib/regression.py > > > Key: SPARK-37426 > URL: https://issues.apache.org/jira/browse/SPARK-37426 > Project: Spark > Issue Type: Sub-task > Components: MLlib, PySpark >Affects Versions: 3.3.0 >Reporter: Maciej Szymkiewicz >Priority: Major > > Inline type hints from python/pyspark/mlib/regression.pyi to > python/pyspark/mllib/regression.py -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-37425) Inline type hints for python/pyspark/mllib/recommendation.py
[ https://issues.apache.org/jira/browse/SPARK-37425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz updated SPARK-37425: --- Summary: Inline type hints for python/pyspark/mllib/recommendation.py (was: Inline type hints for python/pyspark/mllib/recommendation.pypy) > Inline type hints for python/pyspark/mllib/recommendation.py > > > Key: SPARK-37425 > URL: https://issues.apache.org/jira/browse/SPARK-37425 > Project: Spark > Issue Type: Sub-task > Components: MLlib, PySpark >Affects Versions: 3.3.0 >Reporter: Maciej Szymkiewicz >Priority: Major > > Inline type hints from python/pyspark/mlib/recommendation.pyi to > python/pyspark/mllib/recommendation.py -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-37422) Inline type hints for python/pyspark/mllib/feature.py
[ https://issues.apache.org/jira/browse/SPARK-37422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz updated SPARK-37422: --- Summary: Inline type hints for python/pyspark/mllib/feature.py (was: Inline type hints for python/pyspark/mllib/feature.pypy) > Inline type hints for python/pyspark/mllib/feature.py > - > > Key: SPARK-37422 > URL: https://issues.apache.org/jira/browse/SPARK-37422 > Project: Spark > Issue Type: Sub-task > Components: MLlib, PySpark >Affects Versions: 3.3.0 >Reporter: Maciej Szymkiewicz >Priority: Major > > Inline type hints from python/pyspark/mlib/feature.pyi to > python/pyspark/mllib/feature.py -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-37421) Inline type hints for python/pyspark/mllib/evaluation.py
[ https://issues.apache.org/jira/browse/SPARK-37421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz updated SPARK-37421: --- Summary: Inline type hints for python/pyspark/mllib/evaluation.py (was: Inline type hints for python/pyspark/mllib/evaluation.pypy) > Inline type hints for python/pyspark/mllib/evaluation.py > > > Key: SPARK-37421 > URL: https://issues.apache.org/jira/browse/SPARK-37421 > Project: Spark > Issue Type: Sub-task > Components: MLlib, PySpark >Affects Versions: 3.3.0 >Reporter: Maciej Szymkiewicz >Priority: Major > > Inline type hints from python/pyspark/mlib/evaluation.pyi to > python/pyspark/mllib/evaluation.py -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-37236) Inline type hints for KernelDensity.pyi, test.py in python/pyspark/mllib/stat/
[ https://issues.apache.org/jira/browse/SPARK-37236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz updated SPARK-37236: --- Parent: (was: SPARK-37233) Issue Type: Improvement (was: Sub-task) > Inline type hints for KernelDensity.pyi, test.py in python/pyspark/mllib/stat/ > -- > > Key: SPARK-37236 > URL: https://issues.apache.org/jira/browse/SPARK-37236 > Project: Spark > Issue Type: Improvement > Components: PySpark >Affects Versions: 3.3.0 >Reporter: dch nguyen >Assignee: dch nguyen >Priority: Major > Fix For: 3.3.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-37236) Inline type hints for KernelDensity.pyi, test.py in python/pyspark/mllib/stat/
[ https://issues.apache.org/jira/browse/SPARK-37236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz updated SPARK-37236: --- Parent: SPARK-37396 Issue Type: Sub-task (was: Improvement) > Inline type hints for KernelDensity.pyi, test.py in python/pyspark/mllib/stat/ > -- > > Key: SPARK-37236 > URL: https://issues.apache.org/jira/browse/SPARK-37236 > Project: Spark > Issue Type: Sub-task > Components: PySpark >Affects Versions: 3.3.0 >Reporter: dch nguyen >Assignee: dch nguyen >Priority: Major > Fix For: 3.3.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-37235) Inline type hints for distribution.py and __init__.py in python/pyspark/mllib/stat
[ https://issues.apache.org/jira/browse/SPARK-37235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz updated SPARK-37235: --- Parent: SPARK-37396 Issue Type: Sub-task (was: Improvement) > Inline type hints for distribution.py and __init__.py in > python/pyspark/mllib/stat > -- > > Key: SPARK-37235 > URL: https://issues.apache.org/jira/browse/SPARK-37235 > Project: Spark > Issue Type: Sub-task > Components: PySpark >Affects Versions: 3.3.0 >Reporter: dch nguyen >Assignee: dch nguyen >Priority: Major > Fix For: 3.3.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-37235) Inline type hints for distribution.py and __init__.py in python/pyspark/mllib/stat
[ https://issues.apache.org/jira/browse/SPARK-37235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz updated SPARK-37235: --- Parent: (was: SPARK-37396) Issue Type: Improvement (was: Sub-task) > Inline type hints for distribution.py and __init__.py in > python/pyspark/mllib/stat > -- > > Key: SPARK-37235 > URL: https://issues.apache.org/jira/browse/SPARK-37235 > Project: Spark > Issue Type: Improvement > Components: PySpark >Affects Versions: 3.3.0 >Reporter: dch nguyen >Assignee: dch nguyen >Priority: Major > Fix For: 3.3.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-37235) Inline type hints for distribution.py and __init__.py in python/pyspark/mllib/stat
[ https://issues.apache.org/jira/browse/SPARK-37235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz updated SPARK-37235: --- Parent: SPARK-37396 Issue Type: Sub-task (was: Improvement) > Inline type hints for distribution.py and __init__.py in > python/pyspark/mllib/stat > -- > > Key: SPARK-37235 > URL: https://issues.apache.org/jira/browse/SPARK-37235 > Project: Spark > Issue Type: Sub-task > Components: PySpark >Affects Versions: 3.3.0 >Reporter: dch nguyen >Assignee: dch nguyen >Priority: Major > Fix For: 3.3.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-37235) Inline type hints for distribution.py and __init__.py in python/pyspark/mllib/stat
[ https://issues.apache.org/jira/browse/SPARK-37235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz updated SPARK-37235: --- Parent: (was: SPARK-37233) Issue Type: Improvement (was: Sub-task) > Inline type hints for distribution.py and __init__.py in > python/pyspark/mllib/stat > -- > > Key: SPARK-37235 > URL: https://issues.apache.org/jira/browse/SPARK-37235 > Project: Spark > Issue Type: Improvement > Components: PySpark >Affects Versions: 3.3.0 >Reporter: dch nguyen >Assignee: dch nguyen >Priority: Major > Fix For: 3.3.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-37234) Inline type hints for python/pyspark/mllib/stat/_statistics.py
[ https://issues.apache.org/jira/browse/SPARK-37234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz updated SPARK-37234: --- Parent: SPARK-37396 Issue Type: Sub-task (was: Improvement) > Inline type hints for python/pyspark/mllib/stat/_statistics.py > -- > > Key: SPARK-37234 > URL: https://issues.apache.org/jira/browse/SPARK-37234 > Project: Spark > Issue Type: Sub-task > Components: PySpark >Affects Versions: 3.3.0 >Reporter: dch nguyen >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-37234) Inline type hints for python/pyspark/mllib/stat/_statistics.py
[ https://issues.apache.org/jira/browse/SPARK-37234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz updated SPARK-37234: --- Parent: (was: SPARK-37233) Issue Type: Improvement (was: Sub-task) > Inline type hints for python/pyspark/mllib/stat/_statistics.py > -- > > Key: SPARK-37234 > URL: https://issues.apache.org/jira/browse/SPARK-37234 > Project: Spark > Issue Type: Improvement > Components: PySpark >Affects Versions: 3.3.0 >Reporter: dch nguyen >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-37428) Inline type hints for python/pyspark/mllib/util.pypy
Maciej Szymkiewicz created SPARK-37428: -- Summary: Inline type hints for python/pyspark/mllib/util.pypy Key: SPARK-37428 URL: https://issues.apache.org/jira/browse/SPARK-37428 Project: Spark Issue Type: Sub-task Components: PySpark Affects Versions: 3.3.0 Reporter: Maciej Szymkiewicz Inline type hints from python/pyspark/mlib/util.pyi to python/pyspark/mllib/util.py -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-37427) Inline type hints for python/pyspark/mllib/tree.pypy
Maciej Szymkiewicz created SPARK-37427: -- Summary: Inline type hints for python/pyspark/mllib/tree.pypy Key: SPARK-37427 URL: https://issues.apache.org/jira/browse/SPARK-37427 Project: Spark Issue Type: Sub-task Components: MLlib, PySpark Affects Versions: 3.3.0 Reporter: Maciej Szymkiewicz Inline type hints from python/pyspark/mlib/tree.pyi to python/pyspark/mllib/tree.py -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-37425) Inline type hints for python/pyspark/mllib/recommendation.pypy
Maciej Szymkiewicz created SPARK-37425: -- Summary: Inline type hints for python/pyspark/mllib/recommendation.pypy Key: SPARK-37425 URL: https://issues.apache.org/jira/browse/SPARK-37425 Project: Spark Issue Type: Sub-task Components: MLlib, PySpark Affects Versions: 3.3.0 Reporter: Maciej Szymkiewicz Inline type hints from python/pyspark/mlib/recommendation.pyi to python/pyspark/mllib/recommendation.py -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-37426) Inline type hints for python/pyspark/mllib/regression.pypy
Maciej Szymkiewicz created SPARK-37426: -- Summary: Inline type hints for python/pyspark/mllib/regression.pypy Key: SPARK-37426 URL: https://issues.apache.org/jira/browse/SPARK-37426 Project: Spark Issue Type: Sub-task Components: MLlib, PySpark Affects Versions: 3.3.0 Reporter: Maciej Szymkiewicz Inline type hints from python/pyspark/mlib/regression.pyi to python/pyspark/mllib/regression.py -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-37423) Inline type hints for python/pyspark/mllib/fpm.pypy
Maciej Szymkiewicz created SPARK-37423: -- Summary: Inline type hints for python/pyspark/mllib/fpm.pypy Key: SPARK-37423 URL: https://issues.apache.org/jira/browse/SPARK-37423 Project: Spark Issue Type: Sub-task Components: MLlib, PySpark Affects Versions: 3.3.0 Reporter: Maciej Szymkiewicz Inline type hints from python/pyspark/mlib/fpm.pyi to python/pyspark/mllib/fpm.py -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-37424) Inline type hints for python/pyspark/mllib/random.pypy
Maciej Szymkiewicz created SPARK-37424: -- Summary: Inline type hints for python/pyspark/mllib/random.pypy Key: SPARK-37424 URL: https://issues.apache.org/jira/browse/SPARK-37424 Project: Spark Issue Type: Sub-task Components: MLlib, PySpark Affects Versions: 3.3.0 Reporter: Maciej Szymkiewicz Inline type hints from python/pyspark/mlib/random.pyi to python/pyspark/mllib/random.py -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-37422) Inline type hints for python/pyspark/mllib/feature.pypy
Maciej Szymkiewicz created SPARK-37422: -- Summary: Inline type hints for python/pyspark/mllib/feature.pypy Key: SPARK-37422 URL: https://issues.apache.org/jira/browse/SPARK-37422 Project: Spark Issue Type: Sub-task Components: MLlib, PySpark Affects Versions: 3.3.0 Reporter: Maciej Szymkiewicz Inline type hints from python/pyspark/mlib/feature.pyi to python/pyspark/mllib/feature.py -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-37421) Inline type hints for python/pyspark/mllib/evaluation.pypy
Maciej Szymkiewicz created SPARK-37421: -- Summary: Inline type hints for python/pyspark/mllib/evaluation.pypy Key: SPARK-37421 URL: https://issues.apache.org/jira/browse/SPARK-37421 Project: Spark Issue Type: Sub-task Components: MLlib, PySpark Affects Versions: 3.3.0 Reporter: Maciej Szymkiewicz Inline type hints from python/pyspark/mlib/evaluation.pyi to python/pyspark/mllib/evaluation.py -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-37395) Inline type hint files for files in python/pyspark/ml
[
https://issues.apache.org/jira/browse/SPARK-37395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17446996#comment-17446996
]
Maciej Szymkiewicz commented on SPARK-37395:
cc [~hyukjin.kwon] [~ueshin] [~XinrongM] FYI. Also SPARK-37396
> Inline type hint files for files in python/pyspark/ml
> -
>
> Key: SPARK-37395
> URL: https://issues.apache.org/jira/browse/SPARK-37395
> Project: Spark
> Issue Type: Umbrella
> Components: ML, PySpark
>Affects Versions: 3.3.0
>Reporter: Maciej Szymkiewicz
>Priority: Major
>
> Currently there are type hint stub files ({{*.pyi}}) to show the expected
> types for functions, but we can also take advantage of static type checking
> within the functions by inlining the type hints.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
