[jira] [Updated] (SPARK-29328) Incorrect calculation mean seconds per month

[ https://issues.apache.org/jira/browse/SPARK-29328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29328: -- Affects Version/s: 2.1.3 > Incorrect calculation mean seconds per month >

[jira] [Updated] (SPARK-29328) Incorrect calculation mean seconds per month

[ https://issues.apache.org/jira/browse/SPARK-29328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29328: -- Description: Existing implementation assumes 31 days per month or 372 days per year which is

[jira] [Updated] (SPARK-29328) Incorrect calculation mean seconds per month

[ https://issues.apache.org/jira/browse/SPARK-29328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29328: -- Affects Version/s: 2.2.3 > Incorrect calculation mean seconds per month >

[jira] [Updated] (SPARK-29328) Incorrect calculation mean seconds per month

[ https://issues.apache.org/jira/browse/SPARK-29328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29328: -- Affects Version/s: 2.3.4 > Incorrect calculation mean seconds per month >

[jira] [Assigned] (SPARK-29322) History server is stuck reading incomplete event log file compressed with zstd

[ https://issues.apache.org/jira/browse/SPARK-29322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-29322: - Assignee: Jungtaek Lim > History server is stuck reading incomplete event log file

[jira] [Resolved] (SPARK-29322) History server is stuck reading incomplete event log file compressed with zstd

[ https://issues.apache.org/jira/browse/SPARK-29322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-29322. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25996

[jira] [Created] (SPARK-29339) Support Arrow 0.14 in vectoried dapply and gapply (test it in AppVeyor build)

Hyukjin Kwon created SPARK-29339: Summary: Support Arrow 0.14 in vectoried dapply and gapply (test it in AppVeyor build) Key: SPARK-29339 URL: https://issues.apache.org/jira/browse/SPARK-29339

[jira] [Resolved] (SPARK-29338) Add overload for filter with index to Scala/Java API

[ https://issues.apache.org/jira/browse/SPARK-29338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nikolas Vanderhoof resolved SPARK-29338. Resolution: Duplicate > Add overload for filter with index to Scala/Java API >

[jira] [Created] (SPARK-29338) Add overload for filter with index to Scala/Java API

Nikolas Vanderhoof created SPARK-29338: -- Summary: Add overload for filter with index to Scala/Java API Key: SPARK-29338 URL: https://issues.apache.org/jira/browse/SPARK-29338 Project: Spark

[jira] [Commented] (SPARK-28725) Spark ML not able to de-serialize Logistic Regression model saved in previous version of Spark

[ https://issues.apache.org/jira/browse/SPARK-28725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943192#comment-16943192 ] Sharad Varshney commented on SPARK-28725: - Even the same version of Spark 2.4.3 shows huge

[jira] [Commented] (SPARK-18748) UDF multiple evaluations causes very poor performance

[ https://issues.apache.org/jira/browse/SPARK-18748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943176#comment-16943176 ] Anton Baranau commented on SPARK-18748: --- I got the same problem having the code below with 2.4.4

[jira] [Comment Edited] (SPARK-18748) UDF multiple evaluations causes very poor performance

[ https://issues.apache.org/jira/browse/SPARK-18748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943176#comment-16943176 ] Anton Baranau edited comment on SPARK-18748 at 10/2/19 9:32 PM: I got

[jira] [Updated] (SPARK-29336) The implementation of QuantileSummaries.merge does not guarantee that the relativeError will be respected

[ https://issues.apache.org/jira/browse/SPARK-29336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guilherme Souza updated SPARK-29336: Description: Hello Spark maintainers, I was experimenting with my own implementation of

[jira] [Updated] (SPARK-29336) The implementation of QuantileSummaries.merge does not guarantee that the relativeError will be respected

[ https://issues.apache.org/jira/browse/SPARK-29336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guilherme Souza updated SPARK-29336: Description: Hello Spark maintainers, I was experimenting with my own implementation of

[jira] [Updated] (SPARK-29336) The implementation of QuantileSummaries.merge does not guarantee the relativeError will be respected

[ https://issues.apache.org/jira/browse/SPARK-29336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guilherme Souza updated SPARK-29336: Description: Hello Spark maintainers, I was experimenting with my own implementation of

[jira] [Updated] (SPARK-29336) The implementation of QuantileSummaries.merge does not guarantee that the relativeError will be respected

[ https://issues.apache.org/jira/browse/SPARK-29336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guilherme Souza updated SPARK-29336: Summary: The implementation of QuantileSummaries.merge does not guarantee that the

[jira] [Created] (SPARK-29337) How to Cache Table and Pin it in Memory and should not Spill to Disk on Thrift Server

Srini E created SPARK-29337: --- Summary: How to Cache Table and Pin it in Memory and should not Spill to Disk on Thrift Server Key: SPARK-29337 URL: https://issues.apache.org/jira/browse/SPARK-29337

[jira] [Updated] (SPARK-29337) How to Cache Table and Pin it in Memory and should not Spill to Disk on Thrift Server

[ https://issues.apache.org/jira/browse/SPARK-29337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Srini E updated SPARK-29337: Attachment: Cache+Image.png > How to Cache Table and Pin it in Memory and should not Spill to Disk on >

[jira] [Updated] (SPARK-29336) The implementation of QuantileSummaries.merge does not guarantee the relativeError will be respected

[ https://issues.apache.org/jira/browse/SPARK-29336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guilherme Souza updated SPARK-29336: Shepherd: (was: Sean Zhong) Description: (sorry for the early submission, I'm

[jira] [Updated] (SPARK-29335) Cost Based Optimizer stats are not used while evaluating query plans in Spark Sql

[ https://issues.apache.org/jira/browse/SPARK-29335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Srini E updated SPARK-29335: Attachment: explain_plan_cbo_spark.txt > Cost Based Optimizer stats are not used while evaluating query

[jira] [Created] (SPARK-29336) The implementation of QuantileSummaries.merge does not guarantee the relativeError will be respected

Guilherme Souza created SPARK-29336: --- Summary: The implementation of QuantileSummaries.merge does not guarantee the relativeError will be respected Key: SPARK-29336 URL:

[jira] [Created] (SPARK-29335) Cost Based Optimizer stats are not used while evaluating query plans in Spark Sql

Srini E created SPARK-29335: --- Summary: Cost Based Optimizer stats are not used while evaluating query plans in Spark Sql Key: SPARK-29335 URL: https://issues.apache.org/jira/browse/SPARK-29335 Project:

[jira] [Comment Edited] (SPARK-29078) Spark shell fails if read permission is not granted to hive warehouse directory

[ https://issues.apache.org/jira/browse/SPARK-29078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943127#comment-16943127 ] Peter Toth edited comment on SPARK-29078 at 10/2/19 8:15 PM: - I don't think

[jira] [Commented] (SPARK-29078) Spark shell fails if read permission is not granted to hive warehouse directory

[ https://issues.apache.org/jira/browse/SPARK-29078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943127#comment-16943127 ] Peter Toth commented on SPARK-29078: I don't think there should be other databases under

[jira] [Resolved] (SPARK-28962) High-order function: filter(array, function) → array

[ https://issues.apache.org/jira/browse/SPARK-28962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-28962. --- Fix Version/s: 3.0.0 Assignee: Henry Davidge Resolution: Fixed Issue

[jira] [Commented] (SPARK-29078) Spark shell fails if read permission is not granted to hive warehouse directory

[ https://issues.apache.org/jira/browse/SPARK-29078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943118#comment-16943118 ] Mihaly Toth commented on SPARK-29078: - But if the user has access to that directory (which is the

[jira] [Resolved] (SPARK-27297) Add higher order functions to Scala API

[ https://issues.apache.org/jira/browse/SPARK-27297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-27297. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 24232

[jira] [Assigned] (SPARK-27297) Add higher order functions to Scala API

[ https://issues.apache.org/jira/browse/SPARK-27297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin reassigned SPARK-27297: - Assignee: Nikolas Vanderhoof > Add higher order functions to Scala API >

[jira] [Commented] (SPARK-29078) Spark shell fails if read permission is not granted to hive warehouse directory

[ https://issues.apache.org/jira/browse/SPARK-29078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943095#comment-16943095 ] Peter Toth commented on SPARK-29078: [~misutoth], if we look closer at the stacktrace ({{at

[jira] [Resolved] (SPARK-29332) Upgrade zstd-jni library to 1.4.3

[ https://issues.apache.org/jira/browse/SPARK-29332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-29332. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26002

[jira] [Assigned] (SPARK-29332) Upgrade zstd-jni library to 1.4.3

[ https://issues.apache.org/jira/browse/SPARK-29332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-29332: - Assignee: Dongjoon Hyun > Upgrade zstd-jni library to 1.4.3 >

[jira] [Updated] (SPARK-28917) Jobs can hang because of race of RDD.dependencies

[ https://issues.apache.org/jira/browse/SPARK-28917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-28917: - Description: {{RDD.dependencies}} stores the precomputed cache value, but it is not

[jira] [Updated] (SPARK-29334) Supported vector operators in scala should have parity with pySpark

[ https://issues.apache.org/jira/browse/SPARK-29334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Pisciuneri updated SPARK-29334: --- Description: pySpark supports various overloaded operators for the DenseVector type

[jira] [Created] (SPARK-29334) Supported vector operators in scala should have parity with pySpark

Patrick Pisciuneri created SPARK-29334: -- Summary: Supported vector operators in scala should have parity with pySpark Key: SPARK-29334 URL: https://issues.apache.org/jira/browse/SPARK-29334

[jira] [Created] (SPARK-29333) Sample weight in RandomForestRegressor

Jiaqi Guo created SPARK-29333: - Summary: Sample weight in RandomForestRegressor Key: SPARK-29333 URL: https://issues.apache.org/jira/browse/SPARK-29333 Project: Spark Issue Type: New Feature

[jira] [Created] (SPARK-29332) Upgrade zstd-jni library to 1.4.3

Dongjoon Hyun created SPARK-29332: - Summary: Upgrade zstd-jni library to 1.4.3 Key: SPARK-29332 URL: https://issues.apache.org/jira/browse/SPARK-29332 Project: Spark Issue Type: Improvement

[jira] [Resolved] (SPARK-29273) Spark peakExecutionMemory metrics is zero

[ https://issues.apache.org/jira/browse/SPARK-29273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Masiero Vanzin resolved SPARK-29273. Fix Version/s: 3.0.0 Assignee: huangweiyi Resolution:

[jira] [Commented] (SPARK-13802) Fields order in Row(**kwargs) is not consistent with Schema.toInternal method

[ https://issues.apache.org/jira/browse/SPARK-13802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942912#comment-16942912 ] Maciej Szymkiewicz commented on SPARK-13802: [~metasim] namedtuples are the simplest and the

[jira] [Commented] (SPARK-13802) Fields order in Row(**kwargs) is not consistent with Schema.toInternal method

[ https://issues.apache.org/jira/browse/SPARK-13802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942906#comment-16942906 ] Simeon H.K. Fitch commented on SPARK-13802: --- Is there a workaround to this problem? Ordering

[jira] [Created] (SPARK-29331) create DS v2 Write at physical plan

Wenchen Fan created SPARK-29331: --- Summary: create DS v2 Write at physical plan Key: SPARK-29331 URL: https://issues.apache.org/jira/browse/SPARK-29331 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-29330) Allow users to chose the name of Spark Shuffle service

[ https://issues.apache.org/jira/browse/SPARK-29330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Bessonov updated SPARK-29330: --- Description: As of now, Spark uses hardcoded value {{spark_shuffle}} as the name of

[jira] [Created] (SPARK-29330) Allow users to chose the name of Spark Shuffle service

Alexander Bessonov created SPARK-29330: -- Summary: Allow users to chose the name of Spark Shuffle service Key: SPARK-29330 URL: https://issues.apache.org/jira/browse/SPARK-29330 Project: Spark

[jira] [Resolved] (SPARK-28970) implement USE CATALOG/NAMESPACE for Data Source V2

[ https://issues.apache.org/jira/browse/SPARK-28970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-28970. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25771

[jira] [Assigned] (SPARK-28970) implement USE CATALOG/NAMESPACE for Data Source V2

[ https://issues.apache.org/jira/browse/SPARK-28970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-28970: --- Assignee: Terry Kim > implement USE CATALOG/NAMESPACE for Data Source V2 >

[jira] [Comment Edited] (SPARK-29212) Add common classes without using JVM backend

[ https://issues.apache.org/jira/browse/SPARK-29212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942809#comment-16942809 ] Maciej Szymkiewicz edited comment on SPARK-29212 at 10/2/19 1:41 PM: -

[jira] [Commented] (SPARK-29212) Add common classes without using JVM backend

[ https://issues.apache.org/jira/browse/SPARK-29212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942809#comment-16942809 ] Maciej Szymkiewicz commented on SPARK-29212: [~podongfeng] It sounds about right. I will

[jira] [Commented] (SPARK-29329) maven incremental builds not working

[ https://issues.apache.org/jira/browse/SPARK-29329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942790#comment-16942790 ] Thomas Graves commented on SPARK-29329: --- there are few comments on SPARK-28759 in regards to this,

[jira] [Updated] (SPARK-29329) maven incremental builds not working

[ https://issues.apache.org/jira/browse/SPARK-29329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-29329: -- Description: It looks like since we Upgraded scala-maven-plugin to 4.2.0

[jira] [Commented] (SPARK-28759) Upgrade scala-maven-plugin to 4.2.0

[ https://issues.apache.org/jira/browse/SPARK-28759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942787#comment-16942787 ] Thomas Graves commented on SPARK-28759: --- I rolled back this commit and the incremental compile now

[jira] [Created] (SPARK-29329) maven incremental builds not working

Thomas Graves created SPARK-29329: - Summary: maven incremental builds not working Key: SPARK-29329 URL: https://issues.apache.org/jira/browse/SPARK-29329 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-29328) Incorrect calculation mean seconds per month

Maxim Gekk created SPARK-29328: -- Summary: Incorrect calculation mean seconds per month Key: SPARK-29328 URL: https://issues.apache.org/jira/browse/SPARK-29328 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-29327) Support specifying features via multiple columns in Predictor and PredictionModel

Liangcai Li created SPARK-29327: --- Summary: Support specifying features via multiple columns in Predictor and PredictionModel Key: SPARK-29327 URL: https://issues.apache.org/jira/browse/SPARK-29327

[jira] [Created] (SPARK-29326) ANSI store assignment policy: throw exception on insertion failure

Gengliang Wang created SPARK-29326: -- Summary: ANSI store assignment policy: throw exception on insertion failure Key: SPARK-29326 URL: https://issues.apache.org/jira/browse/SPARK-29326 Project: