[jira] [Resolved] (SPARK-30482) Add sub-class of AppenderSkeleton reusable in tests

[ https://issues.apache.org/jira/browse/SPARK-30482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-30482. -- Fix Version/s: 3.0.0 Assignee: Maxim Gekk Resolution: Fixed Resolved

[jira] [Commented] (SPARK-24915) Calling SparkSession.createDataFrame with schema can throw exception

[ https://issues.apache.org/jira/browse/SPARK-24915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014870#comment-17014870 ] Joachim Hereth commented on SPARK-24915: [~bryanc]  Thanks! The PR was against master and can

[jira] [Created] (SPARK-30507) TableCalalog reserved properties shoudn't be changed with options or tblpropeties

Kent Yao created SPARK-30507: Summary: TableCalalog reserved properties shoudn't be changed with options or tblpropeties Key: SPARK-30507 URL: https://issues.apache.org/jira/browse/SPARK-30507 Project:

[jira] [Updated] (SPARK-30502) PeriodicRDDCheckpointer add storageLevel

[ https://issues.apache.org/jira/browse/SPARK-30502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng updated SPARK-30502: - Summary: PeriodicRDDCheckpointer add storageLevel (was: PeriodicRDDCheckpointer &

[jira] [Updated] (SPARK-30502) PeriodicRDDCheckpointer supports storageLevel

[ https://issues.apache.org/jira/browse/SPARK-30502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng updated SPARK-30502: - Summary: PeriodicRDDCheckpointer supports storageLevel (was: PeriodicRDDCheckpointer add

[jira] [Resolved] (SPARK-30501) Remove SQL config spark.sql.parquet.int64AsTimestampMillis

[ https://issues.apache.org/jira/browse/SPARK-30501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30501. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 27169

[jira] [Resolved] (SPARK-30500) Remove SQL config spark.sql.variable.substitute.depth

[ https://issues.apache.org/jira/browse/SPARK-30500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30500. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 27169

[jira] [Assigned] (SPARK-30500) Remove SQL config spark.sql.variable.substitute.depth

[ https://issues.apache.org/jira/browse/SPARK-30500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-30500: Assignee: Maxim Gekk > Remove SQL config spark.sql.variable.substitute.depth >

[jira] [Assigned] (SPARK-30501) Remove SQL config spark.sql.parquet.int64AsTimestampMillis

[ https://issues.apache.org/jira/browse/SPARK-30501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-30501: Assignee: Maxim Gekk > Remove SQL config spark.sql.parquet.int64AsTimestampMillis >

[jira] [Reopened] (SPARK-28670) [UDF] create permanent UDF does not throw Exception if jar does not exist in HDFS path or Local

[ https://issues.apache.org/jira/browse/SPARK-28670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reopened SPARK-28670: -- Reverted at  [https://github.com/apache/spark/commit/6646b3e13e46b220a33b5798ef266d8a14f3c85b] >

[jira] [Updated] (SPARK-28670) [UDF] create permanent UDF does not throw Exception if jar does not exist in HDFS path or Local

[ https://issues.apache.org/jira/browse/SPARK-28670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-28670: - Fix Version/s: (was: 3.0.0) > [UDF] create permanent UDF does not throw Exception if jar

[jira] [Commented] (SPARK-29988) Adjust Jenkins jobs for `hive-1.2/2.3` combination

[ https://issues.apache.org/jira/browse/SPARK-29988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014783#comment-17014783 ] Dongjoon Hyun commented on SPARK-29988: --- Thank you, [~shaneknapp]! > Adjust Jenkins jobs for

[jira] [Resolved] (SPARK-29206) Number of shuffle Netty server threads should be a multiple of number of chunk fetch handler threads

[ https://issues.apache.org/jira/browse/SPARK-29206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-29206. -- Resolution: Won't Fix > Number of shuffle Netty server threads should be a multiple of number

[jira] [Updated] (SPARK-30502) PeriodicRDDCheckpointer & PeriodicGraphCheckpointer add storageLevel

[ https://issues.apache.org/jira/browse/SPARK-30502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng updated SPARK-30502: - Component/s: (was: GraphX) > PeriodicRDDCheckpointer & PeriodicGraphCheckpointer add

[jira] [Updated] (SPARK-30502) PeriodicRDDCheckpointer & PeriodicGraphCheckpointer add storageLevel

[ https://issues.apache.org/jira/browse/SPARK-30502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng updated SPARK-30502: - Description: Intermediate RDDs in ML are cached with

[jira] [Created] (SPARK-30506) Document general file source options

Gengliang Wang created SPARK-30506: -- Summary: Document general file source options Key: SPARK-30506 URL: https://issues.apache.org/jira/browse/SPARK-30506 Project: Spark Issue Type:

[jira] [Commented] (SPARK-30275) Add gitlab-ci.yml file for reproducible builds

[ https://issues.apache.org/jira/browse/SPARK-30275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014749#comment-17014749 ] Jim Kleckner commented on SPARK-30275: -- [~hyukjin.kwon] As an open source project, the builds in

[jira] [Reopened] (SPARK-24915) Calling SparkSession.createDataFrame with schema can throw exception

[ https://issues.apache.org/jira/browse/SPARK-24915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler reopened SPARK-24915: -- > Calling SparkSession.createDataFrame with schema can throw exception >

[jira] [Commented] (SPARK-24915) Calling SparkSession.createDataFrame with schema can throw exception

[ https://issues.apache.org/jira/browse/SPARK-24915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014719#comment-17014719 ] Bryan Cutler commented on SPARK-24915: -- [~jhereth] apologies for closing prematurely, I didn't know

[jira] [Commented] (SPARK-28266) data duplication when `path` serde property is present

[ https://issues.apache.org/jira/browse/SPARK-28266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014681#comment-17014681 ] Dongjoon Hyun commented on SPARK-28266: --- Oh, I see. Thank you for reply. > data duplication when

[jira] [Updated] (SPARK-21997) Spark shows different results on char/varchar columns on Parquet

[ https://issues.apache.org/jira/browse/SPARK-21997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-21997: -- Affects Version/s: 3.0.0 2.3.4 2.4.4 > Spark

[jira] [Resolved] (SPARK-30492) Eliminate deprecation warnings in ORC datasource

[ https://issues.apache.org/jira/browse/SPARK-30492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk resolved SPARK-30492. Resolution: Won't Fix > Eliminate deprecation warnings in ORC datasource >

[jira] [Created] (SPARK-30505) Deprecate Avro option `ignoreExtension` in a doc

Maxim Gekk created SPARK-30505: -- Summary: Deprecate Avro option `ignoreExtension` in a doc Key: SPARK-30505 URL: https://issues.apache.org/jira/browse/SPARK-30505 Project: Spark Issue Type:

[jira] [Reopened] (SPARK-21997) Spark shows different results on char/varchar columns on Parquet

[ https://issues.apache.org/jira/browse/SPARK-21997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reopened SPARK-21997: --- > Spark shows different results on char/varchar columns on Parquet >

[jira] [Updated] (SPARK-21997) Spark shows different results on char/varchar columns on Parquet

[ https://issues.apache.org/jira/browse/SPARK-21997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-21997: -- Labels: (was: bulk-closed) > Spark shows different results on char/varchar columns on

[jira] [Commented] (SPARK-29988) Adjust Jenkins jobs for `hive-1.2/2.3` combination

[ https://issues.apache.org/jira/browse/SPARK-29988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014552#comment-17014552 ] Shane Knapp commented on SPARK-29988: - ok, i think i got everything!  

[jira] [Comment Edited] (SPARK-28266) data duplication when `path` serde property is present

[ https://issues.apache.org/jira/browse/SPARK-28266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014467#comment-17014467 ] Ruslan Dautkhanov edited comment on SPARK-28266 at 1/13/20 4:46 PM:

[jira] [Commented] (SPARK-28266) data duplication when `path` serde property is present

[ https://issues.apache.org/jira/browse/SPARK-28266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014467#comment-17014467 ] Ruslan Dautkhanov commented on SPARK-28266: --- Thank you for checking [~dongjoon] That may have

[jira] [Resolved] (SPARK-30188) Enable Adaptive Query Execution default

[ https://issues.apache.org/jira/browse/SPARK-30188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-30188. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26813

[jira] [Updated] (SPARK-30188) Fix tests when enable Adaptive Query Execution

[ https://issues.apache.org/jira/browse/SPARK-30188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-30188: Issue Type: Test (was: New Feature) > Fix tests when enable Adaptive Query Execution >

[jira] [Updated] (SPARK-30188) Fix tests when enable Adaptive Query Execution

[ https://issues.apache.org/jira/browse/SPARK-30188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-30188: Summary: Fix tests when enable Adaptive Query Execution (was: Enable Adaptive Query Execution

[jira] [Assigned] (SPARK-30188) Enable Adaptive Query Execution default

[ https://issues.apache.org/jira/browse/SPARK-30188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-30188: --- Assignee: Ke Jia > Enable Adaptive Query Execution default >

[jira] [Resolved] (SPARK-30351) BisectingKMeans support instance weighting

[ https://issues.apache.org/jira/browse/SPARK-30351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-30351. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 27035

[jira] [Assigned] (SPARK-30351) BisectingKMeans support instance weighting

[ https://issues.apache.org/jira/browse/SPARK-30351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-30351: Assignee: Huaxin Gao > BisectingKMeans support instance weighting >

[jira] [Assigned] (SPARK-30377) Make Regressors extend abstract class Regressor

[ https://issues.apache.org/jira/browse/SPARK-30377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-30377: Assignee: Huaxin Gao > Make Regressors extend abstract class Regressor >

[jira] [Resolved] (SPARK-30377) Make Regressors extend abstract class Regressor

[ https://issues.apache.org/jira/browse/SPARK-30377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-30377. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 27168

[jira] [Updated] (SPARK-30377) Make Regressors extend abstract class Regressor

[ https://issues.apache.org/jira/browse/SPARK-30377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-30377: - Priority: Minor (was: Major) > Make Regressors extend abstract class Regressor >

[jira] [Commented] (SPARK-30133) Support DELETE Jar and DELETE File functionality in spark

[ https://issues.apache.org/jira/browse/SPARK-30133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014328#comment-17014328 ] Sandeep Katta commented on SPARK-30133: --- [~hyukjin.kwon] I have raised the PR for these, Please

[jira] [Updated] (SPARK-30133) Support DELETE Jar and DELETE File functionality in spark

[ https://issues.apache.org/jira/browse/SPARK-30133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Katta updated SPARK-30133: -- Description: SPARK should support delete jar feature This feature aims at solving below use

[jira] [Updated] (SPARK-30495) How to disable 'spark.security.credentials.${service}.enabled' in Structured streaming while connecting to a kafka cluster

[ https://issues.apache.org/jira/browse/SPARK-30495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi updated SPARK-30495: -- Priority: Minor (was: Major) > How to disable

[jira] [Updated] (SPARK-30495) How to disable 'spark.security.credentials.${service}.enabled' in Structured streaming while connecting to a kafka cluster

[ https://issues.apache.org/jira/browse/SPARK-30495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi updated SPARK-30495: -- Issue Type: Bug (was: Question) > How to disable

[jira] [Updated] (SPARK-30504) OneVsRest and OneVsRestModel _from_java and _to_java should handle weightCol

[ https://issues.apache.org/jira/browse/SPARK-30504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz updated SPARK-30504: --- Description: Current behaviour {code:python} from pyspark.ml.classification import

[jira] [Created] (SPARK-30504) OneVsRest and OneVsRestModel _from_java and _to_java should handle weightCol

Maciej Szymkiewicz created SPARK-30504: -- Summary: OneVsRest and OneVsRestModel _from_java and _to_java should handle weightCol Key: SPARK-30504 URL: https://issues.apache.org/jira/browse/SPARK-30504

[jira] [Commented] (SPARK-30495) How to disable 'spark.security.credentials.${service}.enabled' in Structured streaming while connecting to a kafka cluster

[ https://issues.apache.org/jira/browse/SPARK-30495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014273#comment-17014273 ] act_coder commented on SPARK-30495: --- [~gsomogyi], [~kabhwan] - Thanks..!  Can i change the Jira type

[jira] [Updated] (SPARK-30503) OnlineLDAOptimizer does not handle persistance correctly

[ https://issues.apache.org/jira/browse/SPARK-30503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng updated SPARK-30503: - Affects Version/s: 2.4.4 > OnlineLDAOptimizer does not handle persistance correctly >

[jira] [Updated] (SPARK-30503) OnlineLDAOptimizer does not handle persistance correctly

[ https://issues.apache.org/jira/browse/SPARK-30503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng updated SPARK-30503: - Description: It seems that in {{OnlineLDAOptimizer, }}{{PeriodicGraphCheckpointer}} can not

[jira] [Created] (SPARK-30503) OnlineLDAOptimizer does not handle persistance correctly

zhengruifeng created SPARK-30503: Summary: OnlineLDAOptimizer does not handle persistance correctly Key: SPARK-30503 URL: https://issues.apache.org/jira/browse/SPARK-30503 Project: Spark

[jira] [Commented] (SPARK-22231) Support of map, filter, withColumn, dropColumn in nested list of structures

[ https://issues.apache.org/jira/browse/SPARK-22231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014252#comment-17014252 ] fqaiser94 commented on SPARK-22231: --- [~rxin] do you have any other questions/thoughts on this?  >

[jira] [Created] (SPARK-30502) PeriodicRDDCheckpointer & PeriodicGraphCheckpointer add storageLevel

zhengruifeng created SPARK-30502: Summary: PeriodicRDDCheckpointer & PeriodicGraphCheckpointer add storageLevel Key: SPARK-30502 URL: https://issues.apache.org/jira/browse/SPARK-30502 Project: Spark

[jira] [Created] (SPARK-30501) Remove SQL config spark.sql.parquet.int64AsTimestampMillis

Maxim Gekk created SPARK-30501: -- Summary: Remove SQL config spark.sql.parquet.int64AsTimestampMillis Key: SPARK-30501 URL: https://issues.apache.org/jira/browse/SPARK-30501 Project: Spark Issue

[jira] [Created] (SPARK-30500) Remove SQL config spark.sql.variable.substitute.depth

Maxim Gekk created SPARK-30500: -- Summary: Remove SQL config spark.sql.variable.substitute.depth Key: SPARK-30500 URL: https://issues.apache.org/jira/browse/SPARK-30500 Project: Spark Issue

[jira] [Created] (SPARK-30499) Remove SQL config spark.sql.execution.pandas.respectSessionTimeZone

Maxim Gekk created SPARK-30499: -- Summary: Remove SQL config spark.sql.execution.pandas.respectSessionTimeZone Key: SPARK-30499 URL: https://issues.apache.org/jira/browse/SPARK-30499 Project: Spark

[jira] [Resolved] (SPARK-30496) Pyspark on kubernetes does not support --py-files from remote storage in cluster mode

[ https://issues.apache.org/jira/browse/SPARK-30496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navdeep Poonia resolved SPARK-30496. Fix Version/s: 3.0.0 Resolution: Fixed > Pyspark on kubernetes does not support

[jira] [Resolved] (SPARK-30493) pyspark.ml.classification.OneVsRestModel shouldn't have setClassifier, setLabelCol and setWeightCol methods

[ https://issues.apache.org/jira/browse/SPARK-30493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng resolved SPARK-30493. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 27181

[jira] [Assigned] (SPARK-30493) pyspark.ml.classification.OneVsRestModel shouldn't have setClassifier, setLabelCol and setWeightCol methods

[ https://issues.apache.org/jira/browse/SPARK-30493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng reassigned SPARK-30493: Assignee: Maciej Szymkiewicz > pyspark.ml.classification.OneVsRestModel shouldn't have

[jira] [Commented] (SPARK-30498) Fix some ml parity issues between python and scala

[ https://issues.apache.org/jira/browse/SPARK-30498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014201#comment-17014201 ] Huaxin Gao commented on SPARK-30498: I will fix the problems.  > Fix some ml parity issues between

[jira] [Created] (SPARK-30498) Fix some ml parity issues between python and scala

Huaxin Gao created SPARK-30498: -- Summary: Fix some ml parity issues between python and scala Key: SPARK-30498 URL: https://issues.apache.org/jira/browse/SPARK-30498 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-30480) Pyspark test "test_memory_limit" fails consistently

[ https://issues.apache.org/jira/browse/SPARK-30480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30480. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 27186

[jira] [Assigned] (SPARK-30480) Pyspark test "test_memory_limit" fails consistently

[ https://issues.apache.org/jira/browse/SPARK-30480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-30480: Assignee: Hyukjin Kwon > Pyspark test "test_memory_limit" fails consistently >

[jira] [Assigned] (SPARK-30218) Columns used in inequality conditions for joins not resolved correctly in case of common lineage

[ https://issues.apache.org/jira/browse/SPARK-30218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-30218: - Assignee: Dongjoon Hyun > Columns used in inequality conditions for joins not resolved

[jira] [Commented] (SPARK-30495) How to disable 'spark.security.credentials.${service}.enabled' in Structured streaming while connecting to a kafka cluster

[ https://issues.apache.org/jira/browse/SPARK-30495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014155#comment-17014155 ] Gabor Somogyi commented on SPARK-30495: --- [~act_coder] thanks for reporting, you've found a bug.

[jira] [Commented] (SPARK-30485) Remove SQL configs deprecated before v2.4

[ https://issues.apache.org/jira/browse/SPARK-30485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014117#comment-17014117 ] Wenchen Fan commented on SPARK-30485: - +1 > Remove SQL configs deprecated before v2.4 >