[jira] [Resolved] (SPARK-28615) Add a guide line for dataframe functions to say column signature function is by default

2019-08-06 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-28615. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 25355

[jira] [Assigned] (SPARK-28615) Add a guide line for dataframe functions to say column signature function is by default

2019-08-06 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-28615: Assignee: Weichen Xu > Add a guide line for dataframe functions to say column signature

[jira] [Resolved] (SPARK-27905) Add higher order function`forall`

2019-08-06 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-27905. --- Resolution: Fixed Assignee: Nikolas Vanderhoof Fix Version/s: 3.0.0 Issue

[jira] [Commented] (SPARK-27495) SPIP: Support Stage level resource configuration and scheduling

2019-08-06 Thread Xingbo Jiang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16901475#comment-16901475 ] Xingbo Jiang commented on SPARK-27495: -- Just my two cents: 1. I would treat the resource

[jira] [Created] (SPARK-28640) Only give warning when session catalog is not defined

2019-08-06 Thread John Zhuge (JIRA)
John Zhuge created SPARK-28640: -- Summary: Only give warning when session catalog is not defined Key: SPARK-28640 URL: https://issues.apache.org/jira/browse/SPARK-28640 Project: Spark Issue

[jira] [Commented] (SPARK-28623) Support `dow`, `isodow` and `doy` at `extract()`

2019-08-06 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16901442#comment-16901442 ] Dongjoon Hyun commented on SPARK-28623: --- [~maxgekk]. The PR only covers DOW, ISODOW, DOY. Could

[jira] [Updated] (SPARK-28623) Support `dow`, `isodow` and `doy` at `extract()`

2019-08-06 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-28623: -- Summary: Support `dow`, `isodow` and `doy` at `extract()` (was: Enhance EXTRACT) > Support

[jira] [Assigned] (SPARK-28623) Enhance EXTRACT

2019-08-06 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-28623: - Assignee: Maxim Gekk > Enhance EXTRACT > --- > > Key:

[jira] [Resolved] (SPARK-28623) Enhance EXTRACT

2019-08-06 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-28623. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 25367

[jira] [Assigned] (SPARK-28514) Remove the redundant transformImpl method in RF & GBT

2019-08-06 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-28514: - Assignee: zhengruifeng > Remove the redundant transformImpl method in RF & GBT >

[jira] [Resolved] (SPARK-28514) Remove the redundant transformImpl method in RF & GBT

2019-08-06 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-28514. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 25256

[jira] [Updated] (SPARK-28514) Remove the redundant transformImpl method in RF & GBT

2019-08-06 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-28514: -- Description: In GBTClassifier & RandomForestClassifier, the real transform methods inherit from

[jira] [Updated] (SPARK-28639) Configuration doc for Barrier Execution Mode

2019-08-06 Thread Kousuke Saruta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-28639: --- Description: SPARK-24817 and SPARK-24819 introduced new 3 non-internal properties for

[jira] [Updated] (SPARK-28639) Configuration doc for Barrier Execution Mode

2019-08-06 Thread Kousuke Saruta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-28639: --- Summary: Configuration doc for Barrier Execution Mode (was: Configuration doc for Barrier

[jira] [Created] (SPARK-28639) Configuration doc for Barrier Mode Execution

2019-08-06 Thread Kousuke Saruta (JIRA)
Kousuke Saruta created SPARK-28639: -- Summary: Configuration doc for Barrier Mode Execution Key: SPARK-28639 URL: https://issues.apache.org/jira/browse/SPARK-28639 Project: Spark Issue Type:

[jira] [Updated] (SPARK-28531) Improve Extract Python UDFs optimizer rule to enforce idempotence

2019-08-06 Thread Yesheng Ma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yesheng Ma updated SPARK-28531: --- Summary: Improve Extract Python UDFs optimizer rule to enforce idempotence (was: Fix Extract

[jira] [Assigned] (SPARK-28622) Move PullOutPythonUDFInJoinCondition rule into 'Extract Python UDFs'

2019-08-06 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reassigned SPARK-28622: --- Assignee: Hyukjin Kwon > Move PullOutPythonUDFInJoinCondition rule into 'Extract Python UDFs' >

[jira] [Resolved] (SPARK-28622) Move PullOutPythonUDFInJoinCondition rule into 'Extract Python UDFs'

2019-08-06 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-28622. - Resolution: Fixed Fix Version/s: 3.0.0 > Move PullOutPythonUDFInJoinCondition rule into 'Extract

[jira] [Created] (SPARK-28638) Task summary metrics are wrong when there are running tasks

2019-08-06 Thread Gengliang Wang (JIRA)
Gengliang Wang created SPARK-28638: -- Summary: Task summary metrics are wrong when there are running tasks Key: SPARK-28638 URL: https://issues.apache.org/jira/browse/SPARK-28638 Project: Spark

[jira] [Assigned] (SPARK-27495) SPIP: Support Stage level resource configuration and scheduling

2019-08-06 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-27495: - Assignee: Thomas Graves > SPIP: Support Stage level resource configuration and

[jira] [Commented] (SPARK-27495) SPIP: Support Stage level resource configuration and scheduling

2019-08-06 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16901253#comment-16901253 ] Thomas Graves commented on SPARK-27495: --- I updated the Jira to have the SPIP text in description.

[jira] [Updated] (SPARK-27495) SPIP: Support Stage level resource configuration and scheduling

2019-08-06 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-27495: -- Description: *Q1.* What are you trying to do? Articulate your objectives using absolutely no

[jira] [Updated] (SPARK-27495) SPIP: Support Stage level resource configuration and scheduling

2019-08-06 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-27495: -- Description: *Q1.* What are you trying to do? Articulate your objectives using absolutely no

[jira] [Updated] (SPARK-27495) SPIP: Support Stage level resource configuration and scheduling

2019-08-06 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-27495: -- Issue Type: Epic (was: Story) > SPIP: Support Stage level resource configuration and

[jira] [Created] (SPARK-28637) Thriftserver can not support interval type

2019-08-06 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-28637: --- Summary: Thriftserver can not support interval type Key: SPARK-28637 URL: https://issues.apache.org/jira/browse/SPARK-28637 Project: Spark Issue Type:

[jira] [Created] (SPARK-28636) Thriftserver can not support decimal type with negative scale

2019-08-06 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-28636: --- Summary: Thriftserver can not support decimal type with negative scale Key: SPARK-28636 URL: https://issues.apache.org/jira/browse/SPARK-28636 Project: Spark

[jira] [Created] (SPARK-28635) create CatalogManager to track registered v2 catalogs

2019-08-06 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-28635: --- Summary: create CatalogManager to track registered v2 catalogs Key: SPARK-28635 URL: https://issues.apache.org/jira/browse/SPARK-28635 Project: Spark Issue

[jira] [Commented] (SPARK-28632) Cannot read from a partitioned table by Int

2019-08-06 Thread Juan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16901065#comment-16901065 ] Juan commented on SPARK-28632: -- Unfortunately, I have to develop based off the current version we have in

[jira] [Commented] (SPARK-28280) Convert and port 'group-by.sql' into UDF test base

2019-08-06 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900855#comment-16900855 ] Weichen Xu commented on SPARK-28280: User 'skonto' has created a pull request for this issue:

[jira] [Comment Edited] (SPARK-4502) Spark SQL reads unneccesary nested fields from Parquet

2019-08-06 Thread Tomas Bartalos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16899534#comment-16899534 ] Tomas Bartalos edited comment on SPARK-4502 at 8/6/19 8:59 AM: --- I'm

[jira] [Comment Edited] (SPARK-4502) Spark SQL reads unneccesary nested fields from Parquet

2019-08-06 Thread Tomas Bartalos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16899534#comment-16899534 ] Tomas Bartalos edited comment on SPARK-4502 at 8/6/19 8:57 AM: --- I'm

[jira] [Commented] (SPARK-28634) Failed to start SparkSession with Keytab file

2019-08-06 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900769#comment-16900769 ] Yuming Wang commented on SPARK-28634: - cc [~vanzin] I will provide more information if you need it.

[jira] [Created] (SPARK-28634) Failed to start SparkSession with Keytab file

2019-08-06 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-28634: --- Summary: Failed to start SparkSession with Keytab file Key: SPARK-28634 URL: https://issues.apache.org/jira/browse/SPARK-28634 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-28627) properties file via SPARK_CONF_DIR and --properties-file prevents definition of own properties via secrets/own mounts

2019-08-06 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900695#comment-16900695 ] Hyukjin Kwon commented on SPARK-28627: -- [~rolandjohann], looks it's better to ask in mailing list

[jira] [Resolved] (SPARK-28630) Union fails when column order is different

2019-08-06 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-28630. -- Resolution: Invalid Use {{unionByName}} instead. > Union fails when column order is

[jira] [Commented] (SPARK-28632) Cannot read from a partitioned table by Int

2019-08-06 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900689#comment-16900689 ] Hyukjin Kwon commented on SPARK-28632: -- I think this was fixed in upstream. Can you test it against

[jira] [Resolved] (SPARK-28632) Cannot read from a partitioned table by Int

2019-08-06 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-28632. -- Resolution: Incomplete > Cannot read from a partitioned table by Int >

[jira] [Updated] (SPARK-28632) Cannot read from a partitioned table by Int

2019-08-06 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-28632: - Description: Getting the following error when reading count from the DataFrame: {code} scala>

[jira] [Updated] (SPARK-28632) Cannot read from a partitioned table by Int

2019-08-06 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-28632: - Labels: (was: beginner) > Cannot read from a partitioned table by Int >

[jira] [Resolved] (SPARK-28633) .write.option("mode","overwrite").saveAsTable("foo") fails with 'already exists' if table foo exists

2019-08-06 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-28633. -- Resolution: Won't Fix We can't predict if all options are valid or not because other

[jira] [Reopened] (SPARK-28633) .write.option("mode","overwrite").saveAsTable("foo") fails with 'already exists' if table foo exists

2019-08-06 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reopened SPARK-28633: -- > .write.option("mode","overwrite").saveAsTable("foo") fails with > 'already exists' if table

[jira] [Resolved] (SPARK-28633) .write.option("mode","overwrite").saveAsTable("foo") fails with 'already exists' if table foo exists

2019-08-06 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-28633. -- Resolution: Invalid > .write.option("mode","overwrite").saveAsTable("foo") fails with >

[jira] [Issue Comment Deleted] (SPARK-28280) Convert and port 'group-by.sql' into UDF test base

2019-08-06 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-28280: - Comment: was deleted (was: Followup was made in https://github.com/apache/spark/pull/25362) >

[jira] [Commented] (SPARK-28279) Convert and port 'group-analytics.sql' into UDF test base

2019-08-06 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900658#comment-16900658 ] Hyukjin Kwon commented on SPARK-28279: -- Followup was made in

[jira] [Commented] (SPARK-28280) Convert and port 'group-by.sql' into UDF test base

2019-08-06 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900655#comment-16900655 ] Hyukjin Kwon commented on SPARK-28280: -- Followup was made in