[jira] [Created] (SPARK-26540) Requirement failed when reading numeric arrays from PostgreSQL

2019-01-04 Thread Takeshi Yamamuro (JIRA)
Takeshi Yamamuro created SPARK-26540: Summary: Requirement failed when reading numeric arrays from PostgreSQL Key: SPARK-26540 URL: https://issues.apache.org/jira/browse/SPARK-26540 Project:

[jira] [Comment Edited] (SPARK-26540) Requirement failed when reading numeric arrays from PostgreSQL

2019-01-04 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734715#comment-16734715 ] Dongjoon Hyun edited comment on SPARK-26540 at 1/5/19 12:58 AM: I'm

[jira] [Commented] (SPARK-26540) Requirement failed when reading numeric arrays from PostgreSQL

2019-01-04 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734715#comment-16734715 ] Dongjoon Hyun commented on SPARK-26540: --- I'm checking the information PostgresJDBC ResultSet. I'll

[jira] [Updated] (SPARK-26543) Support the coordinator to demerminte post-shuffle partitions more reasonably

2019-01-04 Thread chenliang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenliang updated SPARK-26543: -- Description: For SparkSQL ,when we open AE by 'set spark.sql.adapative.enable=true',the

[jira] [Updated] (SPARK-26543) Support the coordinator to demerminte post-shuffle partitions more reasonably

2019-01-04 Thread chenliang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenliang updated SPARK-26543: -- Description: For SparkSQL ,when we open AE by 'set spark.sql.adapative.enable=true',the

[jira] [Comment Edited] (SPARK-26540) Requirement failed when reading numeric arrays from PostgreSQL

2019-01-04 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734708#comment-16734708 ] Dongjoon Hyun edited comment on SPARK-26540 at 1/5/19 12:41 AM: For me,

[jira] [Commented] (SPARK-26540) Requirement failed when reading numeric arrays from PostgreSQL

2019-01-04 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734708#comment-16734708 ] Dongjoon Hyun commented on SPARK-26540: --- For me, this is not a correctness issue and not a

[jira] [Updated] (SPARK-26540) Requirement failed when reading numeric arrays from PostgreSQL

2019-01-04 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-26540: -- Priority: Minor (was: Major) > Requirement failed when reading numeric arrays from

[jira] [Updated] (SPARK-26540) Requirement failed when reading numeric arrays from PostgreSQL

2019-01-04 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-26540: -- Issue Type: Improvement (was: Bug) > Requirement failed when reading numeric arrays from

[jira] [Commented] (SPARK-26540) Requirement failed when reading numeric arrays from PostgreSQL

2019-01-04 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734718#comment-16734718 ] Dongjoon Hyun commented on SPARK-26540: --- [~maropu]. Please specify the precision and scale. Then,

[jira] [Assigned] (SPARK-26540) Requirement failed when reading numeric arrays from PostgreSQL

2019-01-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26540: Assignee: (was: Apache Spark) > Requirement failed when reading numeric arrays from

[jira] [Assigned] (SPARK-26540) Requirement failed when reading numeric arrays from PostgreSQL

2019-01-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26540: Assignee: Apache Spark > Requirement failed when reading numeric arrays from PostgreSQL

[jira] [Commented] (SPARK-26540) Requirement failed when reading numeric arrays from PostgreSQL

2019-01-04 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734731#comment-16734731 ] Dongjoon Hyun commented on SPARK-26540: --- Thank you for checking, [~maropu]. I made a PR and will

[jira] [Updated] (SPARK-26544) escape string when serialize map/array make it a valid json (keep alignment with hive)

2019-01-04 Thread EdisonWang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] EdisonWang updated SPARK-26544: --- Summary: escape string when serialize map/array make it a valid json (keep alignment with hive)

[jira] [Updated] (SPARK-26544) escape string when serialize map/array to make it a valid json (keep alignment with hive)

2019-01-04 Thread EdisonWang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] EdisonWang updated SPARK-26544: --- Summary: escape string when serialize map/array to make it a valid json (keep alignment with hive)

[jira] [Commented] (SPARK-26540) Requirement failed when reading numeric arrays from PostgreSQL

2019-01-04 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734701#comment-16734701 ] Takeshi Yamamuro commented on SPARK-26540: -- [~dongjoon] [~smilegator] could you check if this

[jira] [Updated] (SPARK-26540) Requirement failed when reading numeric arrays from PostgreSQL

2019-01-04 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-26540: -- Issue Type: Bug (was: Improvement) > Requirement failed when reading numeric arrays from

[jira] [Resolved] (SPARK-26536) Upgrade Mockito to 2.23.4

2019-01-04 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-26536. --- Resolution: Fixed Assignee: Dongjoon Hyun (was: Apache Spark) Fix

[jira] [Updated] (SPARK-26543) Support the coordinator to demerminte post-shuffle partitions more reasonably

2019-01-04 Thread chenliang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenliang updated SPARK-26543: -- Description: For SparkSQL ,when we open AE by 'set spark.sql.adapative.enable=true',the

[jira] [Updated] (SPARK-26543) Support the coordinator to demerminte post-shuffle partitions more reasonably

2019-01-04 Thread chenliang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenliang updated SPARK-26543: -- Description: For SparkSQL ,when we open AE by 'set spark.sql.adapative.enable=true',the

[jira] [Comment Edited] (SPARK-26540) Requirement failed when reading numeric arrays from PostgreSQL

2019-01-04 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734727#comment-16734727 ] Takeshi Yamamuro edited comment on SPARK-26540 at 1/5/19 1:42 AM: -- I

[jira] [Comment Edited] (SPARK-26540) Requirement failed when reading numeric arrays from PostgreSQL

2019-01-04 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734719#comment-16734719 ] Dongjoon Hyun edited comment on SPARK-26540 at 1/5/19 1:34 AM: --- Given that

[jira] [Created] (SPARK-26541) Add `-Pdocker-integration-tests` to `dev/scalastyle`

2019-01-04 Thread Dongjoon Hyun (JIRA)
Dongjoon Hyun created SPARK-26541: - Summary: Add `-Pdocker-integration-tests` to `dev/scalastyle` Key: SPARK-26541 URL: https://issues.apache.org/jira/browse/SPARK-26541 Project: Spark Issue

[jira] [Assigned] (SPARK-26541) Add `-Pdocker-integration-tests` to `dev/scalastyle`

2019-01-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26541: Assignee: (was: Apache Spark) > Add `-Pdocker-integration-tests` to `dev/scalastyle`

[jira] [Assigned] (SPARK-26541) Add `-Pdocker-integration-tests` to `dev/scalastyle`

2019-01-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26541: Assignee: Apache Spark > Add `-Pdocker-integration-tests` to `dev/scalastyle` >

[jira] [Created] (SPARK-26542) Support the coordinator to demerminte post-shuffle partitions more reasonably

2019-01-04 Thread chenliang (JIRA)
chenliang created SPARK-26542: - Summary: Support the coordinator to demerminte post-shuffle partitions more reasonably Key: SPARK-26542 URL: https://issues.apache.org/jira/browse/SPARK-26542 Project:

[jira] [Updated] (SPARK-26543) Support the coordinator to demerminte post-shuffle partitions more reasonably

2019-01-04 Thread chenliang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenliang updated SPARK-26543: -- Description: For SparkSQL ,when we open AE by 'set spark.sql.adapative.enable=true',the

[jira] [Created] (SPARK-26544) the string serialized from map/array type is not a valid json (while hive is)

2019-01-04 Thread EdisonWang (JIRA)
EdisonWang created SPARK-26544: -- Summary: the string serialized from map/array type is not a valid json (while hive is) Key: SPARK-26544 URL: https://issues.apache.org/jira/browse/SPARK-26544 Project:

[jira] [Updated] (SPARK-26543) Support the coordinator to demerminte post-shuffle partitions more reasonably

2019-01-04 Thread chenliang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenliang updated SPARK-26543: -- Attachment: (was: 15_24_38__12_27_2018.jpg) > Support the coordinator to demerminte post-shuffle

[jira] [Comment Edited] (SPARK-26540) Requirement failed when reading numeric arrays from PostgreSQL

2019-01-04 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734719#comment-16734719 ] Dongjoon Hyun edited comment on SPARK-26540 at 1/5/19 1:41 AM: --- Given that

[jira] [Commented] (SPARK-26540) Requirement failed when reading numeric arrays from PostgreSQL

2019-01-04 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734727#comment-16734727 ] Takeshi Yamamuro commented on SPARK-26540: -- yea, the workaround is good. The query failed in

[jira] [Updated] (SPARK-26543) Support the coordinator to demerminte post-shuffle partitions more reasonably

2019-01-04 Thread chenliang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenliang updated SPARK-26543: -- Description: For SparkSQL ,when we open AE by 'set spark.sql.adapative.enable=true',the

[jira] [Updated] (SPARK-26540) Requirement failed when reading numeric arrays from PostgreSQL

2019-01-04 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-26540: - Description: This bug was reported in spark-user:

[jira] [Updated] (SPARK-26540) Requirement failed when reading numeric arrays from PostgreSQL

2019-01-04 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-26540: -- Affects Version/s: 2.0.2 2.1.3 > Requirement failed when reading

[jira] [Updated] (SPARK-26543) Support the coordinator to demerminte post-shuffle partitions more reasonably

2019-01-04 Thread chenliang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenliang updated SPARK-26543: -- Description: For SparkSQL ,when we open AE by 'set spark.sql.adapative.enable=true',the

[jira] [Updated] (SPARK-26543) Support the coordinator to demerminte post-shuffle partitions more reasonably

2019-01-04 Thread chenliang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenliang updated SPARK-26543: -- Attachment: image-2019-01-05-13-18-30-487.png > Support the coordinator to demerminte post-shuffle

[jira] [Updated] (SPARK-26543) Support the coordinator to demerminte post-shuffle partitions more reasonably

2019-01-04 Thread chenliang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenliang updated SPARK-26543: -- Attachment: screenshot-1.png > Support the coordinator to demerminte post-shuffle partitions more

[jira] [Updated] (SPARK-26543) Support the coordinator to demerminte post-shuffle partitions more reasonably

2019-01-04 Thread chenliang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenliang updated SPARK-26543: -- Attachment: (was: screenshot-1.png) > Support the coordinator to demerminte post-shuffle

[jira] [Commented] (SPARK-26540) Requirement failed when reading numeric arrays from PostgreSQL

2019-01-04 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734719#comment-16734719 ] Dongjoon Hyun commented on SPARK-26540: --- Given that Apache Spark handles the decimal type with

[jira] [Created] (SPARK-26543) Support the coordinator to demerminte post-shuffle partitions more reasonably

2019-01-04 Thread chenliang (JIRA)
chenliang created SPARK-26543: - Summary: Support the coordinator to demerminte post-shuffle partitions more reasonably Key: SPARK-26543 URL: https://issues.apache.org/jira/browse/SPARK-26543 Project:

[jira] [Updated] (SPARK-26543) Support the coordinator to demerminte post-shuffle partitions more reasonably

2019-01-04 Thread chenliang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenliang updated SPARK-26543: -- Attachment: 15_24_38__12_27_2018.jpg > Support the coordinator to demerminte post-shuffle partitions

[jira] [Updated] (SPARK-26078) WHERE .. IN fails to filter rows when used in combination with UNION

2019-01-04 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-26078: -- Fix Version/s: 2.4.1 > WHERE .. IN fails to filter rows when used in combination with UNION >

[jira] [Commented] (SPARK-26511) java.lang.ClassCastException error when loading Spark MLlib model from parquet file

2019-01-04 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734009#comment-16734009 ] Hyukjin Kwon commented on SPARK-26511: -- BTW, please make sure such facts are described in the JIRA

[jira] [Updated] (SPARK-25919) Date value corrupts when tables are "ParquetHiveSerDe" formatted and target table is Partitioned

2019-01-04 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-25919: - Priority: Major (was: Blocker) > Date value corrupts when tables are "ParquetHiveSerDe"

[jira] [Resolved] (SPARK-25873) Date corruption when Spark and Hive both are on different timezones

2019-01-04 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-25873. -- Resolution: Fixed > Date corruption when Spark and Hive both are on different timezones >

[jira] [Resolved] (SPARK-19217) Offer easy cast from vector to array

2019-01-04 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-19217. -- Resolution: Later Let's track this in another JIRA. It's been inactive for so long time. At

[jira] [Assigned] (SPARK-26126) Put scala-library deps into root pom instead of spark-tags module

2019-01-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26126: Assignee: Apache Spark > Put scala-library deps into root pom instead of spark-tags

[jira] [Updated] (SPARK-26334) NullPointerException in CallMethodViaReflection when we apply reflect function for empty field

2019-01-04 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-26334: - Target Version/s: (was: 2.2.0, 2.3.0) > NullPointerException in CallMethodViaReflection when

[jira] [Commented] (SPARK-26334) NullPointerException in CallMethodViaReflection when we apply reflect function for empty field

2019-01-04 Thread chenliang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734142#comment-16734142 ] chenliang commented on SPARK-26334: --- [~hyukjin.kwon]Thanks for your attention,I did know that the

[jira] [Updated] (SPARK-26512) Spark 2.4.0 is not working with Hadoop 2.8.3 in windows 10

2019-01-04 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-26512: - Summary: Spark 2.4.0 is not working with Hadoop 2.8.3 in windows 10 (was: Spark 2.4.0 is not

[jira] [Commented] (SPARK-26254) Move delegation token providers into a separate project

2019-01-04 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734004#comment-16734004 ] Hyukjin Kwon commented on SPARK-26254: -- [~gsomogyi], can you point out which comment is about the

[jira] [Assigned] (SPARK-26126) Put scala-library deps into root pom instead of spark-tags module

2019-01-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26126: Assignee: (was: Apache Spark) > Put scala-library deps into root pom instead of

[jira] [Updated] (SPARK-24661) Window API - using multiple fields for partitioning with WindowSpec API and dataset that is cached causes org.apache.spark.sql.catalyst.errors.package$TreeNodeException

2019-01-04 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-24661: - Component/s: (was: PySpark) > Window API - using multiple fields for partitioning with

[jira] [Commented] (SPARK-26512) Spark 2.4.0 is not working with Hadoop 2.8.3 in windows 10

2019-01-04 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734072#comment-16734072 ] Saisai Shao commented on SPARK-26512: - This seems like a Netty version problem,

[jira] [Updated] (SPARK-26334) NullPointerException in CallMethodViaReflection when we apply reflect function for empty field

2019-01-04 Thread chenliang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenliang updated SPARK-26334: -- Attachment: 1537537286575.jpg > NullPointerException in CallMethodViaReflection when we apply

[jira] [Commented] (SPARK-26334) NullPointerException in CallMethodViaReflection when we apply reflect function for empty field

2019-01-04 Thread chenliang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734138#comment-16734138 ] chenliang commented on SPARK-26334: --- [~hyukjin.kwon] Thank you ,I did know that the method itself

[jira] [Updated] (SPARK-26529) Add debug logs for confArchive when preparing local resource

2019-01-04 Thread liupengcheng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liupengcheng updated SPARK-26529: - Summary: Add debug logs for confArchive when preparing local resource (was: Add logs for

[jira] [Resolved] (SPARK-26334) NullPointerException in CallMethodViaReflection when we apply reflect function for empty field

2019-01-04 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-26334. -- Resolution: Invalid The method itself already throws an NPE: {code} scala>

[jira] [Issue Comment Deleted] (SPARK-26334) NullPointerException in CallMethodViaReflection when we apply reflect function for empty field

2019-01-04 Thread chenliang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenliang updated SPARK-26334: -- Comment: was deleted (was: [~hyukjin.kwon] Thank you ,I did know that the method itself throws an

[jira] [Updated] (SPARK-26334) NullPointerException in CallMethodViaReflection when we apply reflect function for empty field

2019-01-04 Thread chenliang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenliang updated SPARK-26334: -- Attachment: (was: 1537537286575.jpg) > NullPointerException in CallMethodViaReflection when we

[jira] [Updated] (SPARK-26334) NullPointerException in CallMethodViaReflection when we apply reflect function for empty field

2019-01-04 Thread chenliang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenliang updated SPARK-26334: -- Attachment: 21_47_01__01_04_2019.jpg > NullPointerException in CallMethodViaReflection when we apply

[jira] [Resolved] (SPARK-26454) While creating new UDF with JAR though UDF is created successfully, it throws IllegegalArgument Exception

2019-01-04 Thread Udbhav Agrawal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udbhav Agrawal resolved SPARK-26454. Resolution: Invalid Apparently i placed wrong jar in different version, this issue seems

[jira] [Commented] (SPARK-26359) Spark checkpoint restore fails after query restart

2019-01-04 Thread Kaspar Tint (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16733935#comment-16733935 ] Kaspar Tint commented on SPARK-26359: - [~gsomogyi]: Yes, this ticket can be closed. The solution

[jira] [Updated] (SPARK-26529) Add logs for IOException when preparing local resource

2019-01-04 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-26529: - Priority: Trivial (was: Major) > Add logs for IOException when preparing local resource >

[jira] [Commented] (SPARK-26512) Spark 2.4.0 is not working with Hadoop 2.8.3 in windows 10?

2019-01-04 Thread Anubhav Jain (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16733917#comment-16733917 ] Anubhav Jain commented on SPARK-26512: -- Thanks [~jerryshao] , for the reply . I Have attached a pic

[jira] [Closed] (SPARK-26359) Spark checkpoint restore fails after query restart

2019-01-04 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi closed SPARK-26359. - > Spark checkpoint restore fails after query restart >

[jira] [Resolved] (SPARK-26359) Spark checkpoint restore fails after query restart

2019-01-04 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi resolved SPARK-26359. --- Resolution: Information Provided > Spark checkpoint restore fails after query restart >

[jira] [Commented] (SPARK-26254) Move delegation token providers into a separate project

2019-01-04 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16733956#comment-16733956 ] Gabor Somogyi commented on SPARK-26254: --- ping [~vanzin] > Move delegation token providers into a

[jira] [Commented] (SPARK-26462) Use ConfigEntry for hardcoded configs for execution categories.

2019-01-04 Thread pralabhkumar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16733988#comment-16733988 ] pralabhkumar commented on SPARK-26462: -- [~ueshin] Have created pull request with  spark.memory, 

[jira] [Updated] (SPARK-26126) Put scala-library deps into root pom instead of spark-tags module

2019-01-04 Thread liupengcheng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liupengcheng updated SPARK-26126: - Summary: Put scala-library deps into root pom instead of spark-tags module (was: Should put

[jira] [Assigned] (SPARK-26462) Use ConfigEntry for hardcoded configs for execution categories.

2019-01-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26462: Assignee: (was: Apache Spark) > Use ConfigEntry for hardcoded configs for execution

[jira] [Assigned] (SPARK-26462) Use ConfigEntry for hardcoded configs for execution categories.

2019-01-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26462: Assignee: Apache Spark > Use ConfigEntry for hardcoded configs for execution categories.

[jira] [Resolved] (SPARK-26445) Use ConfigEntry for hardcoded configs for driver/executor categories.

2019-01-04 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-26445. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23415

[jira] [Commented] (SPARK-26334) NullPointerException in CallMethodViaReflection when we apply reflect function for empty field

2019-01-04 Thread chenliang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734182#comment-16734182 ] chenliang commented on SPARK-26334: --- Anyway,thanks. > NullPointerException in

[jira] [Resolved] (SPARK-7428) DataFrame.join() could create a new df with duplicate column name

2019-01-04 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-7428. - Resolution: Won't Fix > DataFrame.join() could create a new df with duplicate column name >

[jira] [Resolved] (SPARK-4235) Add union data type support

2019-01-04 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-4235. - Resolution: Later I'm going to leave this resolved as {{Later}}. It's been inactive for few

[jira] [Resolved] (SPARK-7789) sql on security hbase:Token generation only allowed for Kerberos authenticated clients

2019-01-04 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-7789. - Resolution: Incomplete Leaving this resolved due to inactivity from the reporter. > sql on

[jira] [Resolved] (SPARK-8403) Pruner partition won't effective when partition field and fieldSchema exist in sql predicate

2019-01-04 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-8403. - Resolution: Incomplete Leaving this resolved due to reporter's inactivity. > Pruner partition

[jira] [Commented] (SPARK-9218) Falls back to getAllPartitions when getPartitionsByFilter fails

2019-01-04 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734218#comment-16734218 ] Hyukjin Kwon commented on SPARK-9218: - ping [~lian cheng] > Falls back to getAllPartitions when

[jira] [Resolved] (SPARK-8370) Add API for data sources to register databases

2019-01-04 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-8370. - Resolution: Not A Problem > Add API for data sources to register databases >

[jira] [Created] (SPARK-26534) Closure Cleaner Bug

2019-01-04 Thread sam (JIRA)
sam created SPARK-26534: --- Summary: Closure Cleaner Bug Key: SPARK-26534 URL: https://issues.apache.org/jira/browse/SPARK-26534 Project: Spark Issue Type: Bug Components: Spark Core

[jira] [Assigned] (SPARK-26445) Use ConfigEntry for hardcoded configs for driver/executor categories.

2019-01-04 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-26445: Assignee: Takuya Ueshin > Use ConfigEntry for hardcoded configs for driver/executor

[jira] [Resolved] (SPARK-7754) [SQL] Use PartialFunction literals instead of objects in Catalyst

2019-01-04 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-7754. - Resolution: Won't Fix > [SQL] Use PartialFunction literals instead of objects in Catalyst >

[jira] [Resolved] (SPARK-8602) Shared cached DataFrames

2019-01-04 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-8602. - Resolution: Not A Problem > Shared cached DataFrames > > >

[jira] [Commented] (SPARK-26436) Dataframe resulting from a GroupByKey and flatMapGroups operation throws java.lang.UnsupportedException when groupByKey is applied on it.

2019-01-04 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734282#comment-16734282 ] Liang-Chi Hsieh commented on SPARK-26436: - Sorry I don't know what you mean "should groupByKey 

[jira] [Created] (SPARK-26535) Parsing literals as DOUBLE instead of DECIMAL

2019-01-04 Thread Marco Gaido (JIRA)
Marco Gaido created SPARK-26535: --- Summary: Parsing literals as DOUBLE instead of DECIMAL Key: SPARK-26535 URL: https://issues.apache.org/jira/browse/SPARK-26535 Project: Spark Issue Type:

[jira] [Created] (SPARK-26536) Upgrade Mockito to 2.23.4

2019-01-04 Thread Dongjoon Hyun (JIRA)
Dongjoon Hyun created SPARK-26536: - Summary: Upgrade Mockito to 2.23.4 Key: SPARK-26536 URL: https://issues.apache.org/jira/browse/SPARK-26536 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-26436) Dataframe resulting from a GroupByKey and flatMapGroups operation throws java.lang.UnsupportedException when groupByKey is applied on it.

2019-01-04 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734178#comment-16734178 ] Liang-Chi Hsieh commented on SPARK-26436: - This issue is caused by how you create the row:

[jira] [Commented] (SPARK-4476) Use MapType for dict in json which has unique keys in each row.

2019-01-04 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734184#comment-16734184 ] Hyukjin Kwon commented on SPARK-4476: - I think we can resolve this now in favour of {{from_json}} and

[jira] [Resolved] (SPARK-4476) Use MapType for dict in json which has unique keys in each row.

2019-01-04 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-4476. - Resolution: Done > Use MapType for dict in json which has unique keys in each row. >

[jira] [Resolved] (SPARK-7148) Configure Parquet block size (row group size) for ML model import/export

2019-01-04 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-7148. - Resolution: Not A Problem Then, I'm leaving resolved. Please reopen this if I am mistaken. >

[jira] [Resolved] (SPARK-7755) MetadataCache.refresh does not take into account _SUCCESS

2019-01-04 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-7755. - Resolution: Not A Problem > MetadataCache.refresh does not take into account _SUCCESS >

[jira] [Commented] (SPARK-8324) Register Query as view through JDBC interface

2019-01-04 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734203#comment-16734203 ] Hyukjin Kwon commented on SPARK-8324: - Fixed in SPARK-24423 > Register Query as view through JDBC

[jira] [Resolved] (SPARK-8324) Register Query as view through JDBC interface

2019-01-04 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-8324. - Resolution: Duplicate > Register Query as view through JDBC interface >

[jira] [Commented] (SPARK-7754) [SQL] Use PartialFunction literals instead of objects in Catalyst

2019-01-04 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734197#comment-16734197 ] Hyukjin Kwon commented on SPARK-7754: - I don't think we're going to change if it targets mainly

[jira] [Comment Edited] (SPARK-22231) Support of map, filter, withColumn, dropColumn in nested list of structures

2019-01-04 Thread Timothy Pharo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734160#comment-16734160 ] Timothy Pharo edited comment on SPARK-22231 at 1/4/19 2:15 PM: --- [~dbtsai]

[jira] [Resolved] (SPARK-6540) Spark SQL thrift server fails to pass settings to following query in the same session

2019-01-04 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-6540. - Resolution: Incomplete Leaving this resolved due to inactivity from the reporter > Spark SQL

[jira] [Resolved] (SPARK-8102) Big performance difference when joining 3 tables in different order

2019-01-04 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-8102. - Resolution: Not A Problem The code path has been drastically changed. Now we're heading for

[jira] [Commented] (SPARK-8370) Add API for data sources to register databases

2019-01-04 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734205#comment-16734205 ] Hyukjin Kwon commented on SPARK-8370: - I think we're working on multiple catalog support. I am

[jira] [Resolved] (SPARK-8577) ScalaReflectionLock.synchronized can cause deadlock

2019-01-04 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-8577. - Resolution: Not A Problem I'm leaving resolved since we removed 2.10. >

  1   2   >