[jira] [Updated] (SPARK-24976) Allow None for Decimal type conversion (specific to PyArrow 0.9.0)

2018-07-30 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-24976: - Summary: Allow None for Decimal type conversion (specific to PyArrow 0.9.0) (was: Allow None

[jira] [Assigned] (SPARK-24976) Allow None for Decimal type conversion (specific to Arrow 0.9.0)

2018-07-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24976: Assignee: Apache Spark > Allow None for Decimal type conversion (specific to Arrow

[jira] [Assigned] (SPARK-24976) Allow None for Decimal type conversion (specific to Arrow 0.9.0)

2018-07-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24976: Assignee: (was: Apache Spark) > Allow None for Decimal type conversion (specific to

[jira] [Commented] (SPARK-24976) Allow None for Decimal type conversion (specific to Arrow 0.9.0)

2018-07-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16563117#comment-16563117 ] Apache Spark commented on SPARK-24976: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Updated] (SPARK-24930) Exception information is not accurate when using `LOAD DATA LOCAL INPATH`

2018-07-30 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-24930: -- Due Date: (was: 26/Jul/18) Affects Version/s: (was: 2.2.1)

[jira] [Created] (SPARK-24976) Allow None for Decimal type conversion (specific to Arrow 0.9.0)

2018-07-30 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-24976: Summary: Allow None for Decimal type conversion (specific to Arrow 0.9.0) Key: SPARK-24976 URL: https://issues.apache.org/jira/browse/SPARK-24976 Project: Spark

[jira] [Updated] (SPARK-23334) Fix pandas_udf with return type StringType() to handle str type properly in Python 2.

2018-07-30 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-23334: - Issue Type: Sub-task (was: Bug) Parent: SPARK-22216 > Fix pandas_udf with return type

[jira] [Assigned] (SPARK-24820) Fail fast when submitted job contains PartitionPruningRDD in a barrier stage

2018-07-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24820: Assignee: (was: Apache Spark) > Fail fast when submitted job contains

[jira] [Commented] (SPARK-24820) Fail fast when submitted job contains PartitionPruningRDD in a barrier stage

2018-07-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16563109#comment-16563109 ] Apache Spark commented on SPARK-24820: -- User 'jiangxb1987' has created a pull request for this

[jira] [Assigned] (SPARK-24820) Fail fast when submitted job contains PartitionPruningRDD in a barrier stage

2018-07-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24820: Assignee: Apache Spark > Fail fast when submitted job contains PartitionPruningRDD in a

[jira] [Commented] (SPARK-24946) PySpark - Allow np.Arrays and pd.Series in df.approxQuantile

2018-07-30 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16563052#comment-16563052 ] Hyukjin Kwon commented on SPARK-24946: -- Hmmm. mind if I ask a discussion in dev mailing list after

[jira] [Created] (SPARK-24975) Spark history server REST API /api/v1/version returns error 404

2018-07-30 Thread shanyu zhao (JIRA)
shanyu zhao created SPARK-24975: --- Summary: Spark history server REST API /api/v1/version returns error 404 Key: SPARK-24975 URL: https://issues.apache.org/jira/browse/SPARK-24975 Project: Spark

[jira] [Resolved] (SPARK-23633) Update Pandas UDFs section in sql-programming-guide

2018-07-30 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-23633. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21887

[jira] [Assigned] (SPARK-23633) Update Pandas UDFs section in sql-programming-guide

2018-07-30 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-23633: Assignee: Li Jin > Update Pandas UDFs section in sql-programming-guide >

[jira] [Updated] (SPARK-24937) Datasource partition table should load empty static partitions

2018-07-30 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-24937: Summary: Datasource partition table should load empty static partitions (was: Datasource

[jira] [Updated] (SPARK-24882) data source v2 API improvement

2018-07-30 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-24882: Description: Data source V2 is out for a while, see the SPIP

[jira] [Updated] (SPARK-24882) data source v2 API improvement

2018-07-30 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-24882: Description: Data source V2 is out for a while, see the SPIP

[jira] [Updated] (SPARK-24882) data source v2 API improvement

2018-07-30 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-24882: Summary: data source v2 API improvement (was: separate responsibilities of the data source v2

[jira] [Assigned] (SPARK-24952) Support LZMA2 compression by Avro datasource

2018-07-30 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-24952: Assignee: Maxim Gekk > Support LZMA2 compression by Avro datasource >

[jira] [Resolved] (SPARK-24952) Support LZMA2 compression by Avro datasource

2018-07-30 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-24952. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21902

[jira] [Assigned] (SPARK-24972) PivotFirst could not handle pivot columns of complex types

2018-07-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24972: Assignee: Apache Spark > PivotFirst could not handle pivot columns of complex types >

[jira] [Commented] (SPARK-24972) PivotFirst could not handle pivot columns of complex types

2018-07-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16562659#comment-16562659 ] Apache Spark commented on SPARK-24972: -- User 'maryannxue' has created a pull request for this

[jira] [Assigned] (SPARK-24972) PivotFirst could not handle pivot columns of complex types

2018-07-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24972: Assignee: (was: Apache Spark) > PivotFirst could not handle pivot columns of complex

[jira] [Commented] (SPARK-24579) SPIP: Standardize Optimized Data Exchange between Spark and DL/AI frameworks

2018-07-30 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16562638#comment-16562638 ] holdenk commented on SPARK-24579: - [~mengxr]How about you just open comments up in general and then turn

[jira] [Commented] (SPARK-15516) Schema merging in driver fails for parquet when merging LongType and IntegerType

2018-07-30 Thread koert kuipers (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16562632#comment-16562632 ] koert kuipers commented on SPARK-15516: --- we also ran into this on columns that are not key

[jira] [Assigned] (SPARK-24973) Add numIter to Python ClusteringSummary

2018-07-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24973: Assignee: Apache Spark > Add numIter to Python ClusteringSummary >

[jira] [Assigned] (SPARK-24973) Add numIter to Python ClusteringSummary

2018-07-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24973: Assignee: (was: Apache Spark) > Add numIter to Python ClusteringSummary >

[jira] [Commented] (SPARK-24973) Add numIter to Python ClusteringSummary

2018-07-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16562624#comment-16562624 ] Apache Spark commented on SPARK-24973: -- User 'huaxingao' has created a pull request for this issue:

[jira] [Updated] (SPARK-24974) Spark put all file's paths into SharedInMemoryCache even for unused partitions.

2018-07-30 Thread andrzej.stankev...@gmail.com (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] andrzej.stankev...@gmail.com updated SPARK-24974: - Description: SharedInMemoryCache has all  filestatus no matter

[jira] [Updated] (SPARK-24974) Spark put all file's paths into SharedInMemoryCache even for unused partitions.

2018-07-30 Thread andrzej.stankev...@gmail.com (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] andrzej.stankev...@gmail.com updated SPARK-24974: - Description: SharedInMemoryCache has all  filestatus no matter

[jira] [Updated] (SPARK-24974) Spark put all file's paths into SharedInMemoryCache even for unused partitions.

2018-07-30 Thread andrzej.stankev...@gmail.com (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] andrzej.stankev...@gmail.com updated SPARK-24974: - Description: SharedInMemoryCache has all  filestatus no matter

[jira] [Updated] (SPARK-24974) Spark put all file's paths into SharedInMemoryCache even for unused partitions.

2018-07-30 Thread andrzej.stankev...@gmail.com (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] andrzej.stankev...@gmail.com updated SPARK-24974: - Description: SharedInMemoryCache has all  filestatus no matter

[jira] [Updated] (SPARK-24974) Spark put all file's paths into SharedInMemoryCache even for unused partitions.

2018-07-30 Thread andrzej.stankev...@gmail.com (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] andrzej.stankev...@gmail.com updated SPARK-24974: - Description: SharedInMemoryCache has all  filestatus no matter

[jira] [Updated] (SPARK-24974) Spark put all file's paths into SharedInMemoryCache even for unused partitions.

2018-07-30 Thread andrzej.stankev...@gmail.com (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] andrzej.stankev...@gmail.com updated SPARK-24974: - Description: SharedInMemoryCache has all  filestatus no matter

[jira] [Updated] (SPARK-24974) Spark put all file's paths into SharedInMemoryCache even for unused partitions.

2018-07-30 Thread andrzej.stankev...@gmail.com (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] andrzej.stankev...@gmail.com updated SPARK-24974: - Description: SharedInMemoryCache has all  filestatus no matter

[jira] [Created] (SPARK-24974) Spark put all file's paths into SharedInMemoryCache even for unused partitions.

2018-07-30 Thread andrzej.stankev...@gmail.com (JIRA)
andrzej.stankev...@gmail.com created SPARK-24974: Summary: Spark put all file's paths into SharedInMemoryCache even for unused partitions. Key: SPARK-24974 URL:

[jira] [Created] (SPARK-24973) Add numIter to Python ClusteringSummary

2018-07-30 Thread Huaxin Gao (JIRA)
Huaxin Gao created SPARK-24973: -- Summary: Add numIter to Python ClusteringSummary Key: SPARK-24973 URL: https://issues.apache.org/jira/browse/SPARK-24973 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-24972) PivotFirst could not handle pivot columns of complex types

2018-07-30 Thread Maryann Xue (JIRA)
Maryann Xue created SPARK-24972: --- Summary: PivotFirst could not handle pivot columns of complex types Key: SPARK-24972 URL: https://issues.apache.org/jira/browse/SPARK-24972 Project: Spark

[jira] [Commented] (SPARK-24963) Integration tests will fail if they run in a namespace not being the default

2018-07-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16562592#comment-16562592 ] Apache Spark commented on SPARK-24963: -- User 'mccheah' has created a pull request for this issue:

[jira] [Commented] (SPARK-24918) Executor Plugin API

2018-07-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16562521#comment-16562521 ] Apache Spark commented on SPARK-24918: -- User 'squito' has created a pull request for this issue:

[jira] [Assigned] (SPARK-24918) Executor Plugin API

2018-07-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24918: Assignee: (was: Apache Spark) > Executor Plugin API > --- > >

[jira] [Assigned] (SPARK-24918) Executor Plugin API

2018-07-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24918: Assignee: Apache Spark > Executor Plugin API > --- > >

[jira] [Commented] (SPARK-24720) kafka transaction creates Non-consecutive Offsets (due to transaction offset) making streaming fail when failOnDataLoss=true

2018-07-30 Thread Quentin Ambard (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16562464#comment-16562464 ] Quentin Ambard commented on SPARK-24720: Something is wrong with the way the consumers are

[jira] [Commented] (SPARK-24961) sort operation causes out of memory

2018-07-30 Thread Markus Breuer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16562359#comment-16562359 ] Markus Breuer commented on SPARK-24961: --- I think it is an issue and I explained why. But I also

[jira] [Resolved] (SPARK-24963) Integration tests will fail if they run in a namespace not being the default

2018-07-30 Thread Matt Cheah (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Cheah resolved SPARK-24963. Resolution: Fixed Fix Version/s: 2.4.0 > Integration tests will fail if they run in a

[jira] [Commented] (SPARK-24963) Integration tests will fail if they run in a namespace not being the default

2018-07-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16562319#comment-16562319 ] Apache Spark commented on SPARK-24963: -- User 'mccheah' has created a pull request for this issue:

[jira] [Assigned] (SPARK-24963) Integration tests will fail if they run in a namespace not being the default

2018-07-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24963: Assignee: (was: Apache Spark) > Integration tests will fail if they run in a

[jira] [Assigned] (SPARK-24963) Integration tests will fail if they run in a namespace not being the default

2018-07-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24963: Assignee: Apache Spark > Integration tests will fail if they run in a namespace not

[jira] [Assigned] (SPARK-24971) remove SupportsDeprecatedScanRow

2018-07-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24971: Assignee: Wenchen Fan (was: Apache Spark) > remove SupportsDeprecatedScanRow >

[jira] [Commented] (SPARK-24971) remove SupportsDeprecatedScanRow

2018-07-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16562149#comment-16562149 ] Apache Spark commented on SPARK-24971: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Assigned] (SPARK-24971) remove SupportsDeprecatedScanRow

2018-07-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24971: Assignee: Apache Spark (was: Wenchen Fan) > remove SupportsDeprecatedScanRow >

[jira] [Created] (SPARK-24971) remove SupportsDeprecatedScanRow

2018-07-30 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-24971: --- Summary: remove SupportsDeprecatedScanRow Key: SPARK-24971 URL: https://issues.apache.org/jira/browse/SPARK-24971 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-24956) Upgrade maven from 3.3.9 to 3.5.4

2018-07-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16562141#comment-16562141 ] Apache Spark commented on SPARK-24956: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Updated] (SPARK-24934) Complex type and binary type in in-memory partition pruning does not work due to missing upper/lower bounds cases

2018-07-30 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-24934: - Affects Version/s: 2.3.1 > Complex type and binary type in in-memory partition pruning does not

[jira] [Commented] (SPARK-24934) Complex type and binary type in in-memory partition pruning does not work due to missing upper/lower bounds cases

2018-07-30 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16562126#comment-16562126 ] Hyukjin Kwon commented on SPARK-24934: -- I think this has been a bug from the first place. It at

[jira] [Commented] (SPARK-24882) separate responsibilities of the data source v2 read API

2018-07-30 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16562114#comment-16562114 ] Wenchen Fan commented on SPARK-24882: - [~rdblue] I do agree that creating a catalog via reflection

[jira] [Comment Edited] (SPARK-24882) separate responsibilities of the data source v2 read API

2018-07-30 Thread Ryan Blue (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16562058#comment-16562058 ] Ryan Blue edited comment on SPARK-24882 at 7/30/18 4:02 PM: [~cloud_fan],

[jira] [Commented] (SPARK-24882) separate responsibilities of the data source v2 read API

2018-07-30 Thread Ryan Blue (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16562058#comment-16562058 ] Ryan Blue commented on SPARK-24882: --- [~cloud_fan], when you say that "ReadSupport is created via

[jira] [Assigned] (SPARK-24933) SinkProgress should report written rows

2018-07-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24933: Assignee: (was: Apache Spark) > SinkProgress should report written rows >

[jira] [Commented] (SPARK-24933) SinkProgress should report written rows

2018-07-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16562044#comment-16562044 ] Apache Spark commented on SPARK-24933: -- User 'vackosar' has created a pull request for this issue:

[jira] [Assigned] (SPARK-24933) SinkProgress should report written rows

2018-07-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24933: Assignee: Apache Spark > SinkProgress should report written rows >

[jira] [Assigned] (SPARK-24821) Fail fast when submitted job compute on a subset of all the partitions for a barrier stage

2018-07-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24821: Assignee: (was: Apache Spark) > Fail fast when submitted job compute on a subset of

[jira] [Commented] (SPARK-24821) Fail fast when submitted job compute on a subset of all the partitions for a barrier stage

2018-07-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16562031#comment-16562031 ] Apache Spark commented on SPARK-24821: -- User 'jiangxb1987' has created a pull request for this

[jira] [Assigned] (SPARK-24821) Fail fast when submitted job compute on a subset of all the partitions for a barrier stage

2018-07-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24821: Assignee: Apache Spark > Fail fast when submitted job compute on a subset of all the

[jira] [Assigned] (SPARK-24720) kafka transaction creates Non-consecutive Offsets (due to transaction offset) making streaming fail when failOnDataLoss=true

2018-07-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24720: Assignee: Apache Spark > kafka transaction creates Non-consecutive Offsets (due to

[jira] [Commented] (SPARK-24720) kafka transaction creates Non-consecutive Offsets (due to transaction offset) making streaming fail when failOnDataLoss=true

2018-07-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16562000#comment-16562000 ] Apache Spark commented on SPARK-24720: -- User 'QuentinAmbard' has created a pull request for this

[jira] [Assigned] (SPARK-24720) kafka transaction creates Non-consecutive Offsets (due to transaction offset) making streaming fail when failOnDataLoss=true

2018-07-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24720: Assignee: (was: Apache Spark) > kafka transaction creates Non-consecutive Offsets

[jira] [Assigned] (SPARK-24958) Report executors' process tree total memory information to heartbeat signals

2018-07-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24958: Assignee: Apache Spark > Report executors' process tree total memory information to

[jira] [Commented] (SPARK-24958) Report executors' process tree total memory information to heartbeat signals

2018-07-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561990#comment-16561990 ] Apache Spark commented on SPARK-24958: -- User 'rezasafi' has created a pull request for this issue:

[jira] [Assigned] (SPARK-24958) Report executors' process tree total memory information to heartbeat signals

2018-07-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24958: Assignee: (was: Apache Spark) > Report executors' process tree total memory

[jira] [Resolved] (SPARK-24582) Design: Barrier execution mode

2018-07-30 Thread Jiang Xingbo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiang Xingbo resolved SPARK-24582. -- Resolution: Fixed > Design: Barrier execution mode > -- > >

[jira] [Resolved] (SPARK-24581) Design: BarrierTaskContext.barrier()

2018-07-30 Thread Jiang Xingbo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiang Xingbo resolved SPARK-24581. -- Resolution: Fixed > Design: BarrierTaskContext.barrier() >

[jira] [Resolved] (SPARK-22814) JDBC support date/timestamp type as partitionColumn

2018-07-30 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-22814. - Resolution: Fixed Assignee: Takeshi Yamamuro Fix Version/s: 2.4.0 > JDBC support

[jira] [Assigned] (SPARK-24954) Fail fast on job submit if run a barrier stage with dynamic resource allocation enabled

2018-07-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24954: Assignee: (was: Apache Spark) > Fail fast on job submit if run a barrier stage with

[jira] [Commented] (SPARK-24954) Fail fast on job submit if run a barrier stage with dynamic resource allocation enabled

2018-07-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561985#comment-16561985 ] Apache Spark commented on SPARK-24954: -- User 'jiangxb1987' has created a pull request for this

[jira] [Assigned] (SPARK-24954) Fail fast on job submit if run a barrier stage with dynamic resource allocation enabled

2018-07-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24954: Assignee: Apache Spark > Fail fast on job submit if run a barrier stage with dynamic

[jira] [Resolved] (SPARK-24771) Upgrade AVRO version from 1.7.7 to 1.8

2018-07-30 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-24771. - Resolution: Fixed Assignee: Gengliang Wang Fix Version/s: 2.4.0 > Upgrade AVRO version

[jira] [Commented] (SPARK-24965) Spark SQL fails when reading a partitioned hive table with different formats per partition

2018-07-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561976#comment-16561976 ] Apache Spark commented on SPARK-24965: -- User 'krisgeus' has created a pull request for this issue:

[jira] [Assigned] (SPARK-24965) Spark SQL fails when reading a partitioned hive table with different formats per partition

2018-07-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24965: Assignee: Apache Spark > Spark SQL fails when reading a partitioned hive table with

[jira] [Assigned] (SPARK-24965) Spark SQL fails when reading a partitioned hive table with different formats per partition

2018-07-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24965: Assignee: (was: Apache Spark) > Spark SQL fails when reading a partitioned hive

[jira] [Commented] (SPARK-24934) Complex type and binary type in in-memory partition pruning does not work due to missing upper/lower bounds cases

2018-07-30 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561944#comment-16561944 ] Thomas Graves commented on SPARK-24934: --- what is the real affected versions here?  Since it went

[jira] [Updated] (SPARK-24970) Spark Kinesis streaming application fails to recover from streaming checkpoint due to ProvisionedThroughputExceededException

2018-07-30 Thread bruce_zhao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] bruce_zhao updated SPARK-24970: --- Description:   We're using Spark streaming to consume Kinesis data, and found that it reads  more

[jira] [Assigned] (SPARK-24967) Use internal.Logging instead for logging

2018-07-30 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-24967: --- Assignee: Hyukjin Kwon > Use internal.Logging instead for logging >

[jira] [Resolved] (SPARK-24967) Use internal.Logging instead for logging

2018-07-30 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-24967. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21914

[jira] [Resolved] (SPARK-24957) Decimal arithmetic can lead to wrong values using codegen

2018-07-30 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-24957. - Resolution: Fixed Fix Version/s: 2.3.2 2.4.0 Issue resolved by pull

[jira] [Assigned] (SPARK-24957) Decimal arithmetic can lead to wrong values using codegen

2018-07-30 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-24957: --- Assignee: Marco Gaido > Decimal arithmetic can lead to wrong values using codegen >

[jira] [Updated] (SPARK-24957) Decimal arithmetic can lead to wrong values using codegen

2018-07-30 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-24957: Labels: correctness (was: ) > Decimal arithmetic can lead to wrong values using codegen >

[jira] [Comment Edited] (SPARK-24630) SPIP: Support SQLStreaming in Spark

2018-07-30 Thread Jackey Lee (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561798#comment-16561798 ] Jackey Lee edited comment on SPARK-24630 at 7/30/18 11:38 AM: -- For Stream

[jira] [Commented] (SPARK-24630) SPIP: Support SQLStreaming in Spark

2018-07-30 Thread Jackey Lee (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561798#comment-16561798 ] Jackey Lee commented on SPARK-24630: For Stream Table DDL,we have a better way to deal with(such as

[jira] [Comment Edited] (SPARK-24630) SPIP: Support SQLStreaming in Spark

2018-07-30 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561762#comment-16561762 ] Genmao Yu edited comment on SPARK-24630 at 7/30/18 11:27 AM: - Practice to

[jira] [Commented] (SPARK-24946) PySpark - Allow np.Arrays and pd.Series in df.approxQuantile

2018-07-30 Thread Paul Westenthanner (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561784#comment-16561784 ] Paul Westenthanner commented on SPARK-24946: Yes I agree that it's rather sugar than

[jira] [Updated] (SPARK-24970) Spark Kinesis streaming application fails to recover from streaming checkpoint due to ProvisionedThroughputExceededException

2018-07-30 Thread bruce_zhao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] bruce_zhao updated SPARK-24970: --- Description:   We're using Spark streaming to consume Kinesis data, and found that it reads  more

[jira] [Updated] (SPARK-24970) Spark Kinesis streaming application fails to recover from streaming checkpoint due to ProvisionedThroughputExceededException

2018-07-30 Thread bruce_zhao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] bruce_zhao updated SPARK-24970: --- Description:   We're using Spark streaming to consume Kinesis data, and found that it reads  more

[jira] [Created] (SPARK-24970) Spark Kinesis streaming application fails to recover from streaming checkpoint due to ProvisionedThroughputExceededException

2018-07-30 Thread bruce_zhao (JIRA)
bruce_zhao created SPARK-24970: -- Summary: Spark Kinesis streaming application fails to recover from streaming checkpoint due to ProvisionedThroughputExceededException Key: SPARK-24970 URL:

[jira] [Updated] (SPARK-24630) SPIP: Support SQLStreaming in Spark

2018-07-30 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Genmao Yu updated SPARK-24630: -- Attachment: (was: image-2018-07-30-18-48-38-352.png) > SPIP: Support SQLStreaming in Spark >

[jira] [Updated] (SPARK-24630) SPIP: Support SQLStreaming in Spark

2018-07-30 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Genmao Yu updated SPARK-24630: -- Attachment: (was: image-2018-07-30-18-06-30-506.png) > SPIP: Support SQLStreaming in Spark >

[jira] [Comment Edited] (SPARK-24630) SPIP: Support SQLStreaming in Spark

2018-07-30 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561762#comment-16561762 ] Genmao Yu edited comment on SPARK-24630 at 7/30/18 10:53 AM: - Try to add the

[jira] [Commented] (SPARK-24630) SPIP: Support SQLStreaming in Spark

2018-07-30 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561762#comment-16561762 ] Genmao Yu commented on SPARK-24630: --- Try to add the StreamSQL DDL, like this:

[jira] [Updated] (SPARK-24630) SPIP: Support SQLStreaming in Spark

2018-07-30 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Genmao Yu updated SPARK-24630: -- Attachment: image-2018-07-30-18-48-38-352.png > SPIP: Support SQLStreaming in Spark >

[jira] [Updated] (SPARK-24630) SPIP: Support SQLStreaming in Spark

2018-07-30 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Genmao Yu updated SPARK-24630: -- Attachment: image-2018-07-30-18-06-30-506.png > SPIP: Support SQLStreaming in Spark >

  1   2   >