[jira] [Updated] (SPARK-24256) ExpressionEncoder should support user-defined types as fields of Scala case class and tuple

[ https://issues.apache.org/jira/browse/SPARK-24256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fangshi Li updated SPARK-24256: --- Description: Right now, ExpressionEncoder supports ser/de of primitive types, as well as scala case

[jira] [Updated] (SPARK-24256) ExpressionEncoder should support user-defined types as fields of Scala case class and tuple

[ https://issues.apache.org/jira/browse/SPARK-24256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fangshi Li updated SPARK-24256: --- Description: Right now, ExpressionEncoder supports ser/de of primitive types, as well as scala case

[jira] [Created] (SPARK-24256) ExpressionEncoder should support user-defined types as fields of Scala case class and tuple

Fangshi Li created SPARK-24256: -- Summary: ExpressionEncoder should support user-defined types as fields of Scala case class and tuple Key: SPARK-24256 URL: https://issues.apache.org/jira/browse/SPARK-24256

[jira] [Updated] (SPARK-24174) Expose Hadoop config as part of /environment API

[ https://issues.apache.org/jira/browse/SPARK-24174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nikolay Sokolov updated SPARK-24174: Description: Currently, UI or /environment API call of HistoryServer or WebUI exposes only

[jira] [Updated] (SPARK-24174) Expose Hadoop config as part of /environment API

[ https://issues.apache.org/jira/browse/SPARK-24174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nikolay Sokolov updated SPARK-24174: Description: Currently, /environment API call exposes only system properties and 

[jira] [Commented] (SPARK-24174) Expose Hadoop config as part of /environment API

[ https://issues.apache.org/jira/browse/SPARK-24174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472884#comment-16472884 ] Nikolay Sokolov commented on SPARK-24174: - [~jerryshao] as far a I understand, YARN exposes

[jira] [Commented] (SPARK-24255) Require Java 8 in SparkR description

[ https://issues.apache.org/jira/browse/SPARK-24255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472817#comment-16472817 ] Shivaram Venkataraman commented on SPARK-24255: --- Resolved by 

[jira] [Resolved] (SPARK-24255) Require Java 8 in SparkR description

[ https://issues.apache.org/jira/browse/SPARK-24255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivaram Venkataraman resolved SPARK-24255. --- Resolution: Fixed Assignee: Shivaram Venkataraman Fix

[jira] [Created] (SPARK-24255) Require Java 8 in SparkR description

Shivaram Venkataraman created SPARK-24255: - Summary: Require Java 8 in SparkR description Key: SPARK-24255 URL: https://issues.apache.org/jira/browse/SPARK-24255 Project: Spark Issue

[jira] [Commented] (SPARK-23907) Support regr_* functions

[ https://issues.apache.org/jira/browse/SPARK-23907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472797#comment-16472797 ] Apache Spark commented on SPARK-23907: -- User 'rxin' has created a pull request for this issue:

[jira] [Created] (SPARK-24254) Eagerly evaluate some subqueries over LocalRelation

Henry Robinson created SPARK-24254: -- Summary: Eagerly evaluate some subqueries over LocalRelation Key: SPARK-24254 URL: https://issues.apache.org/jira/browse/SPARK-24254 Project: Spark

[jira] [Assigned] (SPARK-22594) Handling spark-submit and master version mismatch

[ https://issues.apache.org/jira/browse/SPARK-22594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-22594: -- Assignee: (was: Marcelo Vanzin) > Handling spark-submit and master version

[jira] [Assigned] (SPARK-22594) Handling spark-submit and master version mismatch

[ https://issues.apache.org/jira/browse/SPARK-22594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-22594: -- Assignee: Marcelo Vanzin > Handling spark-submit and master version mismatch >

[jira] [Assigned] (SPARK-24253) DataSourceV2: Add DeleteSupport for delete and overwrite operations

[ https://issues.apache.org/jira/browse/SPARK-24253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24253: Assignee: Apache Spark > DataSourceV2: Add DeleteSupport for delete and overwrite

[jira] [Commented] (SPARK-24253) DataSourceV2: Add DeleteSupport for delete and overwrite operations

[ https://issues.apache.org/jira/browse/SPARK-24253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472741#comment-16472741 ] Apache Spark commented on SPARK-24253: -- User 'rdblue' has created a pull request for this issue:

[jira] [Assigned] (SPARK-24253) DataSourceV2: Add DeleteSupport for delete and overwrite operations

[ https://issues.apache.org/jira/browse/SPARK-24253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24253: Assignee: (was: Apache Spark) > DataSourceV2: Add DeleteSupport for delete and

[jira] [Created] (SPARK-24253) DataSourceV2: Add DeleteSupport for delete and overwrite operations

Ryan Blue created SPARK-24253: - Summary: DataSourceV2: Add DeleteSupport for delete and overwrite operations Key: SPARK-24253 URL: https://issues.apache.org/jira/browse/SPARK-24253 Project: Spark

[jira] [Commented] (SPARK-24186) add array reverse and concat

[ https://issues.apache.org/jira/browse/SPARK-24186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472718#comment-16472718 ] Apache Spark commented on SPARK-24186: -- User 'huaxingao' has created a pull request for this issue:

[jira] [Commented] (SPARK-22232) Row objects in pyspark created using the `Row(**kwars)` syntax do not get serialized/deserialized properly

[ https://issues.apache.org/jira/browse/SPARK-22232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472714#comment-16472714 ] Bryan Cutler commented on SPARK-22232: -- I'm closing the PR for now, will reopen for Spark 3.0.0.

[jira] [Commented] (SPARK-24252) DataSourceV2: Add catalog support

[ https://issues.apache.org/jira/browse/SPARK-24252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472686#comment-16472686 ] Apache Spark commented on SPARK-24252: -- User 'rdblue' has created a pull request for this issue:

[jira] [Assigned] (SPARK-24252) DataSourceV2: Add catalog support

[ https://issues.apache.org/jira/browse/SPARK-24252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24252: Assignee: Apache Spark > DataSourceV2: Add catalog support >

[jira] [Assigned] (SPARK-24252) DataSourceV2: Add catalog support

[ https://issues.apache.org/jira/browse/SPARK-24252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24252: Assignee: (was: Apache Spark) > DataSourceV2: Add catalog support >

[jira] [Created] (SPARK-24252) DataSourceV2: Add catalog support

Ryan Blue created SPARK-24252: - Summary: DataSourceV2: Add catalog support Key: SPARK-24252 URL: https://issues.apache.org/jira/browse/SPARK-24252 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-23321) DataSourceV2 should apply some validation when writing.

[ https://issues.apache.org/jira/browse/SPARK-23321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472675#comment-16472675 ] Ryan Blue commented on SPARK-23321: --- I've closed the PR associated with this because validating writes

[jira] [Commented] (SPARK-24251) DataSourceV2: Add AppendData logical operation

[ https://issues.apache.org/jira/browse/SPARK-24251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472655#comment-16472655 ] Apache Spark commented on SPARK-24251: -- User 'rdblue' has created a pull request for this issue:

[jira] [Assigned] (SPARK-24251) DataSourceV2: Add AppendData logical operation

[ https://issues.apache.org/jira/browse/SPARK-24251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24251: Assignee: Apache Spark > DataSourceV2: Add AppendData logical operation >

[jira] [Assigned] (SPARK-24251) DataSourceV2: Add AppendData logical operation

[ https://issues.apache.org/jira/browse/SPARK-24251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24251: Assignee: (was: Apache Spark) > DataSourceV2: Add AppendData logical operation >

[jira] [Created] (SPARK-24251) DataSourceV2: Add AppendData logical operation

Ryan Blue created SPARK-24251: - Summary: DataSourceV2: Add AppendData logical operation Key: SPARK-24251 URL: https://issues.apache.org/jira/browse/SPARK-24251 Project: Spark Issue Type:

[jira] [Commented] (SPARK-24232) Allow referring to kubernetes secrets as env variable

[ https://issues.apache.org/jira/browse/SPARK-24232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472581#comment-16472581 ] Stavros Kontopoulos commented on SPARK-24232: - Cool makes sense. > Allow referring to

[jira] [Comment Edited] (SPARK-24232) Allow referring to kubernetes secrets as env variable

[ https://issues.apache.org/jira/browse/SPARK-24232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472572#comment-16472572 ] Stavros Kontopoulos edited comment on SPARK-24232 at 5/11/18 8:12 PM:

[jira] [Commented] (SPARK-24232) Allow referring to kubernetes secrets as env variable

[ https://issues.apache.org/jira/browse/SPARK-24232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472574#comment-16472574 ] Yinan Li commented on SPARK-24232: -- As long as we document it clearly what is for, I think it's OK,

[jira] [Comment Edited] (SPARK-24232) Allow referring to kubernetes secrets as env variable

[ https://issues.apache.org/jira/browse/SPARK-24232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472572#comment-16472572 ] Stavros Kontopoulos edited comment on SPARK-24232 at 5/11/18 8:08 PM:

[jira] [Commented] (SPARK-24232) Allow referring to kubernetes secrets as env variable

[ https://issues.apache.org/jira/browse/SPARK-24232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472572#comment-16472572 ] Stavros Kontopoulos commented on SPARK-24232: - Ok I understand that need for users not to be

[jira] [Comment Edited] (SPARK-24232) Allow referring to kubernetes secrets as env variable

[ https://issues.apache.org/jira/browse/SPARK-24232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472572#comment-16472572 ] Stavros Kontopoulos edited comment on SPARK-24232 at 5/11/18 8:07 PM:

[jira] [Comment Edited] (SPARK-24232) Allow referring to kubernetes secrets as env variable

[ https://issues.apache.org/jira/browse/SPARK-24232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472551#comment-16472551 ] Stavros Kontopoulos edited comment on SPARK-24232 at 5/11/18 7:57 PM:

[jira] [Comment Edited] (SPARK-24232) Allow referring to kubernetes secrets as env variable

[ https://issues.apache.org/jira/browse/SPARK-24232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472561#comment-16472561 ] Yinan Li edited comment on SPARK-24232 at 5/11/18 7:55 PM: --- We should keep the

[jira] [Commented] (SPARK-24232) Allow referring to kubernetes secrets as env variable

[ https://issues.apache.org/jira/browse/SPARK-24232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472561#comment-16472561 ] Yinan Li commented on SPARK-24232: -- We should keep the current semantics of

[jira] [Comment Edited] (SPARK-24232) Allow referring to kubernetes secrets as env variable

[ https://issues.apache.org/jira/browse/SPARK-24232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472551#comment-16472551 ] Stavros Kontopoulos edited comment on SPARK-24232 at 5/11/18 7:52 PM:

[jira] [Comment Edited] (SPARK-24232) Allow referring to kubernetes secrets as env variable

[ https://issues.apache.org/jira/browse/SPARK-24232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472551#comment-16472551 ] Stavros Kontopoulos edited comment on SPARK-24232 at 5/11/18 7:50 PM:

[jira] [Comment Edited] (SPARK-24232) Allow referring to kubernetes secrets as env variable

[ https://issues.apache.org/jira/browse/SPARK-24232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472551#comment-16472551 ] Stavros Kontopoulos edited comment on SPARK-24232 at 5/11/18 7:48 PM:

[jira] [Comment Edited] (SPARK-24232) Allow referring to kubernetes secrets as env variable

[ https://issues.apache.org/jira/browse/SPARK-24232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472551#comment-16472551 ] Stavros Kontopoulos edited comment on SPARK-24232 at 5/11/18 7:47 PM:

[jira] [Comment Edited] (SPARK-24232) Allow referring to kubernetes secrets as env variable

[ https://issues.apache.org/jira/browse/SPARK-24232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472551#comment-16472551 ] Stavros Kontopoulos edited comment on SPARK-24232 at 5/11/18 7:44 PM:

[jira] [Commented] (SPARK-24232) Allow referring to kubernetes secrets as env variable

[ https://issues.apache.org/jira/browse/SPARK-24232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472551#comment-16472551 ] Stavros Kontopoulos commented on SPARK-24232: - [~dharmesh.kakadia] I am working on adding

[jira] [Resolved] (SPARK-10145) Executor exit without useful messages when spark runs in spark-streaming

[ https://issues.apache.org/jira/browse/SPARK-10145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-10145. Resolution: Unresolved I'm closing this since a lot in this area has changed since this

[jira] [Commented] (SPARK-23931) High-order function: zip(array1, array2[, ...]) → array

[ https://issues.apache.org/jira/browse/SPARK-23931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472514#comment-16472514 ] Dylan Guedes commented on SPARK-23931: -- [~mn-mikke] I updated with a working version! Would you mind

[jira] [Commented] (SPARK-23852) Parquet MR bug can lead to incorrect SQL results

[ https://issues.apache.org/jira/browse/SPARK-23852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472494#comment-16472494 ] Apache Spark commented on SPARK-23852: -- User 'henryr' has created a pull request for this issue:

[jira] [Updated] (SPARK-13007) Document where configuration / properties are read and applied

[ https://issues.apache.org/jira/browse/SPARK-13007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-13007: --- Priority: Major (was: Critical) > Document where configuration / properties are read and

[jira] [Updated] (SPARK-9139) Add backwards-compatibility tests for DataType.fromJson()

[ https://issues.apache.org/jira/browse/SPARK-9139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-9139: -- Priority: Major (was: Critical) > Add backwards-compatibility tests for DataType.fromJson() >

[jira] [Updated] (SPARK-8487) Update reduceByKeyAndWindow docs to highlight that filtering Function must be used

[ https://issues.apache.org/jira/browse/SPARK-8487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-8487: -- Priority: Major (was: Critical) > Update reduceByKeyAndWindow docs to highlight that filtering

[jira] [Updated] (SPARK-3528) Reading data from file:/// should be called NODE_LOCAL not PROCESS_LOCAL

[ https://issues.apache.org/jira/browse/SPARK-3528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-3528: -- Priority: Major (was: Critical) > Reading data from file:/// should be called NODE_LOCAL not

[jira] [Updated] (SPARK-21758) `SHOW TBLPROPERTIES` can not get properties start with spark.sql.*

[ https://issues.apache.org/jira/browse/SPARK-21758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-21758: --- Priority: Major (was: Critical) > `SHOW TBLPROPERTIES` can not get properties start with

[jira] [Resolved] (SPARK-24067) Backport SPARK-17147 to 2.3 (Spark Streaming Kafka 0.10 Consumer Can't Handle Non-consecutive Offsets (i.e. Log Compaction))

[ https://issues.apache.org/jira/browse/SPARK-24067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cody Koeninger resolved SPARK-24067. Resolution: Fixed Fix Version/s: 2.3.1 Issue resolved by pull request 21300

[jira] [Updated] (SPARK-23771) Uneven Rowgroup size after repartition

[ https://issues.apache.org/jira/browse/SPARK-23771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-23771: --- Priority: Major (was: Critical) > Uneven Rowgroup size after repartition >

[jira] [Updated] (SPARK-23606) Flakey FileBasedDataSourceSuite

[ https://issues.apache.org/jira/browse/SPARK-23606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-23606: --- Priority: Major (was: Critical) > Flakey FileBasedDataSourceSuite >

[jira] [Resolved] (SPARK-24229) Upgrade to the latest Apache Thrift 0.10.0 release

[ https://issues.apache.org/jira/browse/SPARK-24229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-24229. Resolution: Not A Problem That affects the "Apache Thrift Go client library", which is not

[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472405#comment-16472405 ] Edwina Lu commented on SPARK-23206: --- The design discussion for SPARK-23206 is scheduled for Monday, May

[jira] [Commented] (SPARK-20922) Unsafe deserialization in Spark LauncherConnection

[ https://issues.apache.org/jira/browse/SPARK-20922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472393#comment-16472393 ] Marcelo Vanzin commented on SPARK-20922: You should also be able to use just the spark-launcher

[jira] [Commented] (SPARK-24233) union operation on read of dataframe does nor produce correct result

[ https://issues.apache.org/jira/browse/SPARK-24233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472326#comment-16472326 ] smohr003 commented on SPARK-24233: -- added > union operation on read of dataframe does nor produce

[jira] [Commented] (SPARK-22918) sbt test (spark - local) fail after upgrading to 2.2.1 with: java.security.AccessControlException: access denied org.apache.derby.security.SystemPermission( "engine",

[ https://issues.apache.org/jira/browse/SPARK-22918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472298#comment-16472298 ] Mihaly Toth commented on SPARK-22918: - Yep, probably anybody who introduces a SecurityManager needs

[jira] [Commented] (SPARK-21569) Internal Spark class needs to be kryo-registered

[ https://issues.apache.org/jira/browse/SPARK-21569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472295#comment-16472295 ] Ted Yu commented on SPARK-21569: What would be workaround ? Thanks > Internal Spark class needs to be

[jira] [Resolved] (SPARK-24172) we should not apply operator pushdown to data source v2 many times

[ https://issues.apache.org/jira/browse/SPARK-24172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-24172. - Resolution: Fixed Fix Version/s: 2.4.0 > we should not apply operator pushdown to data source v2

[jira] [Commented] (SPARK-20922) Unsafe deserialization in Spark LauncherConnection

[ https://issues.apache.org/jira/browse/SPARK-20922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472231#comment-16472231 ] Marcelo Vanzin commented on SPARK-20922: I think Spark 1.6 at this point is considered EOL by the

[jira] [Commented] (SPARK-24220) java.lang.NullPointerException at org.apache.spark.sql.execution.UnsafeExternalRowSorter.(UnsafeExternalRowSorter.java:83)

[ https://issues.apache.org/jira/browse/SPARK-24220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472195#comment-16472195 ] Kazuaki Ishizaki commented on SPARK-24220: -- Thank you for reporting an issue. Would it be

[jira] [Assigned] (SPARK-24228) Fix the lint error

[ https://issues.apache.org/jira/browse/SPARK-24228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24228: Assignee: Apache Spark > Fix the lint error > -- > > Key:

[jira] [Assigned] (SPARK-24228) Fix the lint error

[ https://issues.apache.org/jira/browse/SPARK-24228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24228: Assignee: (was: Apache Spark) > Fix the lint error > -- > >

[jira] [Commented] (SPARK-24228) Fix the lint error

[ https://issues.apache.org/jira/browse/SPARK-24228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472187#comment-16472187 ] Apache Spark commented on SPARK-24228: -- User 'kiszk' has created a pull request for this issue:

[jira] [Commented] (SPARK-23931) High-order function: zip(array1, array2[, ...]) → array

[ https://issues.apache.org/jira/browse/SPARK-23931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472169#comment-16472169 ] Marek Novotny commented on SPARK-23931: --- Ok. Good luck! > High-order function: zip(array1,

[jira] [Commented] (SPARK-23931) High-order function: zip(array1, array2[, ...]) → array

[ https://issues.apache.org/jira/browse/SPARK-23931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472161#comment-16472161 ] Dylan Guedes commented on SPARK-23931: -- Hi Marek! I finally get some progress, I think that more a

[jira] [Commented] (SPARK-23931) High-order function: zip(array1, array2[, ...]) → array

[ https://issues.apache.org/jira/browse/SPARK-23931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472157#comment-16472157 ] Marek Novotny commented on SPARK-23931: --- [~DylanGuedes] Any joy? I can take this one if you want.

[jira] [Resolved] (SPARK-22900) remove unnecessary restrict for streaming dynamic allocation

[ https://issues.apache.org/jira/browse/SPARK-22900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-22900. --- Resolution: Not A Problem > remove unnecessary restrict for streaming dynamic allocation >

[jira] [Resolved] (SPARK-22470) Doc that functions.hash is also used internally for shuffle and bucketing

[ https://issues.apache.org/jira/browse/SPARK-22470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-22470. --- Resolution: Won't Fix > Doc that functions.hash is also used internally for shuffle and bucketing >

[jira] [Commented] (SPARK-13158) Show the information of broadcast blocks in WebUI

[ https://issues.apache.org/jira/browse/SPARK-13158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472114#comment-16472114 ] David Moravek commented on SPARK-13158: --- Hello, is there any reason this didn't get merged (I'd

[jira] [Resolved] (SPARK-8605) Exclude files in StreamingContext. textFileStream(directory)

[ https://issues.apache.org/jira/browse/SPARK-8605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-8605. -- Resolution: Won't Fix > Exclude files in StreamingContext. textFileStream(directory) >

[jira] [Assigned] (SPARK-24067) Backport SPARK-17147 to 2.3 (Spark Streaming Kafka 0.10 Consumer Can't Handle Non-consecutive Offsets (i.e. Log Compaction))

[ https://issues.apache.org/jira/browse/SPARK-24067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24067: Assignee: Apache Spark (was: Cody Koeninger) > Backport SPARK-17147 to 2.3 (Spark

[jira] [Assigned] (SPARK-24067) Backport SPARK-17147 to 2.3 (Spark Streaming Kafka 0.10 Consumer Can't Handle Non-consecutive Offsets (i.e. Log Compaction))

[ https://issues.apache.org/jira/browse/SPARK-24067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24067: Assignee: Cody Koeninger (was: Apache Spark) > Backport SPARK-17147 to 2.3 (Spark

[jira] [Commented] (SPARK-24067) Backport SPARK-17147 to 2.3 (Spark Streaming Kafka 0.10 Consumer Can't Handle Non-consecutive Offsets (i.e. Log Compaction))

[ https://issues.apache.org/jira/browse/SPARK-24067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472071#comment-16472071 ] Apache Spark commented on SPARK-24067: -- User 'koeninger' has created a pull request for this issue:

[jira] [Commented] (SPARK-24179) History Server for Kubernetes

[ https://issues.apache.org/jira/browse/SPARK-24179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472035#comment-16472035 ] Abhishek Rao commented on SPARK-24179: -- We have brought up Spark History Server on Kubernetes using

[jira] [Updated] (SPARK-24179) History Server for Kubernetes

[ https://issues.apache.org/jira/browse/SPARK-24179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Rao updated SPARK-24179: - Attachment: Spark2_3_History_Server.PNG > History Server for Kubernetes >

[jira] [Updated] (SPARK-24179) History Server for Kubernetes

[ https://issues.apache.org/jira/browse/SPARK-24179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Rao updated SPARK-24179: - Attachment: Spark2_2_History_Server.PNG > History Server for Kubernetes >

[jira] [Assigned] (SPARK-24250) support accessing SQLConf inside tasks

[ https://issues.apache.org/jira/browse/SPARK-24250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24250: Assignee: Apache Spark (was: Wenchen Fan) > support accessing SQLConf inside tasks >

[jira] [Commented] (SPARK-24250) support accessing SQLConf inside tasks

[ https://issues.apache.org/jira/browse/SPARK-24250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471891#comment-16471891 ] Apache Spark commented on SPARK-24250: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Assigned] (SPARK-24250) support accessing SQLConf inside tasks

[ https://issues.apache.org/jira/browse/SPARK-24250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24250: Assignee: Wenchen Fan (was: Apache Spark) > support accessing SQLConf inside tasks >

[jira] [Created] (SPARK-24250) support accessing SQLConf inside tasks

Wenchen Fan created SPARK-24250: --- Summary: support accessing SQLConf inside tasks Key: SPARK-24250 URL: https://issues.apache.org/jira/browse/SPARK-24250 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-23128) A new approach to do adaptive execution in Spark SQL

[ https://issues.apache.org/jira/browse/SPARK-23128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471812#comment-16471812 ] Li Yuanjian commented on SPARK-23128: - I collected some user cases and performance improve effect

[jira] [Updated] (SPARK-23128) A new approach to do adaptive execution in Spark SQL

[ https://issues.apache.org/jira/browse/SPARK-23128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Yuanjian updated SPARK-23128: Attachment: AdaptiveExecutioninBaidu.pdf > A new approach to do adaptive execution in Spark SQL >

[jira] [Issue Comment Deleted] (SPARK-23128) A new approach to do adaptive execution in Spark SQL

[ https://issues.apache.org/jira/browse/SPARK-23128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Yuanjian updated SPARK-23128: Comment: was deleted (was: I collected some user cases and performance improve effect during Baidu

[jira] [Commented] (SPARK-23128) A new approach to do adaptive execution in Spark SQL

[ https://issues.apache.org/jira/browse/SPARK-23128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471811#comment-16471811 ] Li Yuanjian commented on SPARK-23128: - I collected some user cases and performance improve effect

[jira] [Assigned] (SPARK-24182) Improve error message for client mode when AM fails

[ https://issues.apache.org/jira/browse/SPARK-24182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-24182: --- Assignee: Marcelo Vanzin > Improve error message for client mode when AM fails >

[jira] [Resolved] (SPARK-24182) Improve error message for client mode when AM fails

[ https://issues.apache.org/jira/browse/SPARK-24182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-24182. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21243

[jira] [Commented] (SPARK-20922) Unsafe deserialization in Spark LauncherConnection

[ https://issues.apache.org/jira/browse/SPARK-20922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471600#comment-16471600 ] Ruslan Fialkovsky commented on SPARK-20922: --- I can't update Spark to 2.2. Will you make fix

[jira] [Updated] (SPARK-24249) Spark on kubernetes, pods crashes with spark sql job.

[ https://issues.apache.org/jira/browse/SPARK-24249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kaushik srinivas updated SPARK-24249: - Description: Below is the scenario being tested, Job : Spark sql job is written in

[jira] [Updated] (SPARK-24249) Spark on kubernetes, pods crashes with spark sql job.

[ https://issues.apache.org/jira/browse/SPARK-24249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kaushik srinivas updated SPARK-24249: - Description: Below is the scenario being tested, Job : Spark sql job is written in

[jira] [Updated] (SPARK-24249) Spark on kubernetes, pods crashes with spark sql job.

[ https://issues.apache.org/jira/browse/SPARK-24249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kaushik srinivas updated SPARK-24249: - Attachment: StackTrace4.txt StackTrace3.txt

[jira] [Created] (SPARK-24249) Spark on kubernetes, pods crashes with spark sql job.

kaushik srinivas created SPARK-24249: Summary: Spark on kubernetes, pods crashes with spark sql job. Key: SPARK-24249 URL: https://issues.apache.org/jira/browse/SPARK-24249 Project: Spark

[jira] [Commented] (SPARK-15784) Add Power Iteration Clustering to spark.ml

[ https://issues.apache.org/jira/browse/SPARK-15784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471545#comment-16471545 ] shahid commented on SPARK-15784: Hi [~josephkb] , I can work on it. > Add Power Iteration Clustering to

[jira] [Commented] (SPARK-21187) Complete support for remaining Spark data types in Arrow Converters

[ https://issues.apache.org/jira/browse/SPARK-21187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471519#comment-16471519 ] Eric Wohlstadter commented on SPARK-21187: -- [~bryanc] [~hyukjin.kwon] Hi Bryan,  I'm