[jira] [Resolved] (SPARK-21415) Triage scapegoat warnings, part 1

2017-07-18 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21415. --- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18635

[jira] [Commented] (SPARK-21425) LongAccumulator, DoubleAccumulator not threadsafe

2017-07-18 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091244#comment-16091244 ] Sean Owen commented on SPARK-21425: --- [~zsxwing] if that's true, doesn't it imply that multiple threads

[jira] [Updated] (SPARK-21452) SessionState in HiveClientImpl is never closed

2017-07-18 Thread AnfengYuan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] AnfengYuan updated SPARK-21452: --- Description: When I use beeline to connect to spark thriftserver, there are two SessionState

[jira] [Updated] (SPARK-21453) Streaming kafka source (structured spark)

2017-07-18 Thread Pablo Panero (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pablo Panero updated SPARK-21453: - Description: On a streaming job using built-in kafka source and sink (over SSL), with I am

[jira] [Resolved] (SPARK-21453) Streaming kafka source (structured spark)

2017-07-18 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21453. --- Resolution: Invalid This looks like a Kafka question or issue, not a Spark JIRA. This should start

[jira] [Updated] (SPARK-21453) Streaming kafka source (structured spark)

2017-07-18 Thread Pablo Panero (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pablo Panero updated SPARK-21453: - Description: On a streaming job using built-in kafka source and sink (over SSL), with I am

[jira] [Resolved] (SPARK-21392) Unable to infer schema when loading large Parquet file

2017-07-18 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-21392. -- Resolution: Invalid Let's start this on mailing list. I think we are lost around here about

[jira] [Created] (SPARK-21452) SessionState in HiveClientImpl is never closed

2017-07-18 Thread AnfengYuan (JIRA)
AnfengYuan created SPARK-21452: -- Summary: SessionState in HiveClientImpl is never closed Key: SPARK-21452 URL: https://issues.apache.org/jira/browse/SPARK-21452 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-21453) Streaming kafka source (structured spark)

2017-07-18 Thread Pablo Panero (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pablo Panero updated SPARK-21453: - Description: On a streaming job using built-in kafka source and sink (over SSL), with I am

[jira] [Created] (SPARK-21453) Streaming kafka source (structured spark)

2017-07-18 Thread Pablo Panero (JIRA)
Pablo Panero created SPARK-21453: Summary: Streaming kafka source (structured spark) Key: SPARK-21453 URL: https://issues.apache.org/jira/browse/SPARK-21453 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-20395) Update Scala to 2.11.11 and zinc to 0.3.15

2017-07-18 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-20395. --- Resolution: Won't Fix Target Version/s: (was: 2.3.0) It looks like we can't support

[jira] [Updated] (SPARK-21454) Decimal up cast to higher scale fails while reading parquet to Dataset

2017-07-18 Thread Karim Wadie (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karim Wadie updated SPARK-21454: Description: Given a parquet file with a decimal (38,4) field. One can read it into a dataframe

[jira] [Updated] (SPARK-21452) SessionState in HiveClientImpl is never closed

2017-07-18 Thread AnfengYuan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] AnfengYuan updated SPARK-21452: --- Description: When I use beeline to connect to spark thriftserver, there are two SessionState

[jira] [Resolved] (SPARK-21398) Data on Rest end point is not updating after first fetch

2017-07-18 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21398. --- Resolution: Cannot Reproduce There's not enough detail here to reproduce or establish there's a

[jira] [Resolved] (SPARK-21424) Blockmanger has memory problems in high concurrent scenes

2017-07-18 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21424. --- Resolution: Not A Problem > Blockmanger has memory problems in high concurrent scenes >

[jira] [Commented] (SPARK-17333) Make pyspark interface friendly with static analysis

2017-07-18 Thread Assaf Mendelson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091210#comment-16091210 ] Assaf Mendelson commented on SPARK-17333: - Originally when I suggested this, I envisioned adding

[jira] [Created] (SPARK-21454) Decimal up cast to higher scale fails while reading parquet to Dataset

2017-07-18 Thread Karim Wadie (JIRA)
Karim Wadie created SPARK-21454: --- Summary: Decimal up cast to higher scale fails while reading parquet to Dataset Key: SPARK-21454 URL: https://issues.apache.org/jira/browse/SPARK-21454 Project: Spark

[jira] [Commented] (SPARK-21450) List of NA is flattened inside a SparkR struct type

2017-07-18 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091153#comment-16091153 ] Hyukjin Kwon commented on SPARK-21450: -- I have some time to test this on Scala/Python: *Python:*

[jira] [Commented] (SPARK-21450) List of NA is flattened inside a SparkR struct type

2017-07-18 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091180#comment-16091180 ] Hyukjin Kwon commented on SPARK-21450: -- [~falaki], so, if this describes {code} >

[jira] [Assigned] (SPARK-21451) HiveConf in SparkSQLCLIDriver doesn't respect spark.hadoop.some.hive.variables

2017-07-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21451: Assignee: Apache Spark > HiveConf in SparkSQLCLIDriver doesn't respect

[jira] [Assigned] (SPARK-21451) HiveConf in SparkSQLCLIDriver doesn't respect spark.hadoop.some.hive.variables

2017-07-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21451: Assignee: (was: Apache Spark) > HiveConf in SparkSQLCLIDriver doesn't respect

[jira] [Comment Edited] (SPARK-21450) List of NA is flattened inside a SparkR struct type

2017-07-18 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091180#comment-16091180 ] Hyukjin Kwon edited comment on SPARK-21450 at 7/18/17 6:35 AM: --- [~falaki],

[jira] [Commented] (SPARK-21451) HiveConf in SparkSQLCLIDriver doesn't respect spark.hadoop.some.hive.variables

2017-07-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091197#comment-16091197 ] Apache Spark commented on SPARK-21451: -- User 'yaooqinn' has created a pull request for this issue:

[jira] [Resolved] (SPARK-21450) List of NA is flattened inside a SparkR struct type

2017-07-18 Thread Hossein Falaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hossein Falaki resolved SPARK-21450. Resolution: Not A Bug > List of NA is flattened inside a SparkR struct type >

[jira] [Created] (SPARK-21451) HiveConf in SparkSQLCLIDriver doesn't respect spark.hadoop.some.hive.variables

2017-07-18 Thread Kent Yao (JIRA)
Kent Yao created SPARK-21451: Summary: HiveConf in SparkSQLCLIDriver doesn't respect spark.hadoop.some.hive.variables Key: SPARK-21451 URL: https://issues.apache.org/jira/browse/SPARK-21451 Project:

[jira] [Commented] (SPARK-19908) Direct buffer memory OOM should not cause stage retries.

2017-07-18 Thread Bojan Babic (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091168#comment-16091168 ] Bojan Babic commented on SPARK-19908: - [~skp33] got the same error while running RowMatrix transpose.

[jira] [Commented] (SPARK-21450) List of NA is flattened inside a SparkR struct type

2017-07-18 Thread Hossein Falaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091194#comment-16091194 ] Hossein Falaki commented on SPARK-21450: Thanks guys. I will close it as not an issue. > List of

[jira] [Commented] (SPARK-15703) Make ListenerBus event queue size configurable

2017-07-18 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091531#comment-16091531 ] Thomas Graves commented on SPARK-15703: --- It should not be breaking dynamic allocation, if it is,

[jira] [Commented] (SPARK-21455) RpcFailure should be call on RpcResponseCallback.onFailure

2017-07-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091529#comment-16091529 ] Apache Spark commented on SPARK-21455: -- User 'ConeyLiu' has created a pull request for this issue:

[jira] [Commented] (SPARK-18226) SparkR displaying vector columns in incorrect way

2017-07-18 Thread Kirti (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091382#comment-16091382 ] Kirti commented on SPARK-18226: --- Is there any way around to handle this issue in SparkR? > SparkR

[jira] [Commented] (SPARK-21453) Streaming kafka source (structured spark)

2017-07-18 Thread Pablo Panero (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091353#comment-16091353 ] Pablo Panero commented on SPARK-21453: -- [~srowen] could you point me to the mailing list? I checked

[jira] [Commented] (SPARK-20383) SparkSQL unsupports to create function with the keyword 'OR REPLACE' and 'IF NOT EXISTS'

2017-07-18 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091524#comment-16091524 ] Wenchen Fan commented on SPARK-20383: - This seems like an improvement for the existing CREATE

[jira] [Updated] (SPARK-21420) Support array type code 'q' and 'Q'

2017-07-18 Thread Xiang Gao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiang Gao updated SPARK-21420: -- Description: Python's array type {{q}} and {{Q}} are currently not supported by

[jira] [Updated] (SPARK-21420) Support array type code 'q' and 'Q'

2017-07-18 Thread Xiang Gao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiang Gao updated SPARK-21420: -- Description: Python's array type {{q}} and {{Q}} are currently not supported by

[jira] [Updated] (SPARK-21455) RpcFailure should be call on RpcResponseCallback.onFailure

2017-07-18 Thread Xianyang Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyang Liu updated SPARK-21455: - Description: Currently, when there is a `RpcFailure` need be sent back to client, we call

[jira] [Updated] (SPARK-21455) RpcFailure should be call on RpcResponseCallback.onFailure

2017-07-18 Thread Xianyang Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyang Liu updated SPARK-21455: - Description: Currently, when there is a `RpcFailure` need be sent back to client, we call

[jira] [Updated] (SPARK-21455) RpcFailure should be call on RpcResponseCallback.onFailure

2017-07-18 Thread Xianyang Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyang Liu updated SPARK-21455: - Description: Currently, when there is a `RpcFailure` need be sent back to client, we call

[jira] [Commented] (SPARK-20383) SparkSQL unsupports to create function with the keyword 'OR REPLACE' and 'IF NOT EXISTS'

2017-07-18 Thread Xiaochen Ouyang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091454#comment-16091454 ] Xiaochen Ouyang commented on SPARK-20383: - [~cloud_fan] Hello, whether the type of this issue can

[jira] [Created] (SPARK-21455) RpcFailure should be call on RpcResponseCallback.onFailure

2017-07-18 Thread Xianyang Liu (JIRA)
Xianyang Liu created SPARK-21455: Summary: RpcFailure should be call on RpcResponseCallback.onFailure Key: SPARK-21455 URL: https://issues.apache.org/jira/browse/SPARK-21455 Project: Spark

[jira] [Updated] (SPARK-21455) RpcFailure should be call on RpcResponseCallback.onFailure

2017-07-18 Thread Xianyang Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyang Liu updated SPARK-21455: - Description: Currently, when there is a `RpcFailure` need be sent back to client, we call

[jira] [Updated] (SPARK-21455) RpcFailure should be call on RpcResponseCallback.onFailure

2017-07-18 Thread Xianyang Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyang Liu updated SPARK-21455: - Description: Currently, when there is a `RpcFailure` need be sent back to client, we call

[jira] [Commented] (SPARK-21454) Decimal up cast to higher scale fails while reading parquet to Dataset

2017-07-18 Thread Feng Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091462#comment-16091462 ] Feng Zhu commented on SPARK-21454: -- I don't think it is an issue. The condition for Decimal A casting

[jira] [Assigned] (SPARK-21455) RpcFailure should be call on RpcResponseCallback.onFailure

2017-07-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21455: Assignee: (was: Apache Spark) > RpcFailure should be call on

[jira] [Assigned] (SPARK-21455) RpcFailure should be call on RpcResponseCallback.onFailure

2017-07-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21455: Assignee: Apache Spark > RpcFailure should be call on RpcResponseCallback.onFailure >

[jira] [Updated] (SPARK-21455) RpcFailure should be call on RpcResponseCallback.onFailure

2017-07-18 Thread Xianyang Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyang Liu updated SPARK-21455: - Description: Currently, when there is a `RpcFailure` need be sent back to client, we call

[jira] [Updated] (SPARK-21455) RpcFailure should be call on RpcResponseCallback.onFailure

2017-07-18 Thread Xianyang Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyang Liu updated SPARK-21455: - Description: Currently, when there is a `RpcFailure` need be sent back to client, we call

[jira] [Created] (SPARK-21458) Tear down the framework when failover_timeout > 0 (Mesos cluster mode)

2017-07-18 Thread Susan X. Huynh (JIRA)
Susan X. Huynh created SPARK-21458: -- Summary: Tear down the framework when failover_timeout > 0 (Mesos cluster mode) Key: SPARK-21458 URL: https://issues.apache.org/jira/browse/SPARK-21458 Project:

[jira] [Commented] (SPARK-21457) ExternalCatalog.listPartitions should correctly handle partition values with dot

2017-07-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091684#comment-16091684 ] Apache Spark commented on SPARK-21457: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Assigned] (SPARK-21457) ExternalCatalog.listPartitions should correctly handle partition values with dot

2017-07-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21457: Assignee: Apache Spark (was: Wenchen Fan) > ExternalCatalog.listPartitions should

[jira] [Assigned] (SPARK-21457) ExternalCatalog.listPartitions should correctly handle partition values with dot

2017-07-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21457: Assignee: Wenchen Fan (was: Apache Spark) > ExternalCatalog.listPartitions should

[jira] [Created] (SPARK-21456) Make the driver failover_timeout configurable (Mesos cluster mode)

2017-07-18 Thread Susan X. Huynh (JIRA)
Susan X. Huynh created SPARK-21456: -- Summary: Make the driver failover_timeout configurable (Mesos cluster mode) Key: SPARK-21456 URL: https://issues.apache.org/jira/browse/SPARK-21456 Project:

[jira] [Created] (SPARK-21457) ExternalCatalog.listPartitions should correctly handle partition values with dot

2017-07-18 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-21457: --- Summary: ExternalCatalog.listPartitions should correctly handle partition values with dot Key: SPARK-21457 URL: https://issues.apache.org/jira/browse/SPARK-21457

[jira] [Commented] (SPARK-21419) Support Mesos failover_timeout in driver (Mesos cluster mode)

2017-07-18 Thread Susan X. Huynh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091657#comment-16091657 ] Susan X. Huynh commented on SPARK-21419: I split this into two sub-tasks: (1) making the

[jira] [Resolved] (SPARK-15526) Shade JPMML

2017-07-18 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-15526. Resolution: Fixed Assignee: Sean Owen Fix Version/s: 2.3.0 > Shade JPMML >

[jira] [Created] (SPARK-21460) Spark dynamic allocation breaks when ListenerBus event queue runs full

2017-07-18 Thread Ruslan Dautkhanov (JIRA)
Ruslan Dautkhanov created SPARK-21460: - Summary: Spark dynamic allocation breaks when ListenerBus event queue runs full Key: SPARK-21460 URL: https://issues.apache.org/jira/browse/SPARK-21460

[jira] [Commented] (SPARK-15703) Make ListenerBus event queue size configurable

2017-07-18 Thread Ruslan Dautkhanov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091821#comment-16091821 ] Ruslan Dautkhanov commented on SPARK-15703: --- [~tgraves], filed SPARK-21460. Thanks. > Make

[jira] [Created] (SPARK-21459) Some aggregation functions change the case of nested field names

2017-07-18 Thread David Allsopp (JIRA)
David Allsopp created SPARK-21459: - Summary: Some aggregation functions change the case of nested field names Key: SPARK-21459 URL: https://issues.apache.org/jira/browse/SPARK-21459 Project: Spark

[jira] [Assigned] (SPARK-21447) Spark history server fails to render compressed inprogress history file in some cases.

2017-07-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21447: Assignee: Apache Spark > Spark history server fails to render compressed inprogress

[jira] [Assigned] (SPARK-21447) Spark history server fails to render compressed inprogress history file in some cases.

2017-07-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21447: Assignee: (was: Apache Spark) > Spark history server fails to render compressed

[jira] [Commented] (SPARK-21447) Spark history server fails to render compressed inprogress history file in some cases.

2017-07-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091834#comment-16091834 ] Apache Spark commented on SPARK-21447: -- User 'ericvandenbergfb' has created a pull request for this

[jira] [Commented] (SPARK-21392) Unable to infer schema when loading large Parquet file

2017-07-18 Thread Stuart Reynolds (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091854#comment-16091854 ] Stuart Reynolds commented on SPARK-21392: - Okie dokey:

[jira] [Comment Edited] (SPARK-21392) Unable to infer schema when loading large Parquet file

2017-07-18 Thread Stuart Reynolds (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090159#comment-16090159 ] Stuart Reynolds edited comment on SPARK-21392 at 7/18/17 5:30 PM: --

[jira] [Updated] (SPARK-21459) Some aggregation functions change the case of nested field names

2017-07-18 Thread David Allsopp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Allsopp updated SPARK-21459: -- Description: When working with DataFrames with nested schemas, the behavior of the

[jira] [Commented] (SPARK-21460) Spark dynamic allocation breaks when ListenerBus event queue runs full

2017-07-18 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091964#comment-16091964 ] Shixiong Zhu commented on SPARK-21460: -- [~Tagar] Right. SPARK-18838 probably will create a dedicated

[jira] [Comment Edited] (SPARK-21378) Spark Poll timeout when specific offsets are passed

2017-07-18 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16092088#comment-16092088 ] Shixiong Zhu edited comment on SPARK-21378 at 7/18/17 8:01 PM: --- The data

[jira] [Comment Edited] (SPARK-21378) Spark Poll timeout when specific offsets are passed

2017-07-18 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16092088#comment-16092088 ] Shixiong Zhu edited comment on SPARK-21378 at 7/18/17 8:00 PM: --- The data

[jira] [Commented] (SPARK-21378) Spark Poll timeout when specific offsets are passed

2017-07-18 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16092088#comment-16092088 ] Shixiong Zhu commented on SPARK-21378: -- The data must already be in Kafka when executors try to

[jira] [Created] (SPARK-21461) Spark Streaming crashes if CSV file has no read permissions

2017-07-18 Thread Dan (JIRA)
Dan created SPARK-21461: --- Summary: Spark Streaming crashes if CSV file has no read permissions Key: SPARK-21461 URL: https://issues.apache.org/jira/browse/SPARK-21461 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-21462) Add batchId to the json of StreamingQueryProgress

2017-07-18 Thread Tathagata Das (JIRA)
Tathagata Das created SPARK-21462: - Summary: Add batchId to the json of StreamingQueryProgress Key: SPARK-21462 URL: https://issues.apache.org/jira/browse/SPARK-21462 Project: Spark Issue

[jira] [Commented] (SPARK-21439) Cannot use Spark with Python ABCmeta (exception from cloudpickle)

2017-07-18 Thread Dimuthu Wickramanayake (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16092195#comment-16092195 ] Dimuthu Wickramanayake commented on SPARK-21439: Can i start work on this issue? >

[jira] [Commented] (SPARK-21460) Spark dynamic allocation breaks when ListenerBus event queue runs full

2017-07-18 Thread Ruslan Dautkhanov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091943#comment-16091943 ] Ruslan Dautkhanov commented on SPARK-21460: --- [~zsxwing], according to [~tgraves] spark dynamic

[jira] [Reopened] (SPARK-21460) Spark dynamic allocation breaks when ListenerBus event queue runs full

2017-07-18 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reopened SPARK-21460: -- > Spark dynamic allocation breaks when ListenerBus event queue runs full >

[jira] [Commented] (SPARK-21460) Spark dynamic allocation breaks when ListenerBus event queue runs full

2017-07-18 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16092034#comment-16092034 ] Shixiong Zhu commented on SPARK-21460: -- Make sense. Reopened it. > Spark dynamic allocation breaks

[jira] [Resolved] (SPARK-21408) Default RPC dispatcher thread pool size too large for small executors

2017-07-18 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-21408. Resolution: Fixed Assignee: Marcelo Vanzin Fix Version/s: 2.3.0 > Default

[jira] [Assigned] (SPARK-21463) Output of StructuredStreaming tables don't respect user specified schema when reading back the table

2017-07-18 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz reassigned SPARK-21463: --- Assignee: Burak Yavuz > Output of StructuredStreaming tables don't respect user specified

[jira] [Commented] (SPARK-21456) Make the driver failover_timeout configurable (Mesos cluster mode)

2017-07-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091967#comment-16091967 ] Apache Spark commented on SPARK-21456: -- User 'susanxhuynh' has created a pull request for this

[jira] [Assigned] (SPARK-21456) Make the driver failover_timeout configurable (Mesos cluster mode)

2017-07-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21456: Assignee: (was: Apache Spark) > Make the driver failover_timeout configurable (Mesos

[jira] [Assigned] (SPARK-21456) Make the driver failover_timeout configurable (Mesos cluster mode)

2017-07-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21456: Assignee: Apache Spark > Make the driver failover_timeout configurable (Mesos cluster

[jira] [Resolved] (SPARK-21411) Failed to get new HDFS delegation tokens in AMCredentialRenewer

2017-07-18 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-21411. Resolution: Fixed Assignee: Saisai Shao Fix Version/s: 2.3.0 > Failed to

[jira] [Resolved] (SPARK-21460) Spark dynamic allocation breaks when ListenerBus event queue runs full

2017-07-18 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-21460. -- Resolution: Duplicate This will be addressed in SPARK-18838 > Spark dynamic allocation breaks

[jira] [Assigned] (SPARK-21463) Output of StructuredStreaming tables don't respect user specified schema when reading back the table

2017-07-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21463: Assignee: Apache Spark > Output of StructuredStreaming tables don't respect user

[jira] [Commented] (SPARK-21462) Add batchId to the json of StreamingQueryProgress

2017-07-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16092157#comment-16092157 ] Apache Spark commented on SPARK-21462: -- User 'tdas' has created a pull request for this issue:

[jira] [Assigned] (SPARK-21462) Add batchId to the json of StreamingQueryProgress

2017-07-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21462: Assignee: Apache Spark (was: Tathagata Das) > Add batchId to the json of

[jira] [Commented] (SPARK-21463) Output of StructuredStreaming tables don't respect user specified schema when reading back the table

2017-07-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16092156#comment-16092156 ] Apache Spark commented on SPARK-21463: -- User 'brkyvz' has created a pull request for this issue:

[jira] [Assigned] (SPARK-21463) Output of StructuredStreaming tables don't respect user specified schema when reading back the table

2017-07-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21463: Assignee: (was: Apache Spark) > Output of StructuredStreaming tables don't respect

[jira] [Commented] (SPARK-21460) Spark dynamic allocation breaks when ListenerBus event queue runs full

2017-07-18 Thread Ruslan Dautkhanov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091972#comment-16091972 ] Ruslan Dautkhanov commented on SPARK-21460: --- [~zsxwing] can we keep this jira open?

[jira] [Comment Edited] (SPARK-21425) LongAccumulator, DoubleAccumulator not threadsafe

2017-07-18 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16092064#comment-16092064 ] Shixiong Zhu edited comment on SPARK-21425 at 7/18/17 7:41 PM: --- [~srowen]

[jira] [Assigned] (SPARK-21462) Add batchId to the json of StreamingQueryProgress

2017-07-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21462: Assignee: Tathagata Das (was: Apache Spark) > Add batchId to the json of

[jira] [Created] (SPARK-21463) Output of StructuredStreaming tables don't respect user specified schema when reading back the table

2017-07-18 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-21463: --- Summary: Output of StructuredStreaming tables don't respect user specified schema when reading back the table Key: SPARK-21463 URL:

[jira] [Updated] (SPARK-21411) Failed to get new HDFS delegation tokens in AMCredentialRenewer

2017-07-18 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-21411: --- Affects Version/s: (was: 2.2.0) 2.3.0 > Failed to get new HDFS

[jira] [Commented] (SPARK-21425) LongAccumulator, DoubleAccumulator not threadsafe

2017-07-18 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16092064#comment-16092064 ] Shixiong Zhu commented on SPARK-21425: -- [~srowen] 1. Long/DoubleAccumulator assumes that there is

[jira] [Resolved] (SPARK-21462) Add batchId to the json of StreamingQueryProgress

2017-07-18 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-21462. --- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18675

[jira] [Commented] (SPARK-21464) Minimize deprecation warnings caused by ProcessingTime class

2017-07-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16092456#comment-16092456 ] Apache Spark commented on SPARK-21464: -- User 'tdas' has created a pull request for this issue:

[jira] [Assigned] (SPARK-21464) Minimize deprecation warnings caused by ProcessingTime class

2017-07-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21464: Assignee: Tathagata Das (was: Apache Spark) > Minimize deprecation warnings caused by

[jira] [Assigned] (SPARK-21464) Minimize deprecation warnings caused by ProcessingTime class

2017-07-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21464: Assignee: Apache Spark (was: Tathagata Das) > Minimize deprecation warnings caused by

[jira] [Commented] (SPARK-21273) Decouple stats propagation from logical plan

2017-07-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16092373#comment-16092373 ] Apache Spark commented on SPARK-21273: -- User 'gatorsmile' has created a pull request for this issue:

[jira] [Created] (SPARK-21464) Minimize deprecation warnings caused by ProcessingTime class

2017-07-18 Thread Tathagata Das (JIRA)
Tathagata Das created SPARK-21464: - Summary: Minimize deprecation warnings caused by ProcessingTime class Key: SPARK-21464 URL: https://issues.apache.org/jira/browse/SPARK-21464 Project: Spark

[jira] [Resolved] (SPARK-21457) ExternalCatalog.listPartitions should correctly handle partition values with dot

2017-07-18 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-21457. - Resolution: Fixed Fix Version/s: 2.3.0 2.2.1 > ExternalCatalog.listPartitions

[jira] [Updated] (SPARK-21333) joinWith documents and analysis allow invalid join types

2017-07-18 Thread Corey Woodfield (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Corey Woodfield updated SPARK-21333: Affects Version/s: 2.2.0 > joinWith documents and analysis allow invalid join types >

[jira] [Commented] (SPARK-21439) Cannot use Spark with Python ABCmeta (exception from cloudpickle)

2017-07-18 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16092586#comment-16092586 ] Liang-Chi Hsieh commented on SPARK-21439: - If there's no pr submitted and no one claims working

  1   2   >