[jira] [Commented] (SPARK-10063) Remove DirectParquetOutputCommitter

2016-10-13 Thread Chirag Vaya (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15574339#comment-15574339 ] Chirag Vaya commented on SPARK-10063: - [~mkim] Can you please tell us in what environment(Standalone

[jira] [Created] (SPARK-17932) Failed to run SQL "show table extended like table_name" in Spark2.0.0

2016-10-13 Thread pin_zhang (JIRA)
pin_zhang created SPARK-17932: - Summary: Failed to run SQL "show table extended like table_name" in Spark2.0.0 Key: SPARK-17932 URL: https://issues.apache.org/jira/browse/SPARK-17932 Project: Spark

[jira] [Commented] (SPARK-17884) In the cast expression, casting from empty string to interval type throws NullPointerException

2016-10-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15574305#comment-15574305 ] Apache Spark commented on SPARK-17884: -- User 'priyankagargnitk' has created a pull request for this

[jira] [Updated] (SPARK-17254) Filter operator should have “stop if false” semantics for sorted data

2016-10-13 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-17254: Attachment: stop-after-physical-plan.pdf > Filter operator should have “stop if false”

[jira] [Commented] (SPARK-16632) Vectorized parquet reader fails to read certain fields from Hive tables

2016-10-13 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15574251#comment-15574251 ] Dongjoon Hyun commented on SPARK-16632: --- This was backported at the following commit.

[jira] [Commented] (SPARK-16632) Vectorized parquet reader fails to read certain fields from Hive tables

2016-10-13 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15574253#comment-15574253 ] Dongjoon Hyun commented on SPARK-16632: --- {code} spark-2.0:branch-2.0$ git log --oneline | grep

[jira] [Updated] (SPARK-16632) Vectorized parquet reader fails to read certain fields from Hive tables

2016-10-13 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-16632: -- Fix Version/s: 2.0.1 > Vectorized parquet reader fails to read certain fields from Hive tables

[jira] [Resolved] (SPARK-17927) Remove dead code in WriterContainer

2016-10-13 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-17927. - Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 15477

[jira] [Commented] (SPARK-17781) datetime is serialized as double inside dapply()

2016-10-13 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15574137#comment-15574137 ] Felix Cheung commented on SPARK-17781: -- Hmm.. I'm not quite sure what it is just yet - not seeing

[jira] [Created] (SPARK-17931) taskScheduler has some unneeded serialization

2016-10-13 Thread Guoqiang Li (JIRA)
Guoqiang Li created SPARK-17931: --- Summary: taskScheduler has some unneeded serialization Key: SPARK-17931 URL: https://issues.apache.org/jira/browse/SPARK-17931 Project: Spark Issue Type:

[jira] [Created] (SPARK-17930) The SerializerInstance instance used when deserializing a TaskResult is not reused

2016-10-13 Thread Guoqiang Li (JIRA)
Guoqiang Li created SPARK-17930: --- Summary: The SerializerInstance instance used when deserializing a TaskResult is not reused Key: SPARK-17930 URL: https://issues.apache.org/jira/browse/SPARK-17930

[jira] [Updated] (SPARK-17929) Deadlock when AM restart and send RemoveExecutor on reset

2016-10-13 Thread Weizhong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weizhong updated SPARK-17929: - Summary: Deadlock when AM restart and send RemoveExecutor on reset (was: Deadlock when AM restart send

[jira] [Created] (SPARK-17929) Deadlock when AM restart send RemoveExecutor

2016-10-13 Thread Weizhong (JIRA)
Weizhong created SPARK-17929: Summary: Deadlock when AM restart send RemoveExecutor Key: SPARK-17929 URL: https://issues.apache.org/jira/browse/SPARK-17929 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-17899) add a debug mode to keep raw table properties in HiveExternalCatalog

2016-10-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573961#comment-15573961 ] Apache Spark commented on SPARK-17899: -- User 'gatorsmile' has created a pull request for this issue:

[jira] [Assigned] (SPARK-12664) Expose raw prediction scores in MultilayerPerceptronClassificationModel

2016-10-13 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yanbo Liang reassigned SPARK-12664: --- Assignee: Yanbo Liang > Expose raw prediction scores in

[jira] [Created] (SPARK-17928) No driver.memoryOverhead setting for mesos cluster mode

2016-10-13 Thread Drew Robb (JIRA)
Drew Robb created SPARK-17928: - Summary: No driver.memoryOverhead setting for mesos cluster mode Key: SPARK-17928 URL: https://issues.apache.org/jira/browse/SPARK-17928 Project: Spark Issue

[jira] [Comment Edited] (SPARK-17898) --repositories needs username and password

2016-10-13 Thread lichenglin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573929#comment-15573929 ] lichenglin edited comment on SPARK-17898 at 10/14/16 2:41 AM: -- I know it.

[jira] [Commented] (SPARK-17898) --repositories needs username and password

2016-10-13 Thread lichenglin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573929#comment-15573929 ] lichenglin commented on SPARK-17898: I know it. But how to build these dependencies into my jar.

[jira] [Commented] (SPARK-17812) More granular control of starting offsets (assign)

2016-10-13 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573843#comment-15573843 ] Cody Koeninger commented on SPARK-17812: So I think this is what we're agreed on: Mutually

[jira] [Assigned] (SPARK-17927) Remove dead code in WriterContainer

2016-10-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17927: Assignee: Apache Spark (was: Reynold Xin) > Remove dead code in WriterContainer >

[jira] [Commented] (SPARK-17927) Remove dead code in WriterContainer

2016-10-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573838#comment-15573838 ] Apache Spark commented on SPARK-17927: -- User 'rxin' has created a pull request for this issue:

[jira] [Assigned] (SPARK-17927) Remove dead code in WriterContainer

2016-10-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17927: Assignee: Reynold Xin (was: Apache Spark) > Remove dead code in WriterContainer >

[jira] [Created] (SPARK-17927) Remove dead code in WriterContainer

2016-10-13 Thread Reynold Xin (JIRA)
Reynold Xin created SPARK-17927: --- Summary: Remove dead code in WriterContainer Key: SPARK-17927 URL: https://issues.apache.org/jira/browse/SPARK-17927 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-17813) Maximum data per trigger

2016-10-13 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573806#comment-15573806 ] Cody Koeninger commented on SPARK-17813: So issues to be worked out here (assuming we're still

[jira] [Assigned] (SPARK-17812) More granular control of starting offsets (assign)

2016-10-13 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust reassigned SPARK-17812: Assignee: Cody Koeninger > More granular control of starting offsets (assign) >

[jira] [Commented] (SPARK-17926) Add methods to convert StreamingQueryStatus to json

2016-10-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573788#comment-15573788 ] Apache Spark commented on SPARK-17926: -- User 'tdas' has created a pull request for this issue:

[jira] [Assigned] (SPARK-17926) Add methods to convert StreamingQueryStatus to json

2016-10-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17926: Assignee: Apache Spark (was: Tathagata Das) > Add methods to convert

[jira] [Commented] (SPARK-17812) More granular control of starting offsets (assign)

2016-10-13 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573786#comment-15573786 ] Michael Armbrust commented on SPARK-17812: -- Please do work on it. It might be good to update

[jira] [Assigned] (SPARK-17926) Add methods to convert StreamingQueryStatus to json

2016-10-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17926: Assignee: Tathagata Das (was: Apache Spark) > Add methods to convert

[jira] [Updated] (SPARK-17926) Add methods to convert StreamingQueryStatus to json

2016-10-13 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-17926: -- Description: Useful for recording StreamingQueryStatuses when exposed through

[jira] [Created] (SPARK-17926) Add methods to convert StreamingQueryStatus to json

2016-10-13 Thread Tathagata Das (JIRA)
Tathagata Das created SPARK-17926: - Summary: Add methods to convert StreamingQueryStatus to json Key: SPARK-17926 URL: https://issues.apache.org/jira/browse/SPARK-17926 Project: Spark Issue

[jira] [Commented] (SPARK-17812) More granular control of starting offsets (assign)

2016-10-13 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573766#comment-15573766 ] Cody Koeninger commented on SPARK-17812: OK, failing on start is clear (it's really annoying in

[jira] [Resolved] (SPARK-17368) Scala value classes create encoder problems and break at runtime

2016-10-13 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-17368. -- Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 15284

[jira] [Commented] (SPARK-17916) CSV data source treats empty string as null no matter what nullValue option is

2016-10-13 Thread Hossein Falaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573692#comment-15573692 ] Hossein Falaki commented on SPARK-17916: Thanks for linking it. Yes they are very much same

[jira] [Commented] (SPARK-17812) More granular control of starting offsets (assign)

2016-10-13 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573677#comment-15573677 ] Michael Armbrust commented on SPARK-17812: -- bq. with your proposed interface, what, as a user,

[jira] [Commented] (SPARK-17916) CSV data source treats empty string as null no matter what nullValue option is

2016-10-13 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573668#comment-15573668 ] Hyukjin Kwon commented on SPARK-17916: -- Hi [~falaki], this JIRA rings a bell to me. Do you mind if I

[jira] [Assigned] (SPARK-17925) Break fileSourceInterfaces.scala into multiple pieces

2016-10-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17925: Assignee: Apache Spark (was: Reynold Xin) > Break fileSourceInterfaces.scala into

[jira] [Commented] (SPARK-17925) Break fileSourceInterfaces.scala into multiple pieces

2016-10-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573640#comment-15573640 ] Apache Spark commented on SPARK-17925: -- User 'rxin' has created a pull request for this issue:

[jira] [Assigned] (SPARK-17925) Break fileSourceInterfaces.scala into multiple pieces

2016-10-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17925: Assignee: Reynold Xin (was: Apache Spark) > Break fileSourceInterfaces.scala into

[jira] [Created] (SPARK-17925) Break fileSourceInterfaces.scala into multiple pieces

2016-10-13 Thread Reynold Xin (JIRA)
Reynold Xin created SPARK-17925: --- Summary: Break fileSourceInterfaces.scala into multiple pieces Key: SPARK-17925 URL: https://issues.apache.org/jira/browse/SPARK-17925 Project: Spark Issue

[jira] [Created] (SPARK-17924) Consolidate streaming and batch write path

2016-10-13 Thread Reynold Xin (JIRA)
Reynold Xin created SPARK-17924: --- Summary: Consolidate streaming and batch write path Key: SPARK-17924 URL: https://issues.apache.org/jira/browse/SPARK-17924 Project: Spark Issue Type:

[jira] [Updated] (SPARK-17678) Spark 1.6 Scala-2.11 repl doesn't honor "spark.replClassServer.port"

2016-10-13 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-17678: - Affects Version/s: (was: 1.6.3) 1.6.2 > Spark 1.6 Scala-2.11 repl

[jira] [Resolved] (SPARK-17678) Spark 1.6 Scala-2.11 repl doesn't honor "spark.replClassServer.port"

2016-10-13 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-17678. -- Resolution: Fixed Assignee: Saisai Shao Fix Version/s: 1.6.3 > Spark 1.6

[jira] [Commented] (SPARK-17812) More granular control of starting offsets (assign)

2016-10-13 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573563#comment-15573563 ] Cody Koeninger commented on SPARK-17812: So a short term question - with your proposed interface,

[jira] [Commented] (SPARK-17812) More granular control of starting offsets (assign)

2016-10-13 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573526#comment-15573526 ] Michael Armbrust commented on SPARK-17812: -- As far as I understand it, {{auto.offset.reset}} is

[jira] [Commented] (SPARK-17709) spark 2.0 join - column resolution error

2016-10-13 Thread Ashish Shrowty (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573512#comment-15573512 ] Ashish Shrowty commented on SPARK-17709: [~smilegator] I compiled with the added debug

[jira] [Commented] (SPARK-17812) More granular control of starting offsets (assign)

2016-10-13 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573479#comment-15573479 ] Cody Koeninger commented on SPARK-17812: While some decision is better than none, can you help me

[jira] [Commented] (SPARK-17919) Make timeout to RBackend configurable in SparkR

2016-10-13 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573469#comment-15573469 ] Felix Cheung commented on SPARK-17919: -- Earlier bug:

[jira] [Comment Edited] (SPARK-17812) More granular control of starting offsets (assign)

2016-10-13 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573459#comment-15573459 ] Michael Armbrust edited comment on SPARK-17812 at 10/13/16 10:53 PM: -

[jira] [Comment Edited] (SPARK-17812) More granular control of starting offsets (assign)

2016-10-13 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573459#comment-15573459 ] Michael Armbrust edited comment on SPARK-17812 at 10/13/16 10:53 PM: -

[jira] [Commented] (SPARK-17812) More granular control of starting offsets (assign)

2016-10-13 Thread Ofir Manor (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573457#comment-15573457 ] Ofir Manor commented on SPARK-17812: I'm with you - I warned you it is bikeshedding... I don't have a

[jira] [Commented] (SPARK-17812) More granular control of starting offsets (assign)

2016-10-13 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573459#comment-15573459 ] Michael Armbrust commented on SPARK-17812: -- +1 to the suggested was of subscribing, and for

[jira] [Comment Edited] (SPARK-17812) More granular control of starting offsets (assign)

2016-10-13 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573432#comment-15573432 ] Cody Koeninger edited comment on SPARK-17812 at 10/13/16 10:44 PM: --- If

[jira] [Commented] (SPARK-17812) More granular control of starting offsets (assign)

2016-10-13 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573432#comment-15573432 ] Cody Koeninger commented on SPARK-17812: If you're seriously worried that people are going to get

[jira] [Comment Edited] (SPARK-17812) More granular control of starting offsets (assign)

2016-10-13 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573395#comment-15573395 ] Cody Koeninger edited comment on SPARK-17812 at 10/13/16 10:25 PM: --- 1.

[jira] [Commented] (SPARK-17812) More granular control of starting offsets (assign)

2016-10-13 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573395#comment-15573395 ] Cody Koeninger commented on SPARK-17812: 1. we dont have lists, we have strings. regexes and

[jira] [Comment Edited] (SPARK-17555) ExternalShuffleBlockResolver fails randomly with External Shuffle Service and Dynamic Resource Allocation on Mesos running under Marathon

2016-10-13 Thread Eugene Zhulenev (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573363#comment-15573363 ] Eugene Zhulenev edited comment on SPARK-17555 at 10/13/16 10:21 PM:

[jira] [Commented] (SPARK-14212) Add configuration element for --packages option

2016-10-13 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573381#comment-15573381 ] holdenk commented on SPARK-14212: - Please do! I think I've outlined the basic steps in my comment above,

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

2016-10-13 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573380#comment-15573380 ] holdenk commented on SPARK-9487: +1 to [~srowen]'s comment. I would not be surprised to see some test

[jira] [Created] (SPARK-17923) dateFormat unexpected kwarg to df.write.csv

2016-10-13 Thread Evan Zamir (JIRA)
Evan Zamir created SPARK-17923: -- Summary: dateFormat unexpected kwarg to df.write.csv Key: SPARK-17923 URL: https://issues.apache.org/jira/browse/SPARK-17923 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-12916) Support Row.fromSeq and Row.toSeq methods in pyspark

2016-10-13 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573371#comment-15573371 ] holdenk commented on SPARK-12916: - +1 with Hyukjin, I'll go ahead and close this as a "Won't Fix" >

[jira] [Closed] (SPARK-12916) Support Row.fromSeq and Row.toSeq methods in pyspark

2016-10-13 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk closed SPARK-12916. --- Resolution: Won't Fix Since Row is now a subclass of Tuple we don't really need this anymore. > Support

[jira] [Commented] (SPARK-16720) Loading CSV file with 2k+ columns fails during attribute resolution on action

2016-10-13 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573367#comment-15573367 ] holdenk commented on SPARK-16720: - Sounds good - go ahead and close this :) > Loading CSV file with 2k+

[jira] [Commented] (SPARK-17555) ExternalShuffleBlockResolver fails randomly with External Shuffle Service and Dynamic Resource Allocation on Mesos running under Marathon

2016-10-13 Thread Eugene Zhulenev (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573363#comment-15573363 ] Eugene Zhulenev commented on SPARK-17555: - [~brdwrd] I had the same issue, and I figured out

[jira] [Commented] (SPARK-10972) UDFs in SQL joins

2016-10-13 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573364#comment-15573364 ] holdenk commented on SPARK-10972: - I don't think that actually solves the problem the user is looking

[jira] [Commented] (SPARK-650) Add a "setup hook" API for running initialization code on each executor

2016-10-13 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573360#comment-15573360 ] holdenk commented on SPARK-650: --- Would people feel ok if we marked this as a duplicate of 636 since it does

[jira] [Updated] (SPARK-17922) ClassCastException java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator cannot be cast to org.apache.spark.sql.cataly

2016-10-13 Thread kanika dhuria (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kanika dhuria updated SPARK-17922: -- Description: I am using spark 2.0 Seeing class loading issue because the whole stage code gen

[jira] [Updated] (SPARK-17922) ClassCastException java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator cannot be cast to org.apache.spark.sql.cataly

2016-10-13 Thread kanika dhuria (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kanika dhuria updated SPARK-17922: -- Description: I am using spark 2.0 Seeing class loading issue because the whole stage code gen

[jira] [Commented] (SPARK-17812) More granular control of starting offsets (assign)

2016-10-13 Thread Ofir Manor (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573341#comment-15573341 ] Ofir Manor commented on SPARK-17812: Thanks Cody! great to have a concrete example. I've some

[jira] [Updated] (SPARK-17922) ClassCastException java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator cannot be cast to org.apache.spark.sql.cataly

2016-10-13 Thread kanika dhuria (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kanika dhuria updated SPARK-17922: -- Description: I am using spark 2.0 Seeing class loading issue because the whole stage code gen

[jira] [Created] (SPARK-17922) ClassCastException java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator cannot be cast to org.apache.spark.sql.cataly

2016-10-13 Thread kanika dhuria (JIRA)
kanika dhuria created SPARK-17922: - Summary: ClassCastException java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator cannot be cast to org.apache.spark.sql.catalyst.expressions.UnsafeProjection

[jira] [Updated] (SPARK-17460) Dataset.joinWith broadcasts gigabyte sized table, causes OOM Exception

2016-10-13 Thread Chris Perluss (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Perluss updated SPARK-17460: -- Component/s: SQL > Dataset.joinWith broadcasts gigabyte sized table, causes OOM Exception >

[jira] [Commented] (SPARK-15369) Investigate selectively using Jython for parts of PySpark

2016-10-13 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573287#comment-15573287 ] holdenk commented on SPARK-15369: - I can understand the hesitancy to adopt this long term - I wish we

[jira] [Issue Comment Deleted] (SPARK-17812) More granular control of starting offsets (assign)

2016-10-13 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cody Koeninger updated SPARK-17812: --- Comment: was deleted (was: One other slightly ugly thing... {noformat} // starting

[jira] [Commented] (SPARK-17731) Metrics for Structured Streaming

2016-10-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573216#comment-15573216 ] Apache Spark commented on SPARK-17731: -- User 'tdas' has created a pull request for this issue:

[jira] [Comment Edited] (SPARK-17812) More granular control of starting offsets (assign)

2016-10-13 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573166#comment-15573166 ] Cody Koeninger edited comment on SPARK-17812 at 10/13/16 9:17 PM: --

[jira] [Assigned] (SPARK-17919) Make timeout to RBackend configurable in SparkR

2016-10-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17919: Assignee: (was: Apache Spark) > Make timeout to RBackend configurable in SparkR >

[jira] [Commented] (SPARK-17919) Make timeout to RBackend configurable in SparkR

2016-10-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573210#comment-15573210 ] Apache Spark commented on SPARK-17919: -- User 'falaki' has created a pull request for this issue:

[jira] [Assigned] (SPARK-17919) Make timeout to RBackend configurable in SparkR

2016-10-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17919: Assignee: Apache Spark > Make timeout to RBackend configurable in SparkR >

[jira] [Resolved] (SPARK-17661) Consolidate various listLeafFiles implementations

2016-10-13 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-17661. - Resolution: Fixed Assignee: Peter Lee Fix Version/s: 2.1.0 > Consolidate various

[jira] [Commented] (SPARK-17812) More granular control of starting offsets (assign)

2016-10-13 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573166#comment-15573166 ] Cody Koeninger commented on SPARK-17812: Here's my concrete suggestion: 3 mutually exclusive

[jira] [Assigned] (SPARK-17921) checkpointLocation being set in memory streams fail after restart. Should fail fast

2016-10-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17921: Assignee: (was: Apache Spark) > checkpointLocation being set in memory streams fail

[jira] [Assigned] (SPARK-17921) checkpointLocation being set in memory streams fail after restart. Should fail fast

2016-10-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17921: Assignee: Apache Spark > checkpointLocation being set in memory streams fail after

[jira] [Commented] (SPARK-17921) checkpointLocation being set in memory streams fail after restart. Should fail fast

2016-10-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573163#comment-15573163 ] Apache Spark commented on SPARK-17921: -- User 'brkyvz' has created a pull request for this issue:

[jira] [Created] (SPARK-17921) checkpointLocation being set in memory streams fail after restart. Should fail fast

2016-10-13 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-17921: --- Summary: checkpointLocation being set in memory streams fail after restart. Should fail fast Key: SPARK-17921 URL: https://issues.apache.org/jira/browse/SPARK-17921

[jira] [Resolved] (SPARK-17731) Metrics for Structured Streaming

2016-10-13 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-17731. --- Resolution: Fixed Fix Version/s: 2.1.0 Target Version/s: 2.0.2, 2.1.0 (was:

[jira] [Commented] (SPARK-17812) More granular control of starting offsets (assign)

2016-10-13 Thread Ofir Manor (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573109#comment-15573109 ] Ofir Manor commented on SPARK-17812: Regarding (1) - of course it is *all* data in the source, as of

[jira] [Comment Edited] (SPARK-17812) More granular control of starting offsets (assign)

2016-10-13 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572922#comment-15572922 ] Cody Koeninger edited comment on SPARK-17812 at 10/13/16 8:33 PM: --

[jira] [Resolved] (SPARK-17834) Fetch the earliest offsets manually in KafkaSource instead of counting on KafkaConsumer

2016-10-13 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-17834. -- Resolution: Fixed Fix Version/s: 2.1.0 2.0.2 > Fetch the earliest

[jira] [Assigned] (SPARK-17900) Mark the following Spark SQL APIs as stable

2016-10-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17900: Assignee: Apache Spark (was: Reynold Xin) > Mark the following Spark SQL APIs as stable

[jira] [Commented] (SPARK-17900) Mark the following Spark SQL APIs as stable

2016-10-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573090#comment-15573090 ] Apache Spark commented on SPARK-17900: -- User 'rxin' has created a pull request for this issue:

[jira] [Assigned] (SPARK-17900) Mark the following Spark SQL APIs as stable

2016-10-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17900: Assignee: Reynold Xin (was: Apache Spark) > Mark the following Spark SQL APIs as stable

[jira] [Commented] (SPARK-17812) More granular control of starting offsets (assign)

2016-10-13 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573089#comment-15573089 ] Cody Koeninger commented on SPARK-17812: One other slightly ugly thing... {noformat} // starting

[jira] [Commented] (SPARK-10872) Derby error (XSDB6) when creating new HiveContext after restarting SparkContext

2016-10-13 Thread Dmytro Bielievtsov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573056#comment-15573056 ] Dmytro Bielievtsov commented on SPARK-10872: [~sowen] Can you please give me some pointers in

[jira] [Closed] (SPARK-15369) Investigate selectively using Jython for parts of PySpark

2016-10-13 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin closed SPARK-15369. --- Resolution: Won't Fix In the spirit of having more explicitly accept/rejects, and given the

[jira] [Comment Edited] (SPARK-15565) The default value of spark.sql.warehouse.dir needs to explicitly point to local filesystem

2016-10-13 Thread Alessio (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572976#comment-15572976 ] Alessio edited comment on SPARK-15565 at 10/13/16 7:49 PM: --- Yes Sean, indeed in

[jira] [Commented] (SPARK-15565) The default value of spark.sql.warehouse.dir needs to explicitly point to local filesystem

2016-10-13 Thread Alessio (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572976#comment-15572976 ] Alessio commented on SPARK-15565: - Yes Sean, indeed in my latest issue SPARK-17918 I was referring to

[jira] [Updated] (SPARK-17917) Convert 'Initial job has not accepted any resources..' logWarning to a SparkListener event

2016-10-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-17917: -- Priority: Minor (was: Major) Maybe, I suppose it will be a little tricky to define what the event is

[jira] [Updated] (SPARK-17918) Default Warehouse location apparently in HDFS

2016-10-13 Thread Alessio (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alessio updated SPARK-17918: Description: It seems that the default warehouse location in Spark 2.0.1 not only points at an inexistent

[jira] [Updated] (SPARK-17920) HiveWriterContainer passes null configuration to serde.initialize, causing NullPointerException in AvroSerde when using avro.schema.url

2016-10-13 Thread James Norvell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Norvell updated SPARK-17920: -- Attachment: (was: avro.avsc) > HiveWriterContainer passes null configuration to

  1   2   3   >