[jira] [Comment Edited] (SPARK-18974) FileInputDStream could not detected files which moved to the directory

[ https://issues.apache.org/jira/browse/SPARK-18974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15782284#comment-15782284 ] Adam Wang edited comment on SPARK-18974 at 12/28/16 7:31 AM: - Thanks for

[jira] [Commented] (SPARK-18974) FileInputDStream could not detected files which moved to the directory

[ https://issues.apache.org/jira/browse/SPARK-18974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15782284#comment-15782284 ] Adam Wang commented on SPARK-18974: --- Thanks for remind, this is my fault. This bug seems not so easy to

[jira] [Comment Edited] (SPARK-18974) FileInputDStream could not detected files which moved to the directory

[ https://issues.apache.org/jira/browse/SPARK-18974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15782284#comment-15782284 ] Adam Wang edited comment on SPARK-18974 at 12/28/16 7:30 AM: - Thanks for

[jira] [Updated] (SPARK-19002) Check pep8 against all the python scripts

[ https://issues.apache.org/jira/browse/SPARK-19002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-19002: - Summary: Check pep8 against all the python scripts (was: Check pep8 against dev/*.py scripts)

[jira] [Updated] (SPARK-19002) Check pep8 against all the python scripts

[ https://issues.apache.org/jira/browse/SPARK-19002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-19002: - Description: We can check pep8 against all the python scripts. There are only python scripts

[jira] [Commented] (SPARK-19002) Check pep8 against dev/*.py scripts

[ https://issues.apache.org/jira/browse/SPARK-19002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15782211#comment-15782211 ] Hyukjin Kwon commented on SPARK-19002: -- I changed the priority to major because it seems some

[jira] [Updated] (SPARK-19002) Check pep8 against dev/*.py scripts

[ https://issues.apache.org/jira/browse/SPARK-19002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-19002: - Priority: Major (was: Trivial) > Check pep8 against dev/*.py scripts >

[jira] [Resolved] (SPARK-18931) Create empty staging directory in partitioned table on insert

[ https://issues.apache.org/jira/browse/SPARK-18931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-18931. - Resolution: Duplicate > Create empty staging directory in partitioned table on insert >

[jira] [Commented] (SPARK-18931) Create empty staging directory in partitioned table on insert

[ https://issues.apache.org/jira/browse/SPARK-18931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15782017#comment-15782017 ] Xiao Li commented on SPARK-18931: - Please try it using the latest nightly build:

[jira] [Commented] (SPARK-16563) Repeat calling Spark SQL thrift server fetchResults return empty for ExecuteStatement operation

[ https://issues.apache.org/jira/browse/SPARK-16563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15782003#comment-15782003 ] vishal agrawal commented on SPARK-16563: Fix for this issue is causing Thrift server to hang

[jira] [Resolved] (SPARK-18992) Move spark.sql.hive.thriftServer.singleSession to SQLConf

[ https://issues.apache.org/jira/browse/SPARK-18992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-18992. - Resolution: Fixed Fix Version/s: 2.2.0 > Move spark.sql.hive.thriftServer.singleSession

[jira] [Comment Edited] (SPARK-18737) Serialization setting "spark.serializer" ignored in Spark 2.x

[ https://issues.apache.org/jira/browse/SPARK-18737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15781450#comment-15781450 ] Josh Bacon edited comment on SPARK-18737 at 12/27/16 11:16 PM: --- Thanks for

[jira] [Commented] (SPARK-18393) DataFrame pivot output column names should respect aliases

[ https://issues.apache.org/jira/browse/SPARK-18393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15781473#comment-15781473 ] Andrew Ray commented on SPARK-18393: It wouldn't hurt to backport to 2.0, its a pretty simple fix. >

[jira] [Commented] (SPARK-18737) Serialization setting "spark.serializer" ignored in Spark 2.x

[ https://issues.apache.org/jira/browse/SPARK-18737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15781450#comment-15781450 ] Josh Bacon commented on SPARK-18737: Thanks for the quick reply, We've cut our code down to a

[jira] [Updated] (SPARK-17807) Scalatest listed as compile dependency in spark-tags

[ https://issues.apache.org/jira/browse/SPARK-17807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-17807: -- Assignee: Ryan Williams > Scalatest listed as compile dependency in spark-tags >

[jira] [Updated] (SPARK-19006) should mentioned the max value allowed for spark.kryoserializer.buffer.max in doc

[ https://issues.apache.org/jira/browse/SPARK-19006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-19006: -- Assignee: Yuexin Zhang > should mentioned the max value allowed for spark.kryoserializer.buffer.max in

[jira] [Commented] (SPARK-18393) DataFrame pivot output column names should respect aliases

[ https://issues.apache.org/jira/browse/SPARK-18393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15781436#comment-15781436 ] Hyukjin Kwon commented on SPARK-18393: -- Yup, I checked this against the current master. Hi [~a1ray],

[jira] [Resolved] (SPARK-18737) Serialization setting "spark.serializer" ignored in Spark 2.x

[ https://issues.apache.org/jira/browse/SPARK-18737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-18737. --- Resolution: Not A Problem Yes, see the message right above this. Spark 2.x does not necessarily work

[jira] [Comment Edited] (SPARK-18737) Serialization setting "spark.serializer" ignored in Spark 2.x

[ https://issues.apache.org/jira/browse/SPARK-18737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15781418#comment-15781418 ] Josh Bacon edited comment on SPARK-18737 at 12/27/16 10:33 PM: --- My team is

[jira] [Commented] (SPARK-18737) Serialization setting "spark.serializer" ignored in Spark 2.x

[ https://issues.apache.org/jira/browse/SPARK-18737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15781418#comment-15781418 ] Josh Bacon commented on SPARK-18737: My team is experiencing the exact same issue as described by OP

[jira] [Comment Edited] (SPARK-18737) Serialization setting "spark.serializer" ignored in Spark 2.x

[ https://issues.apache.org/jira/browse/SPARK-18737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15781418#comment-15781418 ] Josh Bacon edited comment on SPARK-18737 at 12/27/16 10:32 PM: --- My team is

[jira] [Commented] (SPARK-18393) DataFrame pivot output column names should respect aliases

[ https://issues.apache.org/jira/browse/SPARK-18393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15781324#comment-15781324 ] Ganesh Chand commented on SPARK-18393: -- [~hyukjin.kwon] - Can you confirm the Spark version? I get

[jira] [Commented] (SPARK-18993) Unable to build/compile Spark in IntelliJ due to missing Scala deps in spark-tags

[ https://issues.apache.org/jira/browse/SPARK-18993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15781302#comment-15781302 ] Apache Spark commented on SPARK-18993: -- User 'srowen' has created a pull request for this issue:

[jira] [Updated] (SPARK-18993) Unable to build/compile Spark in IntelliJ due to missing Scala deps in spark-tags

[ https://issues.apache.org/jira/browse/SPARK-18993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-18993: -- Summary: Unable to build/compile Spark in IntelliJ due to missing Scala deps in spark-tags (was:

[jira] [Commented] (SPARK-18974) FileInputDStream could not detected files which moved to the directory

[ https://issues.apache.org/jira/browse/SPARK-18974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15781279#comment-15781279 ] Shixiong Zhu commented on SPARK-18974: -- Not sure about HDFS. But I just checked my local file system

[jira] [Updated] (SPARK-18974) FileInputDStream could not detected files which moved to the directory

[ https://issues.apache.org/jira/browse/SPARK-18974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18974: - Component/s: (was: Input/Output) > FileInputDStream could not detected files which moved to

[jira] [Updated] (SPARK-19007) Speedup and optimize the GradientBoostedTrees in the "data>memory" scene

[ https://issues.apache.org/jira/browse/SPARK-19007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-19007: -- Affects Version/s: (was: 1.6.3) (was: 1.6.2)

[jira] [Resolved] (SPARK-19006) should mentioned the max value allowed for spark.kryoserializer.buffer.max in doc

[ https://issues.apache.org/jira/browse/SPARK-19006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-19006. --- Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 16412

[jira] [Comment Edited] (SPARK-18970) FileSource failure during file list refresh doesn't cause an application to fail, but stops further processing

[ https://issues.apache.org/jira/browse/SPARK-18970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15771583#comment-15771583 ] Lev edited comment on SPARK-18970 at 12/27/16 8:15 PM: --- Actually this is exactly

[jira] [Updated] (SPARK-19006) should mentioned the max value allowed for spark.kryoserializer.buffer.max in doc

[ https://issues.apache.org/jira/browse/SPARK-19006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-19006: -- Priority: Trivial (was: Major) (Not "Major") > should mentioned the max value allowed for

[jira] [Updated] (SPARK-18997) Recommended upgrade libthrift to 0.9.3

[ https://issues.apache.org/jira/browse/SPARK-18997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-18997: -- Priority: Minor (was: Critical) > Recommended upgrade libthrift to 0.9.3 >

[jira] [Commented] (SPARK-18997) Recommended upgrade libthrift to 0.9.3

[ https://issues.apache.org/jira/browse/SPARK-18997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15781139#comment-15781139 ] Sean Owen commented on SPARK-18997: --- So, the questions are: does this CVE impact Spark at all? and what

[jira] [Updated] (SPARK-18997) Recommended upgrade libthrift to 0.9.3

[ https://issues.apache.org/jira/browse/SPARK-18997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-18997: -- Issue Type: Improvement (was: Bug) > Recommended upgrade libthrift to 0.9.3 >

[jira] [Commented] (SPARK-18757) Models in Pyspark support column setters

[ https://issues.apache.org/jira/browse/SPARK-18757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15781096#comment-15781096 ] Joseph K. Bradley commented on SPARK-18757: --- I think it's useful to make the Python types match

[jira] [Updated] (SPARK-18757) Models in Pyspark support column setters

[ https://issues.apache.org/jira/browse/SPARK-18757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-18757: -- Description: Recently, I found three places in which column setters are missing:

[jira] [Resolved] (SPARK-18842) De-duplicate paths in classpaths in processes for local-cluster mode to work around the length limitation on Windows

[ https://issues.apache.org/jira/browse/SPARK-18842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-18842. --- Resolution: Fixed Issue resolved by pull request 16398 [https://github.com/apache/spark/pull/16398]

[jira] [Commented] (SPARK-19013) java.util.ConcurrentModificationException when using s3 path as checkpointLocation

[ https://issues.apache.org/jira/browse/SPARK-19013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15781029#comment-15781029 ] Tim Chan commented on SPARK-19013: -- Perhaps the documentation should be revised to recommend against

[jira] [Updated] (SPARK-19013) java.util.ConcurrentModificationException when using s3 path as checkpointLocation

[ https://issues.apache.org/jira/browse/SPARK-19013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Chan updated SPARK-19013: - Description: I have a structured stream job running on EMR. The job will fail due to this {code}

[jira] [Resolved] (SPARK-19013) java.util.ConcurrentModificationException when using s3 path as checkpointLocation

[ https://issues.apache.org/jira/browse/SPARK-19013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19013. -- Resolution: Not A Problem This is basically a S3 problem, not a Spark one. >

[jira] [Commented] (SPARK-19013) java.util.ConcurrentModificationException when using s3 path as checkpointLocation

[ https://issues.apache.org/jira/browse/SPARK-19013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15781019#comment-15781019 ] Shixiong Zhu commented on SPARK-19013: -- This is probably because S3 negative cache. "a negative

[jira] [Assigned] (SPARK-18990) make DatasetBenchmark fairer for Dataset

[ https://issues.apache.org/jira/browse/SPARK-18990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18990: Assignee: Wenchen Fan (was: Apache Spark) > make DatasetBenchmark fairer for Dataset >

[jira] [Assigned] (SPARK-18990) make DatasetBenchmark fairer for Dataset

[ https://issues.apache.org/jira/browse/SPARK-18990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18990: Assignee: Apache Spark (was: Wenchen Fan) > make DatasetBenchmark fairer for Dataset >

[jira] [Updated] (SPARK-18990) make DatasetBenchmark fairer for Dataset

[ https://issues.apache.org/jira/browse/SPARK-18990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated SPARK-18990: - Fix Version/s: (was: 2.2.0) > make DatasetBenchmark fairer for Dataset >

[jira] [Reopened] (SPARK-18990) make DatasetBenchmark fairer for Dataset

[ https://issues.apache.org/jira/browse/SPARK-18990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai reopened SPARK-18990: -- > make DatasetBenchmark fairer for Dataset > > >

[jira] [Created] (SPARK-19013) java.util.ConcurrentModificationException when using s3 path as checkpointLocation

Tim Chan created SPARK-19013: Summary: java.util.ConcurrentModificationException when using s3 path as checkpointLocation Key: SPARK-19013 URL: https://issues.apache.org/jira/browse/SPARK-19013 Project:

[jira] [Commented] (SPARK-19012) CreateOrReplaceTempView throws org.apache.spark.sql.catalyst.parser.ParseException when viewName first char is numerical

[ https://issues.apache.org/jira/browse/SPARK-19012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15780545#comment-15780545 ] Jork Zijlstra commented on SPARK-19012: --- The type is viewName: String so you would expect any

[jira] [Resolved] (SPARK-18990) make DatasetBenchmark fairer for Dataset

[ https://issues.apache.org/jira/browse/SPARK-18990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-18990. - Resolution: Fixed Fix Version/s: 2.2.0 > make DatasetBenchmark fairer for Dataset >

[jira] [Updated] (SPARK-19004) Fix `JDBCWriteSuite.testH2Dialect` by removing `getCatalystType`

[ https://issues.apache.org/jira/browse/SPARK-19004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-19004: Assignee: Dongjoon Hyun > Fix `JDBCWriteSuite.testH2Dialect` by removing `getCatalystType` >

[jira] [Resolved] (SPARK-19004) Fix `JDBCWriteSuite.testH2Dialect` by removing `getCatalystType`

[ https://issues.apache.org/jira/browse/SPARK-19004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-19004. - Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 16409

[jira] [Resolved] (SPARK-18999) simplify Literal codegen

[ https://issues.apache.org/jira/browse/SPARK-18999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-18999. - Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 16402

[jira] [Assigned] (SPARK-19010) Include Kryo exception in case of overflow

[ https://issues.apache.org/jira/browse/SPARK-19010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19010: Assignee: Apache Spark > Include Kryo exception in case of overflow >

[jira] [Assigned] (SPARK-19010) Include Kryo exception in case of overflow

[ https://issues.apache.org/jira/browse/SPARK-19010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19010: Assignee: (was: Apache Spark) > Include Kryo exception in case of overflow >

[jira] [Commented] (SPARK-19010) Include Kryo exception in case of overflow

[ https://issues.apache.org/jira/browse/SPARK-19010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15780449#comment-15780449 ] Apache Spark commented on SPARK-19010: -- User 'superbobry' has created a pull request for this issue:

[jira] [Commented] (SPARK-19012) CreateOrReplaceTempView throws org.apache.spark.sql.catalyst.parser.ParseException when viewName first char is numerical

[ https://issues.apache.org/jira/browse/SPARK-19012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15780429#comment-15780429 ] Herman van Hovell commented on SPARK-19012: --- {{"1"}} fails because it is a numeric literal.

[jira] [Updated] (SPARK-19012) CreateOrReplaceTempView throws org.apache.spark.sql.catalyst.parser.ParseException when viewName first char is numerical

[ https://issues.apache.org/jira/browse/SPARK-19012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jork Zijlstra updated SPARK-19012: -- Affects Version/s: 2.0.2 > CreateOrReplaceTempView throws >

[jira] [Updated] (SPARK-19012) CreateOrReplaceTempView throws org.apache.spark.sql.catalyst.parser.ParseException when viewName first char is numerical

[ https://issues.apache.org/jira/browse/SPARK-19012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jork Zijlstra updated SPARK-19012: -- Description: Using a viewName where the the fist char is a numerical value on

[jira] [Created] (SPARK-19012) CreateOrReplaceTempView throws org.apache.spark.sql.catalyst.parser.ParseException when viewName first char is numerical

Jork Zijlstra created SPARK-19012: - Summary: CreateOrReplaceTempView throws org.apache.spark.sql.catalyst.parser.ParseException when viewName first char is numerical Key: SPARK-19012 URL:

[jira] [Updated] (SPARK-19011) ApplicationDescription should add the Submission ID for the standalone cluster

[ https://issues.apache.org/jira/browse/SPARK-19011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hustfxj updated SPARK-19011: Description: A large standalone cluster may have logs of applications and drivers, So I may not know which

[jira] [Commented] (SPARK-19011) ApplicationDescription should add the Submission ID for the standalone cluster

[ https://issues.apache.org/jira/browse/SPARK-19011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15780310#comment-15780310 ] hustfxj commented on SPARK-19011: - A large standalone cluster may have logs of applications and drivers,

[jira] [Created] (SPARK-19011) ApplicationDescription should add the Submission ID for the standalone cluster

hustfxj created SPARK-19011: --- Summary: ApplicationDescription should add the Submission ID for the standalone cluster Key: SPARK-19011 URL: https://issues.apache.org/jira/browse/SPARK-19011 Project: Spark

[jira] [Created] (SPARK-19010) Include Kryo exception in case of overflow

Sergei Lebedev created SPARK-19010: -- Summary: Include Kryo exception in case of overflow Key: SPARK-19010 URL: https://issues.apache.org/jira/browse/SPARK-19010 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-18976) in standlone mode,executor expired by HeartbeanReceiver that still take up cores but no tasks assigned to

[ https://issues.apache.org/jira/browse/SPARK-18976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-18976. --- Resolution: Duplicate > in standlone mode,executor expired by HeartbeanReceiver that still take up

[jira] [Commented] (SPARK-15359) Mesos dispatcher should handle DRIVER_ABORTED status from mesosDriver.run()

[ https://issues.apache.org/jira/browse/SPARK-15359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15780014#comment-15780014 ] Jared commented on SPARK-15359: --- Hi, I also met some similar problem when running spark on mesos. For my