[jira] [Commented] (SPARK-11248) Spark hivethriftserver is using the wrong user to while getting HDFS permissions

[ https://issues.apache.org/jira/browse/SPARK-11248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16243469#comment-16243469 ] 翟玉勇 commented on SPARK-11248: - I have similar issue in Spark 2.1.0 > Spark hivethriftserver is using the

[jira] [Assigned] (SPARK-14922) Alter Table Drop Partition Using Predicate-based Partition Spec

[ https://issues.apache.org/jira/browse/SPARK-14922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-14922: Assignee: Apache Spark > Alter Table Drop Partition Using Predicate-based Partition Spec

[jira] [Commented] (SPARK-17732) ALTER TABLE DROP PARTITION should support comparators

[ https://issues.apache.org/jira/browse/SPARK-17732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16243489#comment-16243489 ] Apache Spark commented on SPARK-17732: -- User 'DazhuangSu' has created a pull request for this issue:

[jira] [Assigned] (SPARK-22462) SQL metrics missing after foreach operation on dataframe

[ https://issues.apache.org/jira/browse/SPARK-22462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22462: Assignee: (was: Apache Spark) > SQL metrics missing after foreach operation on

[jira] [Assigned] (SPARK-22462) SQL metrics missing after foreach operation on dataframe

[ https://issues.apache.org/jira/browse/SPARK-22462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22462: Assignee: Apache Spark > SQL metrics missing after foreach operation on dataframe >

[jira] [Commented] (SPARK-22462) SQL metrics missing after foreach operation on dataframe

[ https://issues.apache.org/jira/browse/SPARK-22462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16243349#comment-16243349 ] Apache Spark commented on SPARK-22462: -- User 'viirya' has created a pull request for this issue:

[jira] [Assigned] (SPARK-20646) Make Executors page use new app state store

[ https://issues.apache.org/jira/browse/SPARK-20646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-20646: Assignee: Marcelo Vanzin > Make Executors page use new app state store >

[jira] [Resolved] (SPARK-22327) R CRAN check fails on non-latest branches

[ https://issues.apache.org/jira/browse/SPARK-22327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung resolved SPARK-22327. -- Resolution: Fixed Assignee: Felix Cheung Fix Version/s: 2.3.0

[jira] [Resolved] (SPARK-20646) Make Executors page use new app state store

[ https://issues.apache.org/jira/browse/SPARK-20646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-20646. -- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19678

[jira] [Assigned] (SPARK-22466) SPARK_CONF_DIR is not is set by Spark's launch scripts with default value

[ https://issues.apache.org/jira/browse/SPARK-22466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22466: Assignee: (was: Apache Spark) > SPARK_CONF_DIR is not is set by Spark's launch

[jira] [Commented] (SPARK-22466) SPARK_CONF_DIR is not is set by Spark's launch scripts with default value

[ https://issues.apache.org/jira/browse/SPARK-22466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16243294#comment-16243294 ] Apache Spark commented on SPARK-22466: -- User 'yaooqinn' has created a pull request for this issue:

[jira] [Commented] (SPARK-4502) Spark SQL reads unneccesary nested fields from Parquet

[ https://issues.apache.org/jira/browse/SPARK-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16243296#comment-16243296 ] Gesly George commented on SPARK-4502: - Will this make it to the 2.3.0 release? > Spark SQL reads

[jira] [Assigned] (SPARK-22466) SPARK_CONF_DIR is not is set by Spark's launch scripts with default value

[ https://issues.apache.org/jira/browse/SPARK-22466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22466: Assignee: Apache Spark > SPARK_CONF_DIR is not is set by Spark's launch scripts with

[jira] [Commented] (SPARK-19644) Memory leak in Spark Streaming (Encoder/Scala Reflection)

[ https://issues.apache.org/jira/browse/SPARK-19644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16243171#comment-16243171 ] Apache Spark commented on SPARK-19644: -- User 'zsxwing' has created a pull request for this issue:

[jira] [Assigned] (SPARK-19644) Memory leak in Spark Streaming (Encoder/Scala Reflection)

[ https://issues.apache.org/jira/browse/SPARK-19644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19644: Assignee: Apache Spark > Memory leak in Spark Streaming (Encoder/Scala Reflection) >

[jira] [Assigned] (SPARK-19644) Memory leak in Spark Streaming (Encoder/Scala Reflection)

[ https://issues.apache.org/jira/browse/SPARK-19644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19644: Assignee: (was: Apache Spark) > Memory leak in Spark Streaming (Encoder/Scala

[jira] [Created] (SPARK-22466) SPARK_CONF_DIR is not is set by Spark's launch scripts with default value

Kent Yao created SPARK-22466: Summary: SPARK_CONF_DIR is not is set by Spark's launch scripts with default value Key: SPARK-22466 URL: https://issues.apache.org/jira/browse/SPARK-22466 Project: Spark

[jira] [Resolved] (SPARK-22209) PySpark does not recognize imports from submodules

[ https://issues.apache.org/jira/browse/SPARK-22209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler resolved SPARK-22209. -- Resolution: Fixed Fix Version/s: 2.3.0 Resolving this as fixed upstream by SPARK-21753,

[jira] [Updated] (SPARK-22324) Upgrade Arrow to version 0.8.0

[ https://issues.apache.org/jira/browse/SPARK-22324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler updated SPARK-22324: - Description: Arrow version 0.8.0 is slated for release in early November, but I'd like to start

[jira] [Resolved] (SPARK-20645) Make Environment page use new app state store

[ https://issues.apache.org/jira/browse/SPARK-20645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-20645. -- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19677

[jira] [Resolved] (SPARK-10367) Support Parquet logical type INTERVAL

[ https://issues.apache.org/jira/browse/SPARK-10367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-10367. - Resolution: Later > Support Parquet logical type INTERVAL >

[jira] [Commented] (SPARK-10367) Support Parquet logical type INTERVAL

[ https://issues.apache.org/jira/browse/SPARK-10367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242997#comment-16242997 ] Wenchen Fan commented on SPARK-10367: - The interval type in Parquet doesn't follow SQL standard, we

[jira] [Assigned] (SPARK-20645) Make Environment page use new app state store

[ https://issues.apache.org/jira/browse/SPARK-20645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-20645: Assignee: Marcelo Vanzin > Make Environment page use new app state store >

[jira] [Updated] (SPARK-22417) createDataFrame from a pandas.DataFrame reads datetime64 values as longs

[ https://issues.apache.org/jira/browse/SPARK-22417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-22417: Fix Version/s: 2.2.1 > createDataFrame from a pandas.DataFrame reads datetime64 values as longs >

[jira] [Resolved] (SPARK-22417) createDataFrame from a pandas.DataFrame reads datetime64 values as longs

[ https://issues.apache.org/jira/browse/SPARK-22417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-22417. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19646

[jira] [Assigned] (SPARK-22417) createDataFrame from a pandas.DataFrame reads datetime64 values as longs

[ https://issues.apache.org/jira/browse/SPARK-22417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-22417: --- Assignee: Bryan Cutler > createDataFrame from a pandas.DataFrame reads datetime64 values as

[jira] [Resolved] (SPARK-22464) <=> is not supported by Hive metastore partition predicate pushdown

[ https://issues.apache.org/jira/browse/SPARK-22464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-22464. - Resolution: Fixed Fix Version/s: 2.3.0 > <=> is not supported by Hive metastore partition

[jira] [Commented] (SPARK-22465) Cogroup of two disproportionate RDDs could lead into 2G limit BUG

[ https://issues.apache.org/jira/browse/SPARK-22465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242856#comment-16242856 ] Sean Owen commented on SPARK-22465: --- Is this not indeed just the 2G limit again? You can work around

[jira] [Comment Edited] (SPARK-20226) Call to sqlContext.cacheTable takes an incredibly long time in some cases

[ https://issues.apache.org/jira/browse/SPARK-20226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16064972#comment-16064972 ] Barry Becker edited comment on SPARK-20226 at 11/7/17 6:09 PM: --- Calling

[jira] [Created] (SPARK-22465) Cogroup of two disproportionate RDDs could lead into 2G limit BUG

Amit Kumar created SPARK-22465: -- Summary: Cogroup of two disproportionate RDDs could lead into 2G limit BUG Key: SPARK-22465 URL: https://issues.apache.org/jira/browse/SPARK-22465 Project: Spark

[jira] [Commented] (SPARK-19680) Offsets out of range with no configured reset policy for partitions

[ https://issues.apache.org/jira/browse/SPARK-19680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242438#comment-16242438 ] Cody Koeninger commented on SPARK-19680: If you got an OffsetOutOfRangeException after a job had

[jira] [Assigned] (SPARK-19759) ALSModel.predict on Dataframes : potential optimization by not using blas

[ https://issues.apache.org/jira/browse/SPARK-19759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19759: Assignee: (was: Apache Spark) > ALSModel.predict on Dataframes : potential

[jira] [Assigned] (SPARK-19759) ALSModel.predict on Dataframes : potential optimization by not using blas

[ https://issues.apache.org/jira/browse/SPARK-19759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19759: Assignee: Apache Spark > ALSModel.predict on Dataframes : potential optimization by not

[jira] [Commented] (SPARK-19759) ALSModel.predict on Dataframes : potential optimization by not using blas

[ https://issues.apache.org/jira/browse/SPARK-19759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242359#comment-16242359 ] Apache Spark commented on SPARK-19759: -- User 'mgaido91' has created a pull request for this issue:

[jira] [Commented] (SPARK-19759) ALSModel.predict on Dataframes : potential optimization by not using blas

[ https://issues.apache.org/jira/browse/SPARK-19759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242314#comment-16242314 ] Marco Gaido commented on SPARK-19759: - I tried comparing the current implementation with an easy for

[jira] [Commented] (SPARK-12297) Add work-around for Parquet/Hive int96 timestamp bug.

[ https://issues.apache.org/jira/browse/SPARK-12297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242261#comment-16242261 ] Zoltan Ivanfi commented on SPARK-12297: --- Yes, we reverted that change, because without a

[jira] [Commented] (SPARK-12297) Add work-around for Parquet/Hive int96 timestamp bug.

[ https://issues.apache.org/jira/browse/SPARK-12297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242232#comment-16242232 ] Wenchen Fan commented on SPARK-12297: - seems the parquet.mr.int96.write.zone stuff is reverted from

[jira] [Commented] (SPARK-22460) Spark De-serialization of Timestamp field is Incorrect

[ https://issues.apache.org/jira/browse/SPARK-22460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242135#comment-16242135 ] Marco Gaido commented on SPARK-22460: - [~saniyat...@gmail.com] I did the same things using both

[jira] [Reopened] (SPARK-22460) Spark De-serialization of Timestamp field is Incorrect

[ https://issues.apache.org/jira/browse/SPARK-22460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saniya Tech reopened SPARK-22460: - See my last comment. > Spark De-serialization of Timestamp field is Incorrect >

[jira] [Commented] (SPARK-22460) Spark De-serialization of Timestamp field is Incorrect

[ https://issues.apache.org/jira/browse/SPARK-22460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242109#comment-16242109 ] Saniya Tech commented on SPARK-22460: - No it is not a spark-avro issue. The spark-avro serializes and

[jira] [Commented] (SPARK-22461) Move Spark ML model summaries into a dedicated package

[ https://issues.apache.org/jira/browse/SPARK-22461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241980#comment-16241980 ] Apache Spark commented on SPARK-22461: -- User 'mgaido91' has created a pull request for this issue:

[jira] [Assigned] (SPARK-22461) Move Spark ML model summaries into a dedicated package

[ https://issues.apache.org/jira/browse/SPARK-22461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22461: Assignee: Apache Spark > Move Spark ML model summaries into a dedicated package >

[jira] [Assigned] (SPARK-22461) Move Spark ML model summaries into a dedicated package

[ https://issues.apache.org/jira/browse/SPARK-22461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22461: Assignee: (was: Apache Spark) > Move Spark ML model summaries into a dedicated

[jira] [Assigned] (SPARK-21657) Spark has exponential time complexity to explode(array of structs)

[ https://issues.apache.org/jira/browse/SPARK-21657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21657: Assignee: Apache Spark > Spark has exponential time complexity to explode(array of

[jira] [Assigned] (SPARK-21657) Spark has exponential time complexity to explode(array of structs)

[ https://issues.apache.org/jira/browse/SPARK-21657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21657: Assignee: (was: Apache Spark) > Spark has exponential time complexity to

[jira] [Commented] (SPARK-21657) Spark has exponential time complexity to explode(array of structs)

[ https://issues.apache.org/jira/browse/SPARK-21657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241887#comment-16241887 ] Ohad Raviv commented on SPARK-21657: Hi, I created a pull request:

[jira] [Commented] (SPARK-21657) Spark has exponential time complexity to explode(array of structs)

[ https://issues.apache.org/jira/browse/SPARK-21657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241883#comment-16241883 ] Apache Spark commented on SPARK-21657: -- User 'uzadude' has created a pull request for this issue:

[jira] [Updated] (SPARK-16996) Hive ACID delta files not seen

[ https://issues.apache.org/jira/browse/SPARK-16996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-16996: -- Affects Version/s: 1.6.3 2.1.2 2.2.0 > Hive ACID

[jira] [Commented] (SPARK-22453) Zookeeper configuration for MesosClusterDispatcher is not documented

[ https://issues.apache.org/jira/browse/SPARK-22453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241821#comment-16241821 ] Jais Sebastian commented on SPARK-22453: Hi Sean, I dont have the configuration steps for

[jira] [Assigned] (SPARK-22376) run-tests.py fails at exec-sbt if run with Python 3

[ https://issues.apache.org/jira/browse/SPARK-22376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-22376: Assignee: Hyukjin Kwon > run-tests.py fails at exec-sbt if run with Python 3 >

[jira] [Resolved] (SPARK-22376) run-tests.py fails at exec-sbt if run with Python 3

[ https://issues.apache.org/jira/browse/SPARK-22376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-22376. -- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19665

[jira] [Resolved] (SPARK-22460) Spark De-serialization of Timestamp field is Incorrect

[ https://issues.apache.org/jira/browse/SPARK-22460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido resolved SPARK-22460. - Resolution: Invalid > Spark De-serialization of Timestamp field is Incorrect >

[jira] [Commented] (SPARK-22460) Spark De-serialization of Timestamp field is Incorrect

[ https://issues.apache.org/jira/browse/SPARK-22460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241806#comment-16241806 ] Marco Gaido commented on SPARK-22460: - this is not a Spark issue, but this is a problem of the

[jira] [Assigned] (SPARK-22024) [pySpark] Speeding up internal conversion for Spark SQL

[ https://issues.apache.org/jira/browse/SPARK-22024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-22024: Assignee: Maciej Bryński > [pySpark] Speeding up internal conversion for Spark SQL >

[jira] [Resolved] (SPARK-22024) [pySpark] Speeding up internal conversion for Spark SQL

[ https://issues.apache.org/jira/browse/SPARK-22024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-22024. -- Resolution: Done > [pySpark] Speeding up internal conversion for Spark SQL >

[jira] [Resolved] (SPARK-22010) Slow fromInternal conversion for TimestampType

[ https://issues.apache.org/jira/browse/SPARK-22010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-22010. -- Resolution: Won't Fix > Slow fromInternal conversion for TimestampType >

[jira] [Resolved] (SPARK-22029) Cache of _parse_datatype_json_string function

[ https://issues.apache.org/jira/browse/SPARK-22029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-22029. -- Resolution: Won't Fix > Cache of _parse_datatype_json_string function >

[jira] [Resolved] (SPARK-22454) ExternalShuffleClient.close() should check null

[ https://issues.apache.org/jira/browse/SPARK-22454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-22454. --- Resolution: Fixed Assignee: Yuming Wang Fix Version/s: 2.3.0 Resolved by

[jira] [Resolved] (SPARK-22436) New function strip() to remove all whitespace from string

[ https://issues.apache.org/jira/browse/SPARK-22436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-22436. --- Resolution: Won't Fix > New function strip() to remove all whitespace from string >

[jira] [Updated] (SPARK-22453) Zookeeper configuration for MesosClusterDispatcher is not documented

[ https://issues.apache.org/jira/browse/SPARK-22453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-22453: -- Priority: Minor (was: Critical) Feel free to submit a doc update > Zookeeper configuration for