[jira] [Comment Edited] (SPARK-23715) from_utc_timestamp returns incorrect results for some UTC date/time values

2018-04-25 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16452790#comment-16452790 ] Bruce Robbins edited comment on SPARK-23715 at 4/25/18 10:00 PM: -

[jira] [Commented] (SPARK-23715) from_utc_timestamp returns incorrect results for some UTC date/time values

2018-04-25 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16452790#comment-16452790 ] Bruce Robbins commented on SPARK-23715: --- [~cloud_fan] I'll give separate answers for String input

[jira] [Commented] (SPARK-24043) InterpretedPredicate.eval fails if expression tree contains Nondeterministic expressions

2018-04-24 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449352#comment-16449352 ] Bruce Robbins commented on SPARK-24043: --- You're half-way there. When whole-stage codegen is off

[jira] [Commented] (SPARK-24043) InterpretedPredicate.eval fails if expression tree contains Nondeterministic expressions

2018-04-23 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449281#comment-16449281 ] Bruce Robbins commented on SPARK-24043: --- [~maropu] > Do I miss any precondition? For this bug to

[jira] [Created] (SPARK-24043) InterpretedPredicate.eval fails if expression tree contains Nondeterministic expressions

2018-04-21 Thread Bruce Robbins (JIRA)
Bruce Robbins created SPARK-24043: - Summary: InterpretedPredicate.eval fails if expression tree contains Nondeterministic expressions Key: SPARK-24043 URL: https://issues.apache.org/jira/browse/SPARK-24043

[jira] [Commented] (SPARK-23963) Queries on text-based Hive tables grow disproportionately slower as the number of columns increase

2018-04-17 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441378#comment-16441378 ] Bruce Robbins commented on SPARK-23963: --- [~Tagar] Yes, although I am a little fuzzy on the process

[jira] [Commented] (SPARK-23936) High-order function: map_concat(map1<K, V>, map2<K, V>, ..., mapN<K, V>) → map<K,V>

2018-04-13 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437985#comment-16437985 ] Bruce Robbins commented on SPARK-23936: --- I will have a WIP pull request tonight or tomorrow

[jira] [Commented] (SPARK-23936) High-order function: map_concat(map1<K, V>, map2<K, V>, ..., mapN<K, V>) → map<K,V>

2018-04-12 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16436133#comment-16436133 ] Bruce Robbins commented on SPARK-23936: --- I would like to take this one, assuming no one has taken

[jira] [Comment Edited] (SPARK-23715) from_utc_timestamp returns incorrect results for some UTC date/time values

2018-04-11 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16431403#comment-16431403 ] Bruce Robbins edited comment on SPARK-23715 at 4/11/18 8:09 PM: I've been

[jira] [Updated] (SPARK-23963) Queries on text-based Hive tables grow disproportionately slower as the number of columns increase

2018-04-11 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-23963: -- Description: TableReader gets disproportionately slower as the number of columns in the query

[jira] [Updated] (SPARK-23963) Queries on text-based Hive tables grow disproportionately slower as the number of columns increase

2018-04-11 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-23963: -- Description: TableReader gets disproportionately slower as the number of columns in the query

[jira] [Created] (SPARK-23963) Queries on text-based Hive tables grow disproportionately slower as the number of columns increase

2018-04-11 Thread Bruce Robbins (JIRA)
Bruce Robbins created SPARK-23963: - Summary: Queries on text-based Hive tables grow disproportionately slower as the number of columns increase Key: SPARK-23963 URL:

[jira] [Commented] (SPARK-23715) from_utc_timestamp returns incorrect results for some UTC date/time values

2018-04-09 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16431403#comment-16431403 ] Bruce Robbins commented on SPARK-23715: --- I've been convinced this is worth fixing, at least for

[jira] [Commented] (SPARK-23776) pyspark-sql tests should display build instructions when components are missing

2018-03-23 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16411854#comment-16411854 ] Bruce Robbins commented on SPARK-23776: --- As it turns out, the building-spark page does have maven

[jira] [Created] (SPARK-23776) pyspark-sql tests should display build instructions when components are missing

2018-03-22 Thread Bruce Robbins (JIRA)
Bruce Robbins created SPARK-23776: - Summary: pyspark-sql tests should display build instructions when components are missing Key: SPARK-23776 URL: https://issues.apache.org/jira/browse/SPARK-23776

[jira] [Updated] (SPARK-23715) from_utc_timestamp returns incorrect results for some UTC date/time values

2018-03-20 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-23715: -- Description: This produces the expected answer: {noformat}

[jira] [Commented] (SPARK-23715) from_utc_timestamp returns incorrect results for some UTC date/time values

2018-03-20 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406836#comment-16406836 ] Bruce Robbins commented on SPARK-23715: --- A fix to this requires some ugly hacking of the implicit

[jira] [Updated] (SPARK-23715) from_utc_timestamp returns incorrect results for some UTC date/time values

2018-03-20 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-23715: -- Description: This produces the expected answer: {noformat}

[jira] [Updated] (SPARK-23715) from_utc_timestamp returns incorrect results for some UTC date/time values

2018-03-20 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-23715: -- Description: This produces the expected answer: {noformat}

[jira] [Updated] (SPARK-23715) from_utc_timestamp returns incorrect results for some UTC date/time values

2018-03-20 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-23715: -- Description: This produces the expected answer: {noformat}

[jira] [Updated] (SPARK-23560) Group by on struct field can add extra shuffle

2018-03-17 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-23560: -- Summary: Group by on struct field can add extra shuffle (was: A joinWith followed by groupBy

[jira] [Updated] (SPARK-23715) from_utc_timestamp returns incorrect results for some UTC date/time values

2018-03-17 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-23715: -- Description: This produces the expected answer: {noformat}

[jira] [Updated] (SPARK-23715) from_utc_timestamp returns incorrect results for some UTC date/time values

2018-03-17 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-23715: -- Description: This produces the expected answer: {noformat}

[jira] [Updated] (SPARK-23715) from_utc_timestamp returns incorrect results for some UTC date/time values

2018-03-17 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-23715: -- Description: This produces the expected answer: {noformat}

[jira] [Commented] (SPARK-23715) from_utc_timestamp returns incorrect results for some UTC date/time values

2018-03-16 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16403183#comment-16403183 ] Bruce Robbins commented on SPARK-23715: --- It almost seems like FromUTCTimestamp needs its own Cast

[jira] [Created] (SPARK-23715) from_utc_timestamp returns incorrect results for some UTC date/time values

2018-03-16 Thread Bruce Robbins (JIRA)
Bruce Robbins created SPARK-23715: - Summary: from_utc_timestamp returns incorrect results for some UTC date/time values Key: SPARK-23715 URL: https://issues.apache.org/jira/browse/SPARK-23715

[jira] [Commented] (SPARK-23560) A joinWith followed by groupBy requires extra shuffle

2018-03-10 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16394369#comment-16394369 ] Bruce Robbins commented on SPARK-23560: --- A simpler example that seems to reproduce this issue

[jira] [Created] (SPARK-23629) Building streaming-kafka-0-8-assembly or streaming-flume-assembly adds incompatible jline jar to assembly

2018-03-08 Thread Bruce Robbins (JIRA)
Bruce Robbins created SPARK-23629: - Summary: Building streaming-kafka-0-8-assembly or streaming-flume-assembly adds incompatible jline jar to assembly Key: SPARK-23629 URL:

[jira] [Updated] (SPARK-23560) A joinWith followed by groupBy requires extra shuffle

2018-03-07 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-23560: -- Description: Depending on the size of the input, a joinWith followed by a groupBy requires

[jira] [Commented] (SPARK-23560) A joinWith followed by groupBy requires extra shuffle

2018-03-07 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390111#comment-16390111 ] Bruce Robbins commented on SPARK-23560: --- The main issue is that an AttributeReference instance

[jira] [Created] (SPARK-23560) A joinWith followed by groupBy requires extra shuffle

2018-03-01 Thread Bruce Robbins (JIRA)
Bruce Robbins created SPARK-23560: - Summary: A joinWith followed by groupBy requires extra shuffle Key: SPARK-23560 URL: https://issues.apache.org/jira/browse/SPARK-23560 Project: Spark

[jira] [Commented] (SPARK-23417) pyspark tests give wrong sbt instructions

2018-02-16 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16368031#comment-16368031 ] Bruce Robbins commented on SPARK-23417: --- This does the trick: {noformat} build/sbt -Pkafka-0-8

[jira] [Comment Edited] (SPARK-23410) Unable to read jsons in charset different from UTF-8

2018-02-14 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364929#comment-16364929 ] Bruce Robbins edited comment on SPARK-23410 at 2/14/18 11:17 PM: -

[jira] [Commented] (SPARK-23410) Unable to read jsons in charset different from UTF-8

2018-02-14 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364929#comment-16364929 ] Bruce Robbins commented on SPARK-23410: --- bq. I am working on a fix, just in case Oh, OK, this one

[jira] [Commented] (SPARK-23410) Unable to read jsons in charset different from UTF-8

2018-02-14 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364916#comment-16364916 ] Bruce Robbins commented on SPARK-23410: --- On Spark 2.2.1, I got the same result as you. But with

[jira] [Comment Edited] (SPARK-23410) Unable to read jsons in charset different from UTF-8

2018-02-14 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364866#comment-16364866 ] Bruce Robbins edited comment on SPARK-23410 at 2/14/18 10:21 PM: -

[jira] [Commented] (SPARK-23410) Unable to read jsons in charset different from UTF-8

2018-02-14 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364866#comment-16364866 ] Bruce Robbins commented on SPARK-23410: --- [~maxgekk] My simple test input of [{"field1": 10,

[jira] [Commented] (SPARK-23410) Unable to read jsons in charset different from UTF-8

2018-02-14 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364788#comment-16364788 ] Bruce Robbins commented on SPARK-23410: --- I am probably misunderstanding the issue, but I couldn't

[jira] [Commented] (SPARK-23240) PythonWorkerFactory issues unhelpful message when pyspark.daemon produces bogus stdout

2018-02-10 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16359568#comment-16359568 ] Bruce Robbins commented on SPARK-23240: --- A little background. A Spark installation had a Python

[jira] [Comment Edited] (SPARK-23251) ClassNotFoundException: scala.Any when there's a missing implicit Map encoder

2018-01-30 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346246#comment-16346246 ] Bruce Robbins edited comment on SPARK-23251 at 1/31/18 4:35 AM: [~srowen] 

[jira] [Commented] (SPARK-23251) ClassNotFoundException: scala.Any when there's a missing implicit Map encoder

2018-01-30 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346246#comment-16346246 ] Bruce Robbins commented on SPARK-23251: --- [~srowen] This also occurs with compiled apps submitted

[jira] [Commented] (SPARK-23251) ClassNotFoundException: scala.Any when there's a missing implicit Map encoder

2018-01-30 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346050#comment-16346050 ] Bruce Robbins commented on SPARK-23251: --- I commented out the following line in 

[jira] [Commented] (SPARK-23240) PythonWorkerFactory issues unhelpful message when pyspark.daemon produces bogus stdout

2018-01-28 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16342688#comment-16342688 ] Bruce Robbins commented on SPARK-23240: --- Hi [~hyukjin.kwon], I am not sure this update covers the

[jira] [Created] (SPARK-23251) ClassNotFoundException: scala.Any when there's a missing implicit Map encoder

2018-01-27 Thread Bruce Robbins (JIRA)
Bruce Robbins created SPARK-23251: - Summary: ClassNotFoundException: scala.Any when there's a missing implicit Map encoder Key: SPARK-23251 URL: https://issues.apache.org/jira/browse/SPARK-23251

[jira] [Commented] (SPARK-23240) PythonWorkerFactory issues unhelpful message when pyspark.daemon produces bogus stdout

2018-01-26 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341609#comment-16341609 ] Bruce Robbins commented on SPARK-23240: --- I will be making a pull request. > PythonWorkerFactory

[jira] [Created] (SPARK-23240) PythonWorkerFactory issues unhelpful message when pyspark.daemon produces bogus stdout

2018-01-26 Thread Bruce Robbins (JIRA)
Bruce Robbins created SPARK-23240: - Summary: PythonWorkerFactory issues unhelpful message when pyspark.daemon produces bogus stdout Key: SPARK-23240 URL: https://issues.apache.org/jira/browse/SPARK-23240

[jira] [Updated] (SPARK-22940) Test suite HiveExternalCatalogVersionsSuite fails on platforms that don't have wget installed

2018-01-02 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-22940: -- Description: On platforms that don't have wget installed (e.g., Mac OS X), test suite

[jira] [Created] (SPARK-22940) Test suite HiveExternalCatalogVersionsSuite fails on platforms that don't have wget installed

2018-01-02 Thread Bruce Robbins (JIRA)
Bruce Robbins created SPARK-22940: - Summary: Test suite HiveExternalCatalogVersionsSuite fails on platforms that don't have wget installed Key: SPARK-22940 URL: https://issues.apache.org/jira/browse/SPARK-22940

[jira] [Updated] (SPARK-22940) Test suite HiveExternalCatalogVersionsSuite fails on platforms that don't have wget installed

2018-01-02 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-22940: -- Description: On platforms that don't have wget installed (e.g., Mac OS X), test suite

<    1   2   3   4   5