[jira] [Resolved] (SPARK-23815) Spark writer dynamic partition overwrite mode fails to write output on multi level partition

[ https://issues.apache.org/jira/browse/SPARK-23815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-23815. - Resolution: Fixed Fix Version/s: 2.3.1 2.4.0 Issue resolved by pull

[jira] [Resolved] (SPARK-23971) Should not leak Spark sessions across test suites

[ https://issues.apache.org/jira/browse/SPARK-23971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-23971. - Resolution: Fixed Assignee: Eric Liang Fix Version/s: 2.4.0 > Should not leak Spark

[jira] [Created] (SPARK-23975) Allow Clustering to take Arrays of Double as input features

Lu Wang created SPARK-23975: --- Summary: Allow Clustering to take Arrays of Double as input features Key: SPARK-23975 URL: https://issues.apache.org/jira/browse/SPARK-23975 Project: Spark Issue

[jira] [Commented] (SPARK-23942) PySpark's collect doesn't trigger QueryExecutionListener

[ https://issues.apache.org/jira/browse/SPARK-23942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16436753#comment-16436753 ] Apache Spark commented on SPARK-23942: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Resolved] (SPARK-23942) PySpark's collect doesn't trigger QueryExecutionListener

[ https://issues.apache.org/jira/browse/SPARK-23942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-23942. -- Resolution: Fixed Assignee: Hyukjin Kwon Fix Version/s: 2.4.0 Fixed in

[jira] [Updated] (SPARK-23974) Do not allocate more containers as expected in dynamic allocation

[ https://issues.apache.org/jira/browse/SPARK-23974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Darcy Shen updated SPARK-23974: --- Description: Using Yarn with dynamic allocation enabled, spark does not allocate more containers

[jira] [Resolved] (SPARK-23748) Support select from temp tables

[ https://issues.apache.org/jira/browse/SPARK-23748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-23748. --- Resolution: Fixed Fix Version/s: 2.3.1 3.0.0 Issue resolved by

[jira] [Assigned] (SPARK-23974) Do not allocate more containers as expected in dynamic allocation

[ https://issues.apache.org/jira/browse/SPARK-23974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23974: Assignee: (was: Apache Spark) > Do not allocate more containers as expected in

[jira] [Commented] (SPARK-23974) Do not allocate more containers as expected in dynamic allocation

[ https://issues.apache.org/jira/browse/SPARK-23974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16436726#comment-16436726 ] Apache Spark commented on SPARK-23974: -- User 'sadhen' has created a pull request for this issue:

[jira] [Assigned] (SPARK-23974) Do not allocate more containers as expected in dynamic allocation

[ https://issues.apache.org/jira/browse/SPARK-23974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23974: Assignee: Apache Spark > Do not allocate more containers as expected in dynamic

[jira] [Commented] (SPARK-23885) trying to spark submit 2.3.0 on minikube

[ https://issues.apache.org/jira/browse/SPARK-23885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16436724#comment-16436724 ] anant pukale commented on SPARK-23885: -- HI Anirudh, Thanks for response , I have already raised

[jira] [Created] (SPARK-23974) Do not allocate more containers as expected in dynamic allocation

Darcy Shen created SPARK-23974: -- Summary: Do not allocate more containers as expected in dynamic allocation Key: SPARK-23974 URL: https://issues.apache.org/jira/browse/SPARK-23974 Project: Spark

[jira] [Resolved] (SPARK-23867) com.codahale.metrics.Counter output in log message has no toString method

[ https://issues.apache.org/jira/browse/SPARK-23867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-23867. - Resolution: Fixed Assignee: Patrick Pisciuneri Fix Version/s: 2.4.0

[jira] [Commented] (SPARK-23965) make python/py4j-src-0.x.y.zip file name Spark version-independent

[ https://issues.apache.org/jira/browse/SPARK-23965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16436658#comment-16436658 ] Hyukjin Kwon commented on SPARK-23965: -- That's easy to work around in any case. We can simply make a

[jira] [Created] (SPARK-23973) Remove consecutive sorts

Henry Robinson created SPARK-23973: -- Summary: Remove consecutive sorts Key: SPARK-23973 URL: https://issues.apache.org/jira/browse/SPARK-23973 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-23972) Upgrade to Parquet 1.10

Henry Robinson created SPARK-23972: -- Summary: Upgrade to Parquet 1.10 Key: SPARK-23972 URL: https://issues.apache.org/jira/browse/SPARK-23972 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-23710) Upgrade Hive to 2.3.2

[ https://issues.apache.org/jira/browse/SPARK-23710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16436400#comment-16436400 ] t oo commented on SPARK-23710: -- Hive 2.3.3 has just been released, any plans for support? > Upgrade Hive to

[jira] [Commented] (SPARK-23852) Parquet MR bug can lead to incorrect SQL results

[ https://issues.apache.org/jira/browse/SPARK-23852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16436292#comment-16436292 ] Henry Robinson commented on SPARK-23852: [Here's a

[jira] [Assigned] (SPARK-23971) Should not leak Spark sessions across test suites

[ https://issues.apache.org/jira/browse/SPARK-23971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23971: Assignee: (was: Apache Spark) > Should not leak Spark sessions across test suites >

[jira] [Assigned] (SPARK-23971) Should not leak Spark sessions across test suites

[ https://issues.apache.org/jira/browse/SPARK-23971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23971: Assignee: Apache Spark > Should not leak Spark sessions across test suites >

[jira] [Commented] (SPARK-23971) Should not leak Spark sessions across test suites

[ https://issues.apache.org/jira/browse/SPARK-23971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16436209#comment-16436209 ] Apache Spark commented on SPARK-23971: -- User 'ericl' has created a pull request for this issue:

[jira] [Created] (SPARK-23971) Should not leak Spark sessions across test suites

Eric Liang created SPARK-23971: -- Summary: Should not leak Spark sessions across test suites Key: SPARK-23971 URL: https://issues.apache.org/jira/browse/SPARK-23971 Project: Spark Issue Type:

[jira] [Commented] (SPARK-23891) Debian based Dockerfile

[ https://issues.apache.org/jira/browse/SPARK-23891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16436196#comment-16436196 ] Erik Erlandson commented on SPARK-23891: I do think that these reports are very useful for

[jira] [Commented] (SPARK-23891) Debian based Dockerfile

[ https://issues.apache.org/jira/browse/SPARK-23891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16436185#comment-16436185 ] Erik Erlandson commented on SPARK-23891: The question of what OS base to use for "canonical"

[jira] [Commented] (SPARK-23931) High-order function: zip(array1, array2[, ...]) → array

[ https://issues.apache.org/jira/browse/SPARK-23931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16436184#comment-16436184 ] Dylan Guedes commented on SPARK-23931: -- I'm having some trouble so I asked for help in the PR -

[jira] [Updated] (SPARK-23891) Debian based Dockerfile

[ https://issues.apache.org/jira/browse/SPARK-23891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Erlandson updated SPARK-23891: --- Priority: Minor (was: Major) Issue Type: New Feature (was: Bug) > Debian based

[jira] [Resolved] (SPARK-23953) Add get_json_scalar function

[ https://issues.apache.org/jira/browse/SPARK-23953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timothy Chen resolved SPARK-23953. -- Resolution: Invalid > Add get_json_scalar function > > >

[jira] [Commented] (SPARK-23933) High-order function: map(array, array) → map<K,V>

[ https://issues.apache.org/jira/browse/SPARK-23933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16436136#comment-16436136 ] Kazuaki Ishizaki commented on SPARK-23933: -- I will work for this, thank you. > High-order

[jira] [Commented] (SPARK-23936) High-order function: map_concat(map1<K, V>, map2<K, V>, ..., mapN<K, V>) → map<K,V>

[ https://issues.apache.org/jira/browse/SPARK-23936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16436133#comment-16436133 ] Bruce Robbins commented on SPARK-23936: --- I would like to take this one, assuming no one has taken

[jira] [Comment Edited] (SPARK-23885) trying to spark submit 2.3.0 on minikube

[ https://issues.apache.org/jira/browse/SPARK-23885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16436094#comment-16436094 ] Anirudh Ramanathan edited comment on SPARK-23885 at 4/12/18 6:21 PM: -

[jira] [Commented] (SPARK-23885) trying to spark submit 2.3.0 on minikube

[ https://issues.apache.org/jira/browse/SPARK-23885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16436094#comment-16436094 ] Anirudh Ramanathan commented on SPARK-23885: Please post questions to the spark-user mailing

[jira] [Assigned] (SPARK-23885) trying to spark submit 2.3.0 on minikube

[ https://issues.apache.org/jira/browse/SPARK-23885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan reassigned SPARK-23885: -- Assignee: Anirudh Ramanathan > trying to spark submit 2.3.0 on minikube >

[jira] [Commented] (SPARK-23891) Debian based Dockerfile

[ https://issues.apache.org/jira/browse/SPARK-23891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16436092#comment-16436092 ] Anirudh Ramanathan commented on SPARK-23891: [~eje] has done a lot of research on these

[jira] [Commented] (SPARK-23849) Tests for the samplingRatio option of json schema inferring

[ https://issues.apache.org/jira/browse/SPARK-23849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16436025#comment-16436025 ] Apache Spark commented on SPARK-23849: -- User 'MaxGekk' has created a pull request for this issue:

[jira] [Resolved] (SPARK-23554) Hive's textinputformat.record.delimiter equivalent in Spark

[ https://issues.apache.org/jira/browse/SPARK-23554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruslan Dautkhanov resolved SPARK-23554. --- Resolution: Duplicate > Hive's textinputformat.record.delimiter equivalent in Spark

[jira] [Commented] (SPARK-23965) make python/py4j-src-0.x.y.zip file name Spark version-independent

[ https://issues.apache.org/jira/browse/SPARK-23965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16435884#comment-16435884 ] Ruslan Dautkhanov commented on SPARK-23965: --- [~hyukjin.kwon] how would you approach things like

[jira] [Assigned] (SPARK-23693) SQL function uuid()

[ https://issues.apache.org/jira/browse/SPARK-23693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23693: Assignee: (was: Apache Spark) > SQL function uuid() > --- > >

[jira] [Assigned] (SPARK-23693) SQL function uuid()

[ https://issues.apache.org/jira/browse/SPARK-23693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23693: Assignee: Apache Spark > SQL function uuid() > --- > >

[jira] [Commented] (SPARK-23693) SQL function uuid()

[ https://issues.apache.org/jira/browse/SPARK-23693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16435869#comment-16435869 ] Apache Spark commented on SPARK-23693: -- User 'tashoyan' has created a pull request for this issue:

[jira] [Commented] (SPARK-23907) Support regr_* functions

[ https://issues.apache.org/jira/browse/SPARK-23907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16435781#comment-16435781 ] Apache Spark commented on SPARK-23907: -- User 'mgaido91' has created a pull request for this issue:

[jira] [Assigned] (SPARK-23907) Support regr_* functions

[ https://issues.apache.org/jira/browse/SPARK-23907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23907: Assignee: (was: Apache Spark) > Support regr_* functions > >

[jira] [Assigned] (SPARK-23907) Support regr_* functions

[ https://issues.apache.org/jira/browse/SPARK-23907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23907: Assignee: Apache Spark > Support regr_* functions > > >

[jira] [Updated] (SPARK-23970) pyspark - simple filter/select doesn't use all tasks when coalesce is set

[ https://issues.apache.org/jira/browse/SPARK-23970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Anthony updated SPARK-23970: Description: Running in (py)spark 2.2.  Marking this as PySpark, but have not confirmed

[jira] [Updated] (SPARK-23970) pyspark - simple filter/select doesn't use all tasks when coalesce is set

[ https://issues.apache.org/jira/browse/SPARK-23970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Anthony updated SPARK-23970: Description: Running in (py)spark 2.2.  Marking this as PySpark, but have not confirmed

[jira] [Created] (SPARK-23970) pyspark - simple filter/select doesn't use all tasks when coalesce is set

Matthew Anthony created SPARK-23970: --- Summary: pyspark - simple filter/select doesn't use all tasks when coalesce is set Key: SPARK-23970 URL: https://issues.apache.org/jira/browse/SPARK-23970

[jira] [Resolved] (SPARK-23762) UTF8StringBuilder uses MemoryBlock

[ https://issues.apache.org/jira/browse/SPARK-23762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-23762. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20871

[jira] [Assigned] (SPARK-23762) UTF8StringBuilder uses MemoryBlock

[ https://issues.apache.org/jira/browse/SPARK-23762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-23762: --- Assignee: Kazuaki Ishizaki > UTF8StringBuilder uses MemoryBlock >

[jira] [Commented] (SPARK-23915) High-order function: array_except(x, y) → array

[ https://issues.apache.org/jira/browse/SPARK-23915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16435666#comment-16435666 ] Kazuaki Ishizaki commented on SPARK-23915: -- I will work for this, thanks. > High-order

[jira] [Assigned] (SPARK-23962) Flaky tests from SQLMetricsTestUtils.currentExecutionIds

[ https://issues.apache.org/jira/browse/SPARK-23962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-23962: --- Assignee: Imran Rashid > Flaky tests from SQLMetricsTestUtils.currentExecutionIds >

[jira] [Resolved] (SPARK-23962) Flaky tests from SQLMetricsTestUtils.currentExecutionIds

[ https://issues.apache.org/jira/browse/SPARK-23962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-23962. - Resolution: Fixed Fix Version/s: 2.3.1 2.4.0 Issue resolved by pull

[jira] [Assigned] (SPARK-23924) High-order function: element_at

[ https://issues.apache.org/jira/browse/SPARK-23924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23924: Assignee: (was: Apache Spark) > High-order function: element_at >

[jira] [Assigned] (SPARK-23924) High-order function: element_at

[ https://issues.apache.org/jira/browse/SPARK-23924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23924: Assignee: Apache Spark > High-order function: element_at >

[jira] [Commented] (SPARK-23924) High-order function: element_at

[ https://issues.apache.org/jira/browse/SPARK-23924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16435543#comment-16435543 ] Apache Spark commented on SPARK-23924: -- User 'kiszk' has created a pull request for this issue:

[jira] [Updated] (SPARK-23904) Big execution plan cause OOM

[ https://issues.apache.org/jira/browse/SPARK-23904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Izek Greenfield updated SPARK-23904: Description: I create a question in

[jira] [Commented] (SPARK-23799) [CBO] FilterEstimation.evaluateInSet produces devision by zero in a case of empty table with analyzed statistics

[ https://issues.apache.org/jira/browse/SPARK-23799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16435439#comment-16435439 ] Apache Spark commented on SPARK-23799: -- User 'mshtelma' has created a pull request for this issue:

[jira] [Commented] (SPARK-12635) More efficient (column batch) serialization for Python/R

[ https://issues.apache.org/jira/browse/SPARK-12635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16435405#comment-16435405 ] Naresh Sundararaji commented on SPARK-12635: I could see the status in In Progress. Any

[jira] [Updated] (SPARK-23969) Using FPGrowth with PipelinedRDD gives EOF Error

[ https://issues.apache.org/jira/browse/SPARK-23969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Srinivasan updated SPARK-23969: --- Attachment: EOFError.docx > Using FPGrowth with PipelinedRDD gives EOF Error >

[jira] [Commented] (SPARK-23967) Description add native sql show in SQL page.

[ https://issues.apache.org/jira/browse/SPARK-23967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16435341#comment-16435341 ] JieFang.He commented on SPARK-23967: I will do it,  thanks. > Description add native sql show in SQL

[jira] [Created] (SPARK-23969) Using FPGrowth with PipelinedRDD gives EOF Error

Srinivasan created SPARK-23969: -- Summary: Using FPGrowth with PipelinedRDD gives EOF Error Key: SPARK-23969 URL: https://issues.apache.org/jira/browse/SPARK-23969 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-12105) Add a DataFrame.show() with argument for output PrintStream

[ https://issues.apache.org/jira/browse/SPARK-12105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16435308#comment-16435308 ] Tomasz Gawęda commented on SPARK-12105: --- It can be used, but it should be so small change to the

[jira] [Commented] (SPARK-17519) [MESOS] Enhance robustness when ExternalShuffleService is broken

[ https://issues.apache.org/jira/browse/SPARK-17519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16435305#comment-16435305 ] Igor Berman commented on SPARK-17519: - Additional usecase: when driver can't connect to one of the

[jira] [Created] (SPARK-23968) allow reading JSON that is composed of pure maps

Andi created SPARK-23968: Summary: allow reading JSON that is composed of pure maps Key: SPARK-23968 URL: https://issues.apache.org/jira/browse/SPARK-23968 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-23967) Description add native sql show in SQL page.

JieFang.He created SPARK-23967: -- Summary: Description add native sql show in SQL page. Key: SPARK-23967 URL: https://issues.apache.org/jira/browse/SPARK-23967 Project: Spark Issue Type:

[jira] [Commented] (SPARK-23751) Kolmogorov-Smirnoff test Python API in pyspark.ml

[ https://issues.apache.org/jira/browse/SPARK-23751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16435232#comment-16435232 ] Apache Spark commented on SPARK-23751: -- User 'WeichenXu123' has created a pull request for this

[jira] [Commented] (SPARK-12105) Add a DataFrame.show() with argument for output PrintStream

[ https://issues.apache.org/jira/browse/SPARK-12105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16435019#comment-16435019 ] Takeshi Yamamuro commented on SPARK-12105: -- I feel it is enough to use Console? {code} val