[jira] [Assigned] (SPARK-9312) The OneVsRest model does not provide confidence factor(not probability) along with the prediction

[ https://issues.apache.org/jira/browse/SPARK-9312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley reassigned SPARK-9312: Assignee: Lu Wang > The OneVsRest model does not provide confidence factor(not

[jira] [Resolved] (SPARK-9312) The OneVsRest model does not provide confidence factor(not probability) along with the prediction

[ https://issues.apache.org/jira/browse/SPARK-9312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley resolved SPARK-9312. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21044

[jira] [Created] (SPARK-23995) initial job has not accept any resources and executor keep exit

Cong Shen created SPARK-23995: - Summary: initial job has not accept any resources and executor keep exit Key: SPARK-23995 URL: https://issues.apache.org/jira/browse/SPARK-23995 Project: Spark

[jira] [Updated] (SPARK-9312) The OneVsRest model does not provide rawPrediction

[ https://issues.apache.org/jira/browse/SPARK-9312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-9312: - Summary: The OneVsRest model does not provide rawPrediction (was: The OneVsRest model

[jira] [Commented] (SPARK-23030) Decrease memory consumption with toPandas() collection using Arrow

[ https://issues.apache.org/jira/browse/SPARK-23030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439727#comment-16439727 ] Bryan Cutler commented on SPARK-23030: -- Hi [~icexelloss], I have something working, just need to

[jira] [Commented] (SPARK-23975) Allow Clustering to take Arrays of Double as input features

[ https://issues.apache.org/jira/browse/SPARK-23975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439793#comment-16439793 ] Apache Spark commented on SPARK-23975: -- User 'ludatabricks' has created a pull request for this

[jira] [Updated] (SPARK-23995) initial job has not accept any resources and executor keep exit

[ https://issues.apache.org/jira/browse/SPARK-23995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cong Shen updated SPARK-23995: -- Environment: Spark version:2.3.0 JDK version: 1.8.0_131 System: CentOS v7 {\{export

[jira] [Resolved] (SPARK-21088) CrossValidator, TrainValidationSplit should collect all models when fitting: Python API

[ https://issues.apache.org/jira/browse/SPARK-21088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley resolved SPARK-21088. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 19627

[jira] [Commented] (SPARK-23904) Big execution plan cause OOM

[ https://issues.apache.org/jira/browse/SPARK-23904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439815#comment-16439815 ] Ruben Berenguel commented on SPARK-23904: - I'll give it a look, maybe there is a way to avoid it

[jira] [Assigned] (SPARK-21088) CrossValidator, TrainValidationSplit should collect all models when fitting: Python API

[ https://issues.apache.org/jira/browse/SPARK-21088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley reassigned SPARK-21088: - Assignee: Weichen Xu > CrossValidator, TrainValidationSplit should collect all

[jira] [Assigned] (SPARK-23975) Allow Clustering to take Arrays of Double as input features

[ https://issues.apache.org/jira/browse/SPARK-23975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23975: Assignee: (was: Apache Spark) > Allow Clustering to take Arrays of Double as input

[jira] [Assigned] (SPARK-23975) Allow Clustering to take Arrays of Double as input features

[ https://issues.apache.org/jira/browse/SPARK-23975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23975: Assignee: Apache Spark > Allow Clustering to take Arrays of Double as input features >

[jira] [Assigned] (SPARK-23992) ShuffleDependency does not need to be deserialized every time

[ https://issues.apache.org/jira/browse/SPARK-23992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23992: Assignee: (was: Apache Spark) > ShuffleDependency does not need to be deserialized

[jira] [Comment Edited] (SPARK-23901) Data Masking Functions

[ https://issues.apache.org/jira/browse/SPARK-23901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437401#comment-16437401 ] Marco Gaido edited comment on SPARK-23901 at 4/16/18 12:19 PM: --- Actually I

[jira] [Commented] (SPARK-23981) ShuffleBlockFetcherIterator - Spamming Logs

[ https://issues.apache.org/jira/browse/SPARK-23981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439614#comment-16439614 ] BELUGA BEHR commented on SPARK-23981: - Or maybe lower per-block logging and debug and produce one

[jira] [Created] (SPARK-23994) Add Host To Blacklist If Shuffle Cannot Complete

BELUGA BEHR created SPARK-23994: --- Summary: Add Host To Blacklist If Shuffle Cannot Complete Key: SPARK-23994 URL: https://issues.apache.org/jira/browse/SPARK-23994 Project: Spark Issue Type:

[jira] [Updated] (SPARK-23986) CompileException when using too many avg aggregation after joining

[ https://issues.apache.org/jira/browse/SPARK-23986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michel Davit updated SPARK-23986: - Description: Considering the following code: {code:java} val df1: DataFrame =

[jira] [Commented] (SPARK-23986) CompileException when using too many avg aggregation after joining

[ https://issues.apache.org/jira/browse/SPARK-23986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439398#comment-16439398 ] Marco Gaido commented on SPARK-23986: - [~RustedBones] I was able to reproduce. Yes, I do agree with

[jira] [Updated] (SPARK-23991) data loss when allocateBlocksToBatch

[ https://issues.apache.org/jira/browse/SPARK-23991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kevin fu updated SPARK-23991: - Description: with checkpoint and WAL enabled, driver will write the allocation of blocks to batch into

[jira] [Updated] (SPARK-23986) CompileException when using too many avg aggregation after joining

[ https://issues.apache.org/jira/browse/SPARK-23986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michel Davit updated SPARK-23986: - Attachment: spark-generated.java > CompileException when using too many avg aggregation after

[jira] [Commented] (SPARK-23986) CompileException when using too many avg aggregation after joining

[ https://issues.apache.org/jira/browse/SPARK-23986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439318#comment-16439318 ] Michel Davit commented on SPARK-23986: -- I tested on Spark v2.3.0 I attached the generated code: 

[jira] [Commented] (SPARK-23992) ShuffleDependency does not need to be deserialized every time

[ https://issues.apache.org/jira/browse/SPARK-23992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439349#comment-16439349 ] Apache Spark commented on SPARK-23992: -- User '10110346' has created a pull request for this issue:

[jira] [Assigned] (SPARK-23992) ShuffleDependency does not need to be deserialized every time

[ https://issues.apache.org/jira/browse/SPARK-23992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23992: Assignee: Apache Spark > ShuffleDependency does not need to be deserialized every time >

[jira] [Created] (SPARK-23991) data loss when allocateBlocksToBatch

kevin fu created SPARK-23991: Summary: data loss when allocateBlocksToBatch Key: SPARK-23991 URL: https://issues.apache.org/jira/browse/SPARK-23991 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-23901) Data Masking Functions

[ https://issues.apache.org/jira/browse/SPARK-23901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439354#comment-16439354 ] Marco Gaido commented on SPARK-23901: - [~ueshin] maybe you have some inputs on my questions above. >

[jira] [Updated] (SPARK-23986) CompileException when using too many avg aggregation after joining

[ https://issues.apache.org/jira/browse/SPARK-23986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michel Davit updated SPARK-23986: - Description: Considering the following code: {code:java} val df1: DataFrame =

[jira] [Assigned] (SPARK-23986) CompileException when using too many avg aggregation after joining

[ https://issues.apache.org/jira/browse/SPARK-23986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23986: Assignee: Apache Spark > CompileException when using too many avg aggregation after

[jira] [Commented] (SPARK-23986) CompileException when using too many avg aggregation after joining

[ https://issues.apache.org/jira/browse/SPARK-23986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439915#comment-16439915 ] Michel Davit commented on SPARK-23986: -- Thx [~mgaido]. I didn't have time to setup the environment

[jira] [Updated] (SPARK-23975) Allow Clustering to take Arrays of Double as input features

[ https://issues.apache.org/jira/browse/SPARK-23975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-23975: -- Shepherd: Joseph K. Bradley > Allow Clustering to take Arrays of Double as input

[jira] [Created] (SPARK-23996) Implement the optimal KLL algorithms for quantiles in streams

Timothy Hunter created SPARK-23996: -- Summary: Implement the optimal KLL algorithms for quantiles in streams Key: SPARK-23996 URL: https://issues.apache.org/jira/browse/SPARK-23996 Project: Spark

[jira] [Resolved] (SPARK-23873) Use accessors in interpreted LambdaVariable

[ https://issues.apache.org/jira/browse/SPARK-23873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell resolved SPARK-23873. --- Resolution: Fixed Assignee: Liang-Chi Hsieh Fix Version/s: 2.4.0 >

[jira] [Commented] (SPARK-19618) Inconsistency wrt max. buckets allowed from Dataframe API vs SQL

[ https://issues.apache.org/jira/browse/SPARK-19618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440085#comment-16440085 ] Fernando Pereira commented on SPARK-19618: -- Opened 

[jira] [Commented] (SPARK-22239) User-defined window functions with pandas udf

[ https://issues.apache.org/jira/browse/SPARK-22239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440094#comment-16440094 ] Apache Spark commented on SPARK-22239: -- User 'icexelloss' has created a pull request for this issue:

[jira] [Assigned] (SPARK-22239) User-defined window functions with pandas udf

[ https://issues.apache.org/jira/browse/SPARK-22239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22239: Assignee: Apache Spark > User-defined window functions with pandas udf >

[jira] [Assigned] (SPARK-22239) User-defined window functions with pandas udf

[ https://issues.apache.org/jira/browse/SPARK-22239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22239: Assignee: (was: Apache Spark) > User-defined window functions with pandas udf >

[jira] [Commented] (SPARK-23638) Spark on k8s: spark.kubernetes.initContainer.image has no effect

[ https://issues.apache.org/jira/browse/SPARK-23638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440067#comment-16440067 ] Yinan Li commented on SPARK-23638: -- Can this be closed? > Spark on k8s:

[jira] [Created] (SPARK-23997) Configurable max number of buckets

Fernando Pereira created SPARK-23997: Summary: Configurable max number of buckets Key: SPARK-23997 URL: https://issues.apache.org/jira/browse/SPARK-23997 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-23985) predicate push down doesn't work with simple compound partition spec

[ https://issues.apache.org/jira/browse/SPARK-23985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439114#comment-16439114 ] Ohad Raviv commented on SPARK-23985: I see in the Optimizer that filters are getting pushed only if

[jira] [Commented] (SPARK-21033) fix the potential OOM in UnsafeExternalSorter

[ https://issues.apache.org/jira/browse/SPARK-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439118#comment-16439118 ] Apache Spark commented on SPARK-21033: -- User 'wangyum' has created a pull request for this issue:

[jira] [Commented] (SPARK-15703) Make ListenerBus event queue size configurable

[ https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440377#comment-16440377 ] rohit verma commented on SPARK-15703: - Hi Guys,  I am worried weather this issue has been resolved

[jira] [Updated] (SPARK-23991) data loss when allocateBlocksToBatch

[ https://issues.apache.org/jira/browse/SPARK-23991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kevin fu updated SPARK-23991: - Description: with checkpoint and WAL enabled, driver will write the allocation of blocks to batch into

[jira] [Created] (SPARK-23992) ShuffleDependency does not need to be deserialized every time

liuxian created SPARK-23992: --- Summary: ShuffleDependency does not need to be deserialized every time Key: SPARK-23992 URL: https://issues.apache.org/jira/browse/SPARK-23992 Project: Spark Issue

[jira] [Created] (SPARK-23993) Support DESC FORMATTED table_name column_name

Volodymyr Glushak created SPARK-23993: - Summary: Support DESC FORMATTED table_name column_name Key: SPARK-23993 URL: https://issues.apache.org/jira/browse/SPARK-23993 Project: Spark

[jira] [Commented] (SPARK-23989) When using `SortShuffleWriter`, the data will be overwritten

[ https://issues.apache.org/jira/browse/SPARK-23989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439384#comment-16439384 ] Wenchen Fan commented on SPARK-23989: - do you have an end-to-end case to show this bug? IIRC we

[jira] [Commented] (SPARK-23986) CompileException when using too many avg aggregation after joining

[ https://issues.apache.org/jira/browse/SPARK-23986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439417#comment-16439417 ] Apache Spark commented on SPARK-23986: -- User 'mgaido91' has created a pull request for this issue:

[jira] [Assigned] (SPARK-23986) CompileException when using too many avg aggregation after joining

[ https://issues.apache.org/jira/browse/SPARK-23986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23986: Assignee: (was: Apache Spark) > CompileException when using too many avg aggregation

[jira] [Updated] (SPARK-23989) When using `SortShuffleWriter`, the data will be overwritten

[ https://issues.apache.org/jira/browse/SPARK-23989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuxian updated SPARK-23989: Description: {color:#33}When using `SortShuffleWriter`, we only insert 

[jira] [Commented] (SPARK-23989) When using `SortShuffleWriter`, the data will be overwritten

[ https://issues.apache.org/jira/browse/SPARK-23989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439148#comment-16439148 ] liuxian commented on SPARK-23989: - [~joshrosen] > When using `SortShuffleWriter`, the data will be

[jira] [Created] (SPARK-23989) When using `SortShuffleWriter`, the data will be overwritten

liuxian created SPARK-23989: --- Summary: When using `SortShuffleWriter`, the data will be overwritten Key: SPARK-23989 URL: https://issues.apache.org/jira/browse/SPARK-23989 Project: Spark Issue

[jira] [Comment Edited] (SPARK-23989) When using `SortShuffleWriter`, the data will be overwritten

[ https://issues.apache.org/jira/browse/SPARK-23989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439148#comment-16439148 ] liuxian edited comment on SPARK-23989 at 4/16/18 9:00 AM: -- [~joshrosen]

[jira] [Commented] (SPARK-23989) When using `SortShuffleWriter`, the data will be overwritten

[ https://issues.apache.org/jira/browse/SPARK-23989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440250#comment-16440250 ] liuxian commented on SPARK-23989: - If we make 'BypassMergeSortShuffleHandle' and

[jira] [Comment Edited] (SPARK-23206) Additional Memory Tuning Metrics

[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438879#comment-16438879 ] Edwina Lu edited comment on SPARK-23206 at 4/17/18 3:14 AM: After discussion

[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440334#comment-16440334 ] Imran Rashid commented on SPARK-23206: -- thanks, shared doc works for me now! > Additional Memory

[jira] [Comment Edited] (SPARK-23206) Additional Memory Tuning Metrics

[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438879#comment-16438879 ] Edwina Lu edited comment on SPARK-23206 at 4/17/18 3:15 AM: After discussion

[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440326#comment-16440326 ] Edwina Lu commented on SPARK-23206: --- [~smilegator], please try:

[jira] [Commented] (SPARK-23989) When using `SortShuffleWriter`, the data will be overwritten

[ https://issues.apache.org/jira/browse/SPARK-23989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439205#comment-16439205 ] Wenchen Fan commented on SPARK-23989: - Can you be more specific about what the problem is? > When

[jira] [Commented] (SPARK-23989) When using `SortShuffleWriter`, the data will be overwritten

[ https://issues.apache.org/jira/browse/SPARK-23989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439231#comment-16439231 ] liuxian commented on SPARK-23989: - For {color:#33}`SortShuffleWriter`{color},  `records:

[jira] [Comment Edited] (SPARK-23989) When using `SortShuffleWriter`, the data will be overwritten

[ https://issues.apache.org/jira/browse/SPARK-23989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439231#comment-16439231 ] liuxian edited comment on SPARK-23989 at 4/16/18 10:18 AM: --- For

[jira] [Created] (SPARK-23990) Instruments logging improvements - ML regression package

Weichen Xu created SPARK-23990: -- Summary: Instruments logging improvements - ML regression package Key: SPARK-23990 URL: https://issues.apache.org/jira/browse/SPARK-23990 Project: Spark Issue

[jira] [Updated] (SPARK-23990) Instruments logging improvements - ML regression package

[ https://issues.apache.org/jira/browse/SPARK-23990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu updated SPARK-23990: --- Issue Type: Sub-task (was: Bug) Parent: SPARK-23686 > Instruments logging improvements - ML

[jira] [Commented] (SPARK-23990) Instruments logging improvements - ML regression package

[ https://issues.apache.org/jira/browse/SPARK-23990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439268#comment-16439268 ] Apache Spark commented on SPARK-23990: -- User 'WeichenXu123' has created a pull request for this

[jira] [Assigned] (SPARK-23990) Instruments logging improvements - ML regression package

[ https://issues.apache.org/jira/browse/SPARK-23990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23990: Assignee: (was: Apache Spark) > Instruments logging improvements - ML regression

[jira] [Assigned] (SPARK-23990) Instruments logging improvements - ML regression package

[ https://issues.apache.org/jira/browse/SPARK-23990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23990: Assignee: Apache Spark > Instruments logging improvements - ML regression package >