[jira] [Commented] (SPARK-1018) take and collect don't work on HadoopRDD

[ https://issues.apache.org/jira/browse/SPARK-1018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14574472#comment-14574472 ] Igor Berman commented on SPARK-1018: Hi Patrick, We spent some time to understand why

[jira] [Commented] (SPARK-4879) Missing output partitions after job completes with speculative execution

[ https://issues.apache.org/jira/browse/SPARK-4879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14586266#comment-14586266 ] Igor Berman commented on SPARK-4879: I'm experiencing this issue. Sometimes rdd with 4

[jira] [Commented] (SPARK-4879) Missing output partitions after job completes with speculative execution

[ https://issues.apache.org/jira/browse/SPARK-4879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14708331#comment-14708331 ] Igor Berman commented on SPARK-4879: there is blog post

[jira] [Updated] (SPARK-17666) take() or isEmpty() on dataset leaks s3a connections

[ https://issues.apache.org/jira/browse/SPARK-17666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Berman updated SPARK-17666: Description: I'm experiensing problems with s3a and working with parquet with dataset api the

[jira] [Commented] (SPARK-17666) take() or isEmpty() on dataset leaks s3a connections

[ https://issues.apache.org/jira/browse/SPARK-17666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15523303#comment-15523303 ] Igor Berman commented on SPARK-17666: - sorry, still no idea. What's the difference between take and

[jira] [Commented] (SPARK-17666) take() or isEmpty() on dataset leaks s3a connections

[ https://issues.apache.org/jira/browse/SPARK-17666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15523286#comment-15523286 ] Igor Berman commented on SPARK-17666: - [~sowen], yes, I believe so rdd.isEmpty() uses take inside as

[jira] [Created] (SPARK-17666) take() or isEmpty() on dataset leaks s3a connections

Igor Berman created SPARK-17666: --- Summary: take() or isEmpty() on dataset leaks s3a connections Key: SPARK-17666 URL: https://issues.apache.org/jira/browse/SPARK-17666 Project: Spark Issue

[jira] [Updated] (SPARK-17666) take() or isEmpty() on dataset leaks s3a connections

[ https://issues.apache.org/jira/browse/SPARK-17666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Berman updated SPARK-17666: Description: I'm experiensing problems with s3a and working with parquet with dataset api the

[jira] [Created] (SPARK-24223) ExternalShuffleService looses registrations

Igor Berman created SPARK-24223: --- Summary: ExternalShuffleService looses registrations Key: SPARK-24223 URL: https://issues.apache.org/jira/browse/SPARK-24223 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-24567) nodeBlacklist does not get updated if a spark executor fails to launch on a mesos node

Igor Berman created SPARK-24567: --- Summary: nodeBlacklist does not get updated if a spark executor fails to launch on a mesos node Key: SPARK-24567 URL: https://issues.apache.org/jira/browse/SPARK-24567

[jira] [Commented] (SPARK-16787) SparkContext.addFile() should not fail if called twice with the same file

[ https://issues.apache.org/jira/browse/SPARK-16787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16515029#comment-16515029 ] Igor Berman commented on SPARK-16787: - [~joshrosen] I'm late for the party, but the new behaviour of

[jira] [Commented] (SPARK-24641) Spark-Mesos integration doesn't respect request to abort itself

[ https://issues.apache.org/jira/browse/SPARK-24641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16521613#comment-16521613 ] Igor Berman commented on SPARK-24641: - And another comment. I'm wondering if at initial design there

[jira] [Commented] (SPARK-24641) Spark-Mesos integration doesn't respect request to abort itself

[ https://issues.apache.org/jira/browse/SPARK-24641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16521609#comment-16521609 ] Igor Berman commented on SPARK-24641: - [~skonto] thanks for the comment! I'll look at link you've

[jira] [Created] (SPARK-24641) Spark-Mesos integration doesn't respect request to abort itself

Igor Berman created SPARK-24641: --- Summary: Spark-Mesos integration doesn't respect request to abort itself Key: SPARK-24641 URL: https://issues.apache.org/jira/browse/SPARK-24641 Project: Spark

[jira] [Created] (SPARK-23286) Partial registration to external shuffle services on Mesos

Igor Berman created SPARK-23286: --- Summary: Partial registration to external shuffle services on Mesos Key: SPARK-23286 URL: https://issues.apache.org/jira/browse/SPARK-23286 Project: Spark

[jira] [Created] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

Igor Berman created SPARK-23423: --- Summary: Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors Key: SPARK-23423 URL:

[jira] [Updated] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

[ https://issues.apache.org/jira/browse/SPARK-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Berman updated SPARK-23423: Description: Hi I've noticed rather strange behavior of MesosCoarseGrainedSchedulerBackend when

[jira] [Commented] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

[ https://issues.apache.org/jira/browse/SPARK-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365194#comment-16365194 ] Igor Berman commented on SPARK-23423: - Hi [~skonto], thanks for your response! Yes you are right,

[jira] [Updated] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

[ https://issues.apache.org/jira/browse/SPARK-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Berman updated SPARK-23423: Description: Hi Mesos Version:1.1.0 I've noticed rather strange behavior of 

[jira] [Commented] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

[ https://issues.apache.org/jira/browse/SPARK-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365438#comment-16365438 ] Igor Berman commented on SPARK-23423: - [~skonto] do you think it's could be connected to

[jira] [Updated] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

[ https://issues.apache.org/jira/browse/SPARK-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Berman updated SPARK-23423: Labels: Mesos dynamic_allocation (was: ) > Application declines any offers when killed+active

[jira] [Commented] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

[ https://issues.apache.org/jira/browse/SPARK-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365436#comment-16365436 ] Igor Berman commented on SPARK-23423: - [~skonto], yes this is correct. Besides TASK_RUNNING

[jira] [Comment Edited] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

[ https://issues.apache.org/jira/browse/SPARK-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369221#comment-16369221 ] Igor Berman edited comment on SPARK-23423 at 2/20/18 7:33 AM: -- [~skonto] so

[jira] [Commented] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

[ https://issues.apache.org/jira/browse/SPARK-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369221#comment-16369221 ] Igor Berman commented on SPARK-23423: - [~skonto] so I've run application today with relevant logs at

[jira] [Commented] (SPARK-19755) Blacklist is always active for MesosCoarseGrainedSchedulerBackend. As result - scheduler cannot create an executor after some time.

[ https://issues.apache.org/jira/browse/SPARK-19755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369227#comment-16369227 ] Igor Berman commented on SPARK-19755: - This Jira is very relevant for the case when running with  

[jira] [Comment Edited] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

[ https://issues.apache.org/jira/browse/SPARK-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369221#comment-16369221 ] Igor Berman edited comment on SPARK-23423 at 2/19/18 3:47 PM: -- [~skonto] so

[jira] [Issue Comment Deleted] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

[ https://issues.apache.org/jira/browse/SPARK-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Berman updated SPARK-23423: Comment: was deleted (was: [~skonto] today I haven't managed to run with dynamic allocation on,

[jira] [Commented] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

[ https://issues.apache.org/jira/browse/SPARK-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366162#comment-16366162 ] Igor Berman commented on SPARK-23423: - [~skonto] I'll at Sunday probably > Application declines any

[jira] [Comment Edited] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

[ https://issues.apache.org/jira/browse/SPARK-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366162#comment-16366162 ] Igor Berman edited comment on SPARK-23423 at 2/15/18 7:30 PM: -- [~skonto]

[jira] [Commented] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

[ https://issues.apache.org/jira/browse/SPARK-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16370114#comment-16370114 ] Igor Berman commented on SPARK-23423: - [~skonto] I've investigated failures a bit more so found 2

[jira] [Resolved] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

[ https://issues.apache.org/jira/browse/SPARK-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Berman resolved SPARK-23423. - Resolution: Duplicate > Application declines any offers when killed+active executors rich >

[jira] [Updated] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

[ https://issues.apache.org/jira/browse/SPARK-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Berman updated SPARK-23423: Attachment: (was: no-dyn-allocation-failed-no-statusUpdate.txt) > Application declines any

[jira] [Updated] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

[ https://issues.apache.org/jira/browse/SPARK-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Berman updated SPARK-23423: Attachment: no-dyn-allocation-failed-no-statusUpdate.txt > Application declines any offers when

[jira] [Commented] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

[ https://issues.apache.org/jira/browse/SPARK-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16368594#comment-16368594 ] Igor Berman commented on SPARK-23423: - [~skonto] today I haven't managed to run with dynamic

[jira] [Commented] (SPARK-17519) [MESOS] Enhance robustness when ExternalShuffleService is broken

[ https://issues.apache.org/jira/browse/SPARK-17519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16435305#comment-16435305 ] Igor Berman commented on SPARK-17519: - Additional usecase: when driver can't connect to one of the

[jira] [Commented] (SPARK-21444) Fetch failure due to node reboot causes job failure

[ https://issues.apache.org/jira/browse/SPARK-21444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16843357#comment-16843357 ] Igor Berman commented on SPARK-21444: - this Jira affects 2.2.3 version as well  as commit causing

[jira] [Commented] (SPARK-24641) Spark-Mesos integration doesn't respect request to abort itself

[ https://issues.apache.org/jira/browse/SPARK-24641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16833549#comment-16833549 ] Igor Berman commented on SPARK-24641: - I believe this issue is connected to 

[jira] [Comment Edited] (SPARK-23207) Shuffle+Repartition on an DataFrame could lead to incorrect answers

[ https://issues.apache.org/jira/browse/SPARK-23207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17551465#comment-17551465 ] Igor Berman edited comment on SPARK-23207 at 6/8/22 8:36 AM: - We are still

[jira] [Commented] (SPARK-23207) Shuffle+Repartition on an DataFrame could lead to incorrect answers

[ https://issues.apache.org/jira/browse/SPARK-23207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17551465#comment-17551465 ] Igor Berman commented on SPARK-23207: - We are still facing this issue in production with v3.1.2 at

[jira] [Comment Edited] (SPARK-39467) Count on distinct asterisk not equals to the count with column names provided

[ https://issues.apache.org/jira/browse/SPARK-39467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17556033#comment-17556033 ] Igor Berman edited comment on SPARK-39467 at 6/19/22 6:31 AM: --

[jira] [Commented] (SPARK-39467) Count on distinct asterisk not equals to the count with column names provided

[ https://issues.apache.org/jira/browse/SPARK-39467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17556033#comment-17556033 ] Igor Berman commented on SPARK-39467: - [~hyukjin.kwon] Hi, please take a look at a difference