[jira] [Comment Edited] (SPARK-46125) Memory leak when using createDataFrame with persist

[ https://issues.apache.org/jira/browse/SPARK-46125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17868582#comment-17868582 ] Igor Berman edited comment on SPARK-46125 at 7/25/24 7:01 AM:

[jira] [Updated] (SPARK-46125) Memory leak when using createDataFrame with persist

[ https://issues.apache.org/jira/browse/SPARK-46125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Berman updated SPARK-46125: Attachment: screenshot-1.png > Memory leak when using createDataFrame with persist > -

[jira] [Commented] (SPARK-46125) Memory leak when using createDataFrame with persist

[ https://issues.apache.org/jira/browse/SPARK-46125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17868582#comment-17868582 ] Igor Berman commented on SPARK-46125: - We experiencing kind of same memory leak in l

[jira] [Comment Edited] (SPARK-39467) Count on distinct asterisk not equals to the count with column names provided

[ https://issues.apache.org/jira/browse/SPARK-39467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17556033#comment-17556033 ] Igor Berman edited comment on SPARK-39467 at 6/19/22 6:31 AM:

[jira] [Commented] (SPARK-39467) Count on distinct asterisk not equals to the count with column names provided

[ https://issues.apache.org/jira/browse/SPARK-39467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17556033#comment-17556033 ] Igor Berman commented on SPARK-39467: - [~hyukjin.kwon] Hi, please take a look at a

[jira] [Comment Edited] (SPARK-23207) Shuffle+Repartition on an DataFrame could lead to incorrect answers

[ https://issues.apache.org/jira/browse/SPARK-23207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17551465#comment-17551465 ] Igor Berman edited comment on SPARK-23207 at 6/8/22 8:36 AM: -

[jira] [Commented] (SPARK-23207) Shuffle+Repartition on an DataFrame could lead to incorrect answers

[ https://issues.apache.org/jira/browse/SPARK-23207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17551465#comment-17551465 ] Igor Berman commented on SPARK-23207: - We are still facing this issue in production

[jira] [Commented] (SPARK-21444) Fetch failure due to node reboot causes job failure

[ https://issues.apache.org/jira/browse/SPARK-21444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16843357#comment-16843357 ] Igor Berman commented on SPARK-21444: - this Jira affects 2.2.3 version as well  as

[jira] [Commented] (SPARK-24641) Spark-Mesos integration doesn't respect request to abort itself

[ https://issues.apache.org/jira/browse/SPARK-24641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16833549#comment-16833549 ] Igor Berman commented on SPARK-24641: - I believe this issue is connected to  https:/

[jira] [Commented] (SPARK-24641) Spark-Mesos integration doesn't respect request to abort itself

[ https://issues.apache.org/jira/browse/SPARK-24641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16521613#comment-16521613 ] Igor Berman commented on SPARK-24641: - And another comment. I'm wondering if at init

[jira] [Commented] (SPARK-24641) Spark-Mesos integration doesn't respect request to abort itself

[ https://issues.apache.org/jira/browse/SPARK-24641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16521609#comment-16521609 ] Igor Berman commented on SPARK-24641: - [~skonto] thanks for the comment! I'll look a

[jira] [Created] (SPARK-24641) Spark-Mesos integration doesn't respect request to abort itself

Igor Berman created SPARK-24641: --- Summary: Spark-Mesos integration doesn't respect request to abort itself Key: SPARK-24641 URL: https://issues.apache.org/jira/browse/SPARK-24641 Project: Spark

[jira] [Commented] (SPARK-16787) SparkContext.addFile() should not fail if called twice with the same file

[ https://issues.apache.org/jira/browse/SPARK-16787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16515029#comment-16515029 ] Igor Berman commented on SPARK-16787: - [~joshrosen] I'm late for the party, but the

[jira] [Created] (SPARK-24567) nodeBlacklist does not get updated if a spark executor fails to launch on a mesos node

Igor Berman created SPARK-24567: --- Summary: nodeBlacklist does not get updated if a spark executor fails to launch on a mesos node Key: SPARK-24567 URL: https://issues.apache.org/jira/browse/SPARK-24567

[jira] [Created] (SPARK-24223) ExternalShuffleService looses registrations

Igor Berman created SPARK-24223: --- Summary: ExternalShuffleService looses registrations Key: SPARK-24223 URL: https://issues.apache.org/jira/browse/SPARK-24223 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-17519) [MESOS] Enhance robustness when ExternalShuffleService is broken

[ https://issues.apache.org/jira/browse/SPARK-17519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16435305#comment-16435305 ] Igor Berman commented on SPARK-17519: - Additional usecase: when driver can't connect

[jira] [Resolved] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

[ https://issues.apache.org/jira/browse/SPARK-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Berman resolved SPARK-23423. - Resolution: Duplicate > Application declines any offers when killed+active executors rich > spar

[jira] [Commented] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

[ https://issues.apache.org/jira/browse/SPARK-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16370114#comment-16370114 ] Igor Berman commented on SPARK-23423: - [~skonto] I've investigated failures a bit mor

[jira] [Comment Edited] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

[ https://issues.apache.org/jira/browse/SPARK-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16369221#comment-16369221 ] Igor Berman edited comment on SPARK-23423 at 2/20/18 7:33 AM: -

[jira] [Updated] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

[ https://issues.apache.org/jira/browse/SPARK-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Berman updated SPARK-23423: Attachment: (was: no-dyn-allocation-failed-no-statusUpdate.txt) > Application declines any offe

[jira] [Comment Edited] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

[ https://issues.apache.org/jira/browse/SPARK-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16369221#comment-16369221 ] Igor Berman edited comment on SPARK-23423 at 2/19/18 3:47 PM: -

[jira] [Issue Comment Deleted] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

[ https://issues.apache.org/jira/browse/SPARK-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Berman updated SPARK-23423: Comment: was deleted (was: [~skonto] today I haven't managed to run with dynamic allocation on, ho

[jira] [Commented] (SPARK-19755) Blacklist is always active for MesosCoarseGrainedSchedulerBackend. As result - scheduler cannot create an executor after some time.

[ https://issues.apache.org/jira/browse/SPARK-19755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16369227#comment-16369227 ] Igor Berman commented on SPARK-19755: - This Jira is very relevant for the case when r

[jira] [Commented] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

[ https://issues.apache.org/jira/browse/SPARK-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16369221#comment-16369221 ] Igor Berman commented on SPARK-23423: - [~skonto] so I've run application today with r

[jira] [Commented] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

[ https://issues.apache.org/jira/browse/SPARK-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16368594#comment-16368594 ] Igor Berman commented on SPARK-23423: - [~skonto] today I haven't managed to run with

[jira] [Updated] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

[ https://issues.apache.org/jira/browse/SPARK-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Berman updated SPARK-23423: Attachment: no-dyn-allocation-failed-no-statusUpdate.txt > Application declines any offers when kil

[jira] [Comment Edited] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

[ https://issues.apache.org/jira/browse/SPARK-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16366162#comment-16366162 ] Igor Berman edited comment on SPARK-23423 at 2/15/18 7:30 PM: -

[jira] [Commented] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

[ https://issues.apache.org/jira/browse/SPARK-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16366162#comment-16366162 ] Igor Berman commented on SPARK-23423: - [~skonto] I'll at Sunday probably > Applicati

[jira] [Updated] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

[ https://issues.apache.org/jira/browse/SPARK-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Berman updated SPARK-23423: Labels: Mesos dynamic_allocation (was: ) > Application declines any offers when killed+active exec

[jira] [Commented] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

[ https://issues.apache.org/jira/browse/SPARK-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16365438#comment-16365438 ] Igor Berman commented on SPARK-23423: - [~skonto] do you think it's could be connected

[jira] [Commented] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

[ https://issues.apache.org/jira/browse/SPARK-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16365436#comment-16365436 ] Igor Berman commented on SPARK-23423: - [~skonto], yes this is correct. Besides TASK_R

[jira] [Commented] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

[ https://issues.apache.org/jira/browse/SPARK-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16365194#comment-16365194 ] Igor Berman commented on SPARK-23423: - Hi [~skonto], thanks for your response! Yes y

[jira] [Updated] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

[ https://issues.apache.org/jira/browse/SPARK-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Berman updated SPARK-23423: Description: Hi Mesos Version:1.1.0 I've noticed rather strange behavior of MesosCoarseGrainedSch

[jira] [Updated] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

[ https://issues.apache.org/jira/browse/SPARK-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Berman updated SPARK-23423: Description: Hi I've noticed rather strange behavior of MesosCoarseGrainedSchedulerBackend when r

[jira] [Created] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

Igor Berman created SPARK-23423: --- Summary: Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors Key: SPARK-23423 URL: https://issues.apache.org/jira/browse/SPARK-234

[jira] [Created] (SPARK-23286) Partial registration to external shuffle services on Mesos

Igor Berman created SPARK-23286: --- Summary: Partial registration to external shuffle services on Mesos Key: SPARK-23286 URL: https://issues.apache.org/jira/browse/SPARK-23286 Project: Spark Issu

[jira] [Updated] (SPARK-17666) take() or isEmpty() on dataset leaks s3a connections

[ https://issues.apache.org/jira/browse/SPARK-17666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Berman updated SPARK-17666: Description: I'm experiensing problems with s3a and working with parquet with dataset api the sympt

[jira] [Commented] (SPARK-17666) take() or isEmpty() on dataset leaks s3a connections

[ https://issues.apache.org/jira/browse/SPARK-17666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15523303#comment-15523303 ] Igor Berman commented on SPARK-17666: - sorry, still no idea. What's the difference be

[jira] [Updated] (SPARK-17666) take() or isEmpty() on dataset leaks s3a connections

[ https://issues.apache.org/jira/browse/SPARK-17666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Berman updated SPARK-17666: Description: I'm experiensing problems with s3a and working with parquet with dataset api the sympt

[jira] [Commented] (SPARK-17666) take() or isEmpty() on dataset leaks s3a connections

[ https://issues.apache.org/jira/browse/SPARK-17666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15523286#comment-15523286 ] Igor Berman commented on SPARK-17666: - [~sowen], yes, I believe so rdd.isEmpty() uses

[jira] [Created] (SPARK-17666) take() or isEmpty() on dataset leaks s3a connections

Igor Berman created SPARK-17666: --- Summary: take() or isEmpty() on dataset leaks s3a connections Key: SPARK-17666 URL: https://issues.apache.org/jira/browse/SPARK-17666 Project: Spark Issue Type

[jira] [Commented] (SPARK-4879) Missing output partitions after job completes with speculative execution

[ https://issues.apache.org/jira/browse/SPARK-4879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14708331#comment-14708331 ] Igor Berman commented on SPARK-4879: there is blog post http://tech.grammarly.com/blo

[jira] [Commented] (SPARK-4879) Missing output partitions after job completes with speculative execution

[ https://issues.apache.org/jira/browse/SPARK-4879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14586266#comment-14586266 ] Igor Berman commented on SPARK-4879: I'm experiencing this issue. Sometimes rdd with 4

[jira] [Commented] (SPARK-1018) take and collect don't work on HadoopRDD

[ https://issues.apache.org/jira/browse/SPARK-1018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14574472#comment-14574472 ] Igor Berman commented on SPARK-1018: Hi Patrick, We spent some time to understand why