[jira] [Resolved] (SPARK-32923) Add support to properly handle different type of stage retries

[ https://issues.apache.org/jira/browse/SPARK-32923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-32923. - Resolution: Fixed This has been handled by SPARK-32923 > Add support to properl

[jira] [Assigned] (SPARK-36378) Minor changes to address a few identified server side inefficiencies

[ https://issues.apache.org/jira/browse/SPARK-36378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-36378: --- Assignee: Mridul Muralidharan > Minor changes to address a few identified s

[jira] [Resolved] (SPARK-36378) Minor changes to address a few identified server side inefficiencies

[ https://issues.apache.org/jira/browse/SPARK-36378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-36378. - Resolution: Won't Fix Let us move this outside of the SPIP and into individual j

[jira] [Assigned] (SPARK-35917) Disable push-based shuffle until the feature is complete

[ https://issues.apache.org/jira/browse/SPARK-35917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-35917: --- Assignee: Mridul Muralidharan > Disable push-based shuffle until the featur

[jira] [Resolved] (SPARK-35917) Disable push-based shuffle until the feature is complete

[ https://issues.apache.org/jira/browse/SPARK-35917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-35917. - Resolution: Won't Fix > Disable push-based shuffle until the feature is complete

[jira] [Commented] (SPARK-35917) Disable push-based shuffle until the feature is complete

[ https://issues.apache.org/jira/browse/SPARK-35917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17391313#comment-17391313 ] Mridul Muralidharan commented on SPARK-35917: - Closing this Jira - as push b

[jira] [Resolved] (SPARK-36266) Rename classes in shuffle RPC used for block push operations

[ https://issues.apache.org/jira/browse/SPARK-36266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-36266. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 333

[jira] [Assigned] (SPARK-35546) Enable push-based shuffle when multiple app attempts are enabled and manage concurrent access to the state in a better way

[ https://issues.apache.org/jira/browse/SPARK-35546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-35546: --- Assignee: Ye Zhou > Enable push-based shuffle when multiple app attempts ar

[jira] [Assigned] (SPARK-35276) Write checksum files for shuffle

[ https://issues.apache.org/jira/browse/SPARK-35276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-35276: --- Assignee: wuyi > Write checksum files for shuffle > --

[jira] [Resolved] (SPARK-35276) Write checksum files for shuffle

[ https://issues.apache.org/jira/browse/SPARK-35276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-35276. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 324

[jira] [Resolved] (SPARK-32922) Add support for ShuffleBlockFetcherIterator to read from merged shuffle partitions and to fallback to original shuffle blocks if encountering failures

[ https://issues.apache.org/jira/browse/SPARK-32922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-32922. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 321

[jira] [Assigned] (SPARK-32922) Add support for ShuffleBlockFetcherIterator to read from merged shuffle partitions and to fallback to original shuffle blocks if encountering failures

[ https://issues.apache.org/jira/browse/SPARK-32922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-32922: --- Assignee: Chandni Singh > Add support for ShuffleBlockFetcherIterator to re

[jira] [Resolved] (SPARK-35258) Enhance ESS ExternalBlockHandler with additional block rate-based metrics and histograms

[ https://issues.apache.org/jira/browse/SPARK-35258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-35258. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 323

[jira] [Assigned] (SPARK-35258) Enhance ESS ExternalBlockHandler with additional block rate-based metrics and histograms

[ https://issues.apache.org/jira/browse/SPARK-35258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-35258: --- Assignee: Erik Krogen > Enhance ESS ExternalBlockHandler with additional bl

[jira] [Resolved] (SPARK-35836) Remove reference to spark.shuffle.push.based.enabled in ShuffleBlockPusherSuite

[ https://issues.apache.org/jira/browse/SPARK-35836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-35836. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 329

[jira] [Assigned] (SPARK-35836) Remove reference to spark.shuffle.push.based.enabled in ShuffleBlockPusherSuite

[ https://issues.apache.org/jira/browse/SPARK-35836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-35836: --- Assignee: Chandni Singh > Remove reference to spark.shuffle.push.based.enab

[jira] [Assigned] (SPARK-35671) Add Support in the ESS to serve merged shuffle block meta and data to executors

[ https://issues.apache.org/jira/browse/SPARK-35671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-35671: --- Assignee: Chandni Singh > Add Support in the ESS to serve merged shuffle bl

[jira] [Resolved] (SPARK-35671) Add Support in the ESS to serve merged shuffle block meta and data to executors

[ https://issues.apache.org/jira/browse/SPARK-35671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-35671. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 328

[jira] [Resolved] (SPARK-34898) Send ExecutorMetricsUpdate EventLog appropriately

[ https://issues.apache.org/jira/browse/SPARK-34898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-34898. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 319

[jira] [Resolved] (SPARK-35613) Cache commonly occurring strings from SQLMetrics, JsonProtocol and AccumulatorV2

[ https://issues.apache.org/jira/browse/SPARK-35613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-35613. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 327

[jira] [Assigned] (SPARK-35613) Cache commonly occurring strings from SQLMetrics, JsonProtocol and AccumulatorV2

[ https://issues.apache.org/jira/browse/SPARK-35613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-35613: --- Assignee: Venkata krishnan Sowrirajan > Cache commonly occurring strings fr

[jira] [Assigned] (SPARK-33350) Add support to DiskBlockManager to create merge directory and to get the local shuffle merged data

[ https://issues.apache.org/jira/browse/SPARK-33350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-33350: --- Assignee: Ye Zhou > Add support to DiskBlockManager to create merge directo

[jira] [Resolved] (SPARK-33350) Add support to DiskBlockManager to create merge directory and to get the local shuffle merged data

[ https://issues.apache.org/jira/browse/SPARK-33350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-33350. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 320

[jira] [Resolved] (SPARK-32920) Add support in Spark driver to coordinate the finalization of the push/merge phase in push-based shuffle for a given shuffle and the initiation of the reduce stage

[ https://issues.apache.org/jira/browse/SPARK-32920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-32920. - Resolution: Fixed > Add support in Spark driver to coordinate the finalization o

[jira] [Updated] (SPARK-32920) Add support in Spark driver to coordinate the finalization of the push/merge phase in push-based shuffle for a given shuffle and the initiation of the reduce stage

[ https://issues.apache.org/jira/browse/SPARK-32920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-32920: Fix Version/s: 3.2.0 > Add support in Spark driver to coordinate the finalization

[jira] [Assigned] (SPARK-32920) Add support in Spark driver to coordinate the finalization of the push/merge phase in push-based shuffle for a given shuffle and the initiation of the reduce stage

[ https://issues.apache.org/jira/browse/SPARK-32920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-32920: --- Assignee: Venkata krishnan Sowrirajan > Add support in Spark driver to coor

[jira] [Resolved] (SPARK-35263) Refactor ShuffleBlockFetcherIteratorSuite to reduce duplicated code

[ https://issues.apache.org/jira/browse/SPARK-35263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-35263. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 323

[jira] [Assigned] (SPARK-35263) Refactor ShuffleBlockFetcherIteratorSuite to reduce duplicated code

[ https://issues.apache.org/jira/browse/SPARK-35263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-35263: --- Assignee: Erik Krogen > Refactor ShuffleBlockFetcherIteratorSuite to reduce

[jira] [Resolved] (SPARK-32921) Extend MapOutputTracker to support tracking and serving the metadata about each merged shuffle partitions for a given shuffle in push-based shuffle scenario

[ https://issues.apache.org/jira/browse/SPARK-32921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-32921. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 304

[jira] [Assigned] (SPARK-32921) Extend MapOutputTracker to support tracking and serving the metadata about each merged shuffle partitions for a given shuffle in push-based shuffle scenario

[ https://issues.apache.org/jira/browse/SPARK-32921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-32921: --- Assignee: Venkata krishnan Sowrirajan > Extend MapOutputTracker to support

[jira] [Assigned] (SPARK-35049) Remove unused MapOutputTracker in BlockStoreShuffleReader

[ https://issues.apache.org/jira/browse/SPARK-35049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-35049: --- Assignee: angerszhu > Remove unused MapOutputTracker in BlockStoreShuffleRe

[jira] [Resolved] (SPARK-35049) Remove unused MapOutputTracker in BlockStoreShuffleReader

[ https://issues.apache.org/jira/browse/SPARK-35049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-35049. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 321

[jira] [Assigned] (SPARK-34949) Executor.reportHeartBeat reregisters blockManager even when Executor is shutting down

[ https://issues.apache.org/jira/browse/SPARK-34949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-34949: --- Assignee: Sumeet > Executor.reportHeartBeat reregisters blockManager even w

[jira] [Resolved] (SPARK-34949) Executor.reportHeartBeat reregisters blockManager even when Executor is shutting down

[ https://issues.apache.org/jira/browse/SPARK-34949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-34949. - Fix Version/s: 3.1.2 3.2.0 Resolution: Fixed Issue res

[jira] [Resolved] (SPARK-34840) Fix cases of corruption in merged shuffle blocks that are pushed

[ https://issues.apache.org/jira/browse/SPARK-34840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-34840. - Fix Version/s: 3.1.2 3.2.0 Resolution: Fixed Issue res

[jira] [Assigned] (SPARK-34840) Fix cases of corruption in merged shuffle blocks that are pushed

[ https://issues.apache.org/jira/browse/SPARK-34840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-34840: --- Assignee: Chandni Singh > Fix cases of corruption in merged shuffle blocks

[jira] [Assigned] (SPARK-24818) Ensure all the barrier tasks in the same stage are launched together

[ https://issues.apache.org/jira/browse/SPARK-24818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-24818: --- Assignee: wuyi > Ensure all the barrier tasks in the same stage are launche

[jira] [Resolved] (SPARK-24818) Ensure all the barrier tasks in the same stage are launched together

[ https://issues.apache.org/jira/browse/SPARK-24818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-24818. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 306

[jira] [Assigned] (SPARK-34069) Kill barrier tasks should respect SPARK_JOB_INTERRUPT_ON_CANCEL

[ https://issues.apache.org/jira/browse/SPARK-34069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-34069: --- Assignee: ulysses you > Kill barrier tasks should respect SPARK_JOB_INTERRU

[jira] [Resolved] (SPARK-34069) Kill barrier tasks should respect SPARK_JOB_INTERRUPT_ON_CANCEL

[ https://issues.apache.org/jira/browse/SPARK-34069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-34069. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 311

[jira] [Assigned] (SPARK-32917) Add support for executors to push shuffle blocks after successful map task completion

[ https://issues.apache.org/jira/browse/SPARK-32917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-32917: --- Assignee: Chandni Singh > Add support for executors to push shuffle blocks

[jira] [Resolved] (SPARK-32917) Add support for executors to push shuffle blocks after successful map task completion

[ https://issues.apache.org/jira/browse/SPARK-32917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-32917. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 303

[jira] [Assigned] (SPARK-33669) Wrong error message from YARN application state monitor when sc.stop in yarn client mode

[ https://issues.apache.org/jira/browse/SPARK-33669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-33669: --- Assignee: Su Qilong > Wrong error message from YARN application state monit

[jira] [Resolved] (SPARK-33669) Wrong error message from YARN application state monitor when sc.stop in yarn client mode

[ https://issues.apache.org/jira/browse/SPARK-33669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-33669. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 306

[jira] [Resolved] (SPARK-33185) YARN: Print direct links to driver logs alongside application report in cluster mode

[ https://issues.apache.org/jira/browse/SPARK-33185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-33185. - Resolution: Fixed Issue resolved by pull request 30450 [https://github.com/apach

[jira] [Assigned] (SPARK-32918) RPC implementation to support control plane coordination for push-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-32918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-32918: --- Assignee: Ye Zhou > RPC implementation to support control plane coordinatio

[jira] [Resolved] (SPARK-32918) RPC implementation to support control plane coordination for push-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-32918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-32918. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 301

[jira] [Assigned] (SPARK-32919) Add support in Spark driver to coordinate the shuffle map stage in push-based shuffle by selecting external shuffle services for merging shuffle partitions

[ https://issues.apache.org/jira/browse/SPARK-32919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-32919: --- Assignee: Venkata krishnan Sowrirajan > Add support in Spark driver to coor

[jira] [Resolved] (SPARK-32919) Add support in Spark driver to coordinate the shuffle map stage in push-based shuffle by selecting external shuffle services for merging shuffle partitions

[ https://issues.apache.org/jira/browse/SPARK-32919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-32919. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 301

[jira] [Assigned] (SPARK-31069) high cpu caused by chunksBeingTransferred in external shuffle service

[ https://issues.apache.org/jira/browse/SPARK-31069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-31069: --- Assignee: angerszhu > high cpu caused by chunksBeingTransferred in external

[jira] [Assigned] (SPARK-31069) high cpu caused by chunksBeingTransferred in external shuffle service

[ https://issues.apache.org/jira/browse/SPARK-31069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-31069: --- Assignee: angerszhu (was: angerszhu) > high cpu caused by chunksBeingTrans

[jira] [Resolved] (SPARK-31069) high cpu caused by chunksBeingTransferred in external shuffle service

[ https://issues.apache.org/jira/browse/SPARK-31069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-31069. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 301

[jira] [Assigned] (SPARK-32915) RPC implementation to support pushing and merging shuffle blocks

[ https://issues.apache.org/jira/browse/SPARK-32915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-32915: --- Assignee: Min Shen > RPC implementation to support pushing and merging shuf

[jira] [Resolved] (SPARK-32916) Add support for external shuffle service in YARN deployment mode to leverage push-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-32916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-32916. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 300

[jira] [Assigned] (SPARK-32916) Add support for external shuffle service in YARN deployment mode to leverage push-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-32916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-32916: --- Assignee: Chandni Singh > Add support for external shuffle service in YARN

[jira] [Assigned] (SPARK-33185) YARN: Print direct links to driver logs alongside application report in cluster mode

[ https://issues.apache.org/jira/browse/SPARK-33185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-33185: --- Assignee: Erik Krogen > YARN: Print direct links to driver logs alongside a

[jira] [Resolved] (SPARK-33185) YARN: Print direct links to driver logs alongside application report in cluster mode

[ https://issues.apache.org/jira/browse/SPARK-33185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-33185. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 300

[jira] [Resolved] (SPARK-33088) Enhance ExecutorPlugin API to include methods for task start and end events

[ https://issues.apache.org/jira/browse/SPARK-33088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-33088. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 299

[jira] [Resolved] (SPARK-32915) RPC implementation to support pushing and merging shuffle blocks

[ https://issues.apache.org/jira/browse/SPARK-32915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-32915. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 298

[jira] [Updated] (SPARK-32738) thread safe endpoints may hang due to fatal error

[ https://issues.apache.org/jira/browse/SPARK-32738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-32738: Fix Version/s: 2.4.8 > thread safe endpoints may hang due to fatal error > ---

[jira] [Updated] (SPARK-32738) thread safe endpoints may hang due to fatal error

[ https://issues.apache.org/jira/browse/SPARK-32738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-32738: Fix Version/s: 3.0.2 > thread safe endpoints may hang due to fatal error > ---

[jira] [Commented] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17183622#comment-17183622 ] Mridul Muralidharan commented on SPARK-30602: - SPIP proposal document: [htt

[jira] [Assigned] (SPARK-32663) TransportClient getting closed when there are outstanding requests to the server

[ https://issues.apache.org/jira/browse/SPARK-32663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-32663: --- Assignee: Attila Zsolt Piros > TransportClient getting closed when there ar

[jira] [Resolved] (SPARK-32663) TransportClient getting closed when there are outstanding requests to the server

[ https://issues.apache.org/jira/browse/SPARK-32663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-32663. - Fix Version/s: 3.1.0 3.0.1 Resolution: Fixed Issue res

[jira] [Resolved] (SPARK-32119) ExecutorPlugin doesn't work with Standalone Cluster and Kubernetes with --jars

[ https://issues.apache.org/jira/browse/SPARK-32119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-32119. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 289

[jira] [Commented] (SPARK-25594) OOM in long running applications even with UI disabled

[ https://issues.apache.org/jira/browse/SPARK-25594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17150729#comment-17150729 ] Mridul Muralidharan commented on SPARK-25594: - Given regression in functiona

[jira] [Resolved] (SPARK-25594) OOM in long running applications even with UI disabled

[ https://issues.apache.org/jira/browse/SPARK-25594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-25594. - Resolution: Won't Fix > OOM in long running applications even with UI disabled >

[jira] [Commented] (SPARK-29302) dynamic partition overwrite with speculation enabled

[ https://issues.apache.org/jira/browse/SPARK-29302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17086124#comment-17086124 ] Mridul Muralidharan commented on SPARK-29302: - I agree with [~feiwang], it l

[jira] [Issue Comment Deleted] (SPARK-29302) dynamic partition overwrite with speculation enabled

[ https://issues.apache.org/jira/browse/SPARK-29302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-29302: Comment: was deleted (was: Drive by observations: * Speculative execution does no

[jira] [Commented] (SPARK-29302) dynamic partition overwrite with speculation enabled

[ https://issues.apache.org/jira/browse/SPARK-29302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17086117#comment-17086117 ] Mridul Muralidharan commented on SPARK-29302: - Drive by observations: * Spe

[jira] [Commented] (SPARK-26688) Provide configuration of initially blacklisted YARN nodes

[ https://issues.apache.org/jira/browse/SPARK-26688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16750477#comment-16750477 ] Mridul Muralidharan commented on SPARK-26688: - If this is a legitimate useca

[jira] [Commented] (SPARK-26688) Provide configuration of initially blacklisted YARN nodes

[ https://issues.apache.org/jira/browse/SPARK-26688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16748915#comment-16748915 ] Mridul Muralidharan commented on SPARK-26688: - What is the usecase for this

[jira] [Commented] (SPARK-25732) Allow specifying a keytab/principal for proxy user for token renewal

[ https://issues.apache.org/jira/browse/SPARK-25732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16651301#comment-16651301 ] Mridul Muralidharan commented on SPARK-25732: - [~vanzin] With long running a

[jira] [Commented] (SPARK-25594) OOM in long running applications even with UI disabled

[ https://issues.apache.org/jira/browse/SPARK-25594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16635122#comment-16635122 ] Mridul Muralidharan commented on SPARK-25594: - Task level information is req

[jira] [Created] (SPARK-25594) OOM in long running applications even with UI disabled

Mridul Muralidharan created SPARK-25594: --- Summary: OOM in long running applications even with UI disabled Key: SPARK-25594 URL: https://issues.apache.org/jira/browse/SPARK-25594 Project: Spark

[jira] [Resolved] (SPARK-24948) SHS filters wrongly some applications due to permission check

[ https://issues.apache.org/jira/browse/SPARK-24948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-24948. - Resolution: Fixed > SHS filters wrongly some applications due to permission chec

[jira] [Updated] (SPARK-24948) SHS filters wrongly some applications due to permission check

[ https://issues.apache.org/jira/browse/SPARK-24948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-24948: Fix Version/s: 2.4.0 > SHS filters wrongly some applications due to permission che

[jira] [Comment Edited] (SPARK-24375) Design sketch: support barrier scheduling in Apache Spark

[ https://issues.apache.org/jira/browse/SPARK-24375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16569363#comment-16569363 ] Mridul Muralidharan edited comment on SPARK-24375 at 8/5/18 4:42 AM: -

[jira] [Commented] (SPARK-24375) Design sketch: support barrier scheduling in Apache Spark

[ https://issues.apache.org/jira/browse/SPARK-24375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16569363#comment-16569363 ] Mridul Muralidharan commented on SPARK-24375: - {quote} We've thought hard on

[jira] [Commented] (SPARK-24375) Design sketch: support barrier scheduling in Apache Spark

[ https://issues.apache.org/jira/browse/SPARK-24375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16568664#comment-16568664 ] Mridul Muralidharan commented on SPARK-24375: - {quote} It's not desired beha

[jira] [Commented] (SPARK-24615) Accelerator-aware task scheduling for Spark

[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16551198#comment-16551198 ] Mridul Muralidharan commented on SPARK-24615: - [~tgraves] This was indeed a

[jira] [Commented] (SPARK-24755) Executor loss can cause task to not be resubmitted

[ https://issues.apache.org/jira/browse/SPARK-24755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16536016#comment-16536016 ] Mridul Muralidharan commented on SPARK-24755: - Go for it - thanks [~hthuynh2

[jira] [Updated] (SPARK-24755) Executor loss can cause task to not be resubmitted

[ https://issues.apache.org/jira/browse/SPARK-24755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-24755: Description: As part of SPARK-22074, when an executor is lost, TSM.executorLost cu

[jira] [Updated] (SPARK-24755) Executor loss can cause task to not be resubmitted

[ https://issues.apache.org/jira/browse/SPARK-24755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-24755: Summary: Executor loss can cause task to not be resubmitted (was: Executor loss c

[jira] [Updated] (SPARK-24755) Executor loss can cause task to be not resubmitted

[ https://issues.apache.org/jira/browse/SPARK-24755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-24755: Description: As part of SPARK-22074, when an executor is lost, TSM.executorLost cu

[jira] [Created] (SPARK-24755) Executor loss can cause task to be not resubmitted

Mridul Muralidharan created SPARK-24755: --- Summary: Executor loss can cause task to be not resubmitted Key: SPARK-24755 URL: https://issues.apache.org/jira/browse/SPARK-24755 Project: Spark

[jira] [Comment Edited] (SPARK-24375) Design sketch: support barrier scheduling in Apache Spark

[ https://issues.apache.org/jira/browse/SPARK-24375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16516411#comment-16516411 ] Mridul Muralidharan edited comment on SPARK-24375 at 6/18/18 10:17 PM: ---

[jira] [Commented] (SPARK-24375) Design sketch: support barrier scheduling in Apache Spark

[ https://issues.apache.org/jira/browse/SPARK-24375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16516411#comment-16516411 ] Mridul Muralidharan commented on SPARK-24375: - [~jiangxb1987] A couple of c

[jira] [Comment Edited] (SPARK-24375) Design sketch: support barrier scheduling in Apache Spark

[ https://issues.apache.org/jira/browse/SPARK-24375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16516411#comment-16516411 ] Mridul Muralidharan edited comment on SPARK-24375 at 6/18/18 10:15 PM: ---

[jira] [Updated] (SPARK-23721) Enhance BlockManagerId to include container's underlying host machine hostname

[ https://issues.apache.org/jira/browse/SPARK-23721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-23721: Summary: Enhance BlockManagerId to include container's underlying host machine host

[jira] [Updated] (SPARK-23721) Enhance BlockManagerId to include container's underlying host machie hostname

[ https://issues.apache.org/jira/browse/SPARK-23721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-23721: Summary: Enhance BlockManagerId to include container's underlying host machie hostn

[jira] [Updated] (SPARK-23721) Enhance BlockManagerId to include container's underlying host name

[ https://issues.apache.org/jira/browse/SPARK-23721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-23721: Summary: Enhance BlockManagerId to include container's underlying host name (was:

[jira] [Created] (SPARK-23721) Use actual node's hostname for host and rack locality computation

Mridul Muralidharan created SPARK-23721: --- Summary: Use actual node's hostname for host and rack locality computation Key: SPARK-23721 URL: https://issues.apache.org/jira/browse/SPARK-23721 Proje

[jira] [Created] (SPARK-23720) Leverage shuffle service when running in non-host networking mode in hadoop 3 docker support

Mridul Muralidharan created SPARK-23720: --- Summary: Leverage shuffle service when running in non-host networking mode in hadoop 3 docker support Key: SPARK-23720 URL: https://issues.apache.org/jira/browse/SPA

[jira] [Updated] (SPARK-23718) Document using docker in host networking mode in hadoop 3

[ https://issues.apache.org/jira/browse/SPARK-23718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-23718: Issue Type: Documentation (was: Task) > Document using docker in host networking m

[jira] [Created] (SPARK-23719) Use correct hostname in non-host networking mode in hadoop 3 docker support

Mridul Muralidharan created SPARK-23719: --- Summary: Use correct hostname in non-host networking mode in hadoop 3 docker support Key: SPARK-23719 URL: https://issues.apache.org/jira/browse/SPARK-23719

[jira] [Updated] (SPARK-23718) Document using docker in host networking mode in hadoop 3

[ https://issues.apache.org/jira/browse/SPARK-23718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-23718: Issue Type: Improvement (was: Documentation) > Document using docker in host netwo

[jira] [Updated] (SPARK-23718) Document using docker in host networking mode in hadoop 3

[ https://issues.apache.org/jira/browse/SPARK-23718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-23718: Issue Type: Task (was: Improvement) > Document using docker in host networking mod

[jira] [Created] (SPARK-23718) Document using docker in host networking mode in hadoop 3

Mridul Muralidharan created SPARK-23718: --- Summary: Document using docker in host networking mode in hadoop 3 Key: SPARK-23718 URL: https://issues.apache.org/jira/browse/SPARK-23718 Project: Spark

[jira] [Created] (SPARK-23717) Leverage docker support in Hadoop 3

Mridul Muralidharan created SPARK-23717: --- Summary: Leverage docker support in Hadoop 3 Key: SPARK-23717 URL: https://issues.apache.org/jira/browse/SPARK-23717 Project: Spark Issue Type:

<    1   2   3   4   5   6   7   >