[jira] [Commented] (SPARK-27396) SPIP: Public APIs for extended Columnar Processing Support

[ https://issues.apache.org/jira/browse/SPARK-27396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819153#comment-16819153 ] Thomas Graves commented on SPARK-27396: --- Since I don't hear any strong objections

[jira] [Created] (SPARK-27488) Driver interface to support GPU resources

Thomas Graves created SPARK-27488: - Summary: Driver interface to support GPU resources Key: SPARK-27488 URL: https://issues.apache.org/jira/browse/SPARK-27488 Project: Spark Issue Type: Stor

[jira] [Commented] (SPARK-27488) Driver interface to support GPU resources

[ https://issues.apache.org/jira/browse/SPARK-27488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16820119#comment-16820119 ] Thomas Graves commented on SPARK-27488: --- Note, the api design is here: https://iss

[jira] [Resolved] (SPARK-27364) User-facing APIs for GPU-aware scheduling

[ https://issues.apache.org/jira/browse/SPARK-27364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-27364. --- Resolution: Fixed > User-facing APIs for GPU-aware scheduling >

[jira] [Commented] (SPARK-27364) User-facing APIs for GPU-aware scheduling

[ https://issues.apache.org/jira/browse/SPARK-27364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16820120#comment-16820120 ] Thomas Graves commented on SPARK-27364: --- based on no comments on this I'm going to

[jira] [Reopened] (SPARK-27364) User-facing APIs for GPU-aware scheduling

[ https://issues.apache.org/jira/browse/SPARK-27364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reopened SPARK-27364: --- reopening since it has a subtask > User-facing APIs for GPU-aware scheduling >

[jira] [Created] (SPARK-27489) UI updates to show executor resource information

Thomas Graves created SPARK-27489: - Summary: UI updates to show executor resource information Key: SPARK-27489 URL: https://issues.apache.org/jira/browse/SPARK-27489 Project: Spark Issue Type

[jira] [Created] (SPARK-27492) High level user documentation

Thomas Graves created SPARK-27492: - Summary: High level user documentation Key: SPARK-27492 URL: https://issues.apache.org/jira/browse/SPARK-27492 Project: Spark Issue Type: Story C

[jira] [Updated] (SPARK-27495) Support Stage level resource configuration and scheduling

[ https://issues.apache.org/jira/browse/SPARK-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-27495: -- Summary: Support Stage level resource configuration and scheduling (was: Support Stage level

[jira] [Created] (SPARK-27495) Support Stage level resource scheduling

Thomas Graves created SPARK-27495: - Summary: Support Stage level resource scheduling Key: SPARK-27495 URL: https://issues.apache.org/jira/browse/SPARK-27495 Project: Spark Issue Type: Story

[jira] [Commented] (SPARK-24655) [K8S] Custom Docker Image Expectations and Documentation

[ https://issues.apache.org/jira/browse/SPARK-24655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16821270#comment-16821270 ] Thomas Graves commented on SPARK-24655: --- >From the linked issues it seems the goal

[jira] [Commented] (SPARK-27492) High level user documentation

[ https://issues.apache.org/jira/browse/SPARK-27492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16821900#comment-16821900 ] Thomas Graves commented on SPARK-27492: --- Sorry, it is under the epic and didn't re

[jira] [Updated] (SPARK-27492) GPU scheduling - High level user documentation

[ https://issues.apache.org/jira/browse/SPARK-27492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-27492: -- Summary: GPU scheduling - High level user documentation (was: High level user documentation)

[jira] [Updated] (SPARK-27492) GPU scheduling - High level user documentation

[ https://issues.apache.org/jira/browse/SPARK-27492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-27492: -- Description: For the SPIP - Accelerator-aware task scheduling for Spark,  https://issues.apache

[jira] [Commented] (SPARK-27495) Support Stage level resource configuration and scheduling

[ https://issues.apache.org/jira/browse/SPARK-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16822166#comment-16822166 ] Thomas Graves commented on SPARK-27495: --- Unfortunately the link to the original de

[jira] [Comment Edited] (SPARK-27495) Support Stage level resource configuration and scheduling

[ https://issues.apache.org/jira/browse/SPARK-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16822166#comment-16822166 ] Thomas Graves edited comment on SPARK-27495 at 4/19/19 8:28 PM: --

[jira] [Comment Edited] (SPARK-27495) Support Stage level resource configuration and scheduling

[ https://issues.apache.org/jira/browse/SPARK-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16822166#comment-16822166 ] Thomas Graves edited comment on SPARK-27495 at 4/19/19 8:29 PM: --

[jira] [Commented] (SPARK-27396) SPIP: Public APIs for extended Columnar Processing Support

[ https://issues.apache.org/jira/browse/SPARK-27396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16823097#comment-16823097 ] Thomas Graves commented on SPARK-27396: --- thanks for the questions and commenting,

[jira] [Commented] (SPARK-27396) SPIP: Public APIs for extended Columnar Processing Support

[ https://issues.apache.org/jira/browse/SPARK-27396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16835014#comment-16835014 ] Thomas Graves commented on SPARK-27396: --- I just started a vote thread on this SPIP

[jira] [Updated] (SPARK-27492) GPU scheduling - High level user documentation

[ https://issues.apache.org/jira/browse/SPARK-27492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-27492: -- Description: For the SPIP - Accelerator-aware task scheduling for Spark,  https://issues.apach

[jira] [Commented] (SPARK-27520) Introduce a global config system to replace hadoopConfiguration

[ https://issues.apache.org/jira/browse/SPARK-27520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16836730#comment-16836730 ] Thomas Graves commented on SPARK-27520: --- Can we add more to the description to exp

[jira] [Commented] (SPARK-18085) Scalability enhancements for the History Server

[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15608719#comment-15608719 ] Thomas Graves commented on SPARK-18085: --- So there are a few things unclear to me ab

[jira] [Commented] (SPARK-18085) Scalability enhancements for the History Server

[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15608724#comment-15608724 ] Thomas Graves commented on SPARK-18085: --- Hadn't refreshed and seen your comment abo

[jira] [Commented] (SPARK-18085) Scalability enhancements for the History Server

[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15608760#comment-15608760 ] Thomas Graves commented on SPARK-18085: --- I should also say I don't think we should

[jira] [Commented] (SPARK-18085) Scalability enhancements for the History Server

[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15609099#comment-15609099 ] Thomas Graves commented on SPARK-18085: --- Can you expand on is the source of most of

[jira] [Commented] (SPARK-18085) Scalability enhancements for the History Server

[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15612086#comment-15612086 ] Thomas Graves commented on SPARK-18085: --- Perhaps we can clarify the title on this j

[jira] [Resolved] (SPARK-18099) Spark distributed cache should throw exception if same file is specified to dropped in --files --archives

[ https://issues.apache.org/jira/browse/SPARK-18099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-18099. --- Resolution: Fixed Assignee: Kishor Patil Fix Version/s: 2.2.0

[jira] [Created] (SPARK-18357) YARN --files/--archives broke

Thomas Graves created SPARK-18357: - Summary: YARN --files/--archives broke Key: SPARK-18357 URL: https://issues.apache.org/jira/browse/SPARK-18357 Project: Spark Issue Type: Bug Com

[jira] [Updated] (SPARK-18357) YARN --files/--archives broke

[ https://issues.apache.org/jira/browse/SPARK-18357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-18357: -- Priority: Blocker (was: Major) > YARN --files/--archives broke > -

[jira] [Commented] (SPARK-18357) YARN --files/--archives broke

[ https://issues.apache.org/jira/browse/SPARK-18357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15647941#comment-15647941 ] Thomas Graves commented on SPARK-18357: --- [~kishorvpatil] can you fix please > YARN

[jira] [Updated] (SPARK-18357) YARN --files/--archives broke

[ https://issues.apache.org/jira/browse/SPARK-18357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-18357: -- Assignee: Kishor Patil > YARN --files/--archives broke > - > >

[jira] [Resolved] (SPARK-18357) YARN --files/--archives broke

[ https://issues.apache.org/jira/browse/SPARK-18357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-18357. --- Resolution: Fixed Fix Version/s: 2.2.0 2.1.0 > YARN --files/--archi

[jira] [Resolved] (SPARK-17843) History Server gives no feedback about the application list being incomplete

[ https://issues.apache.org/jira/browse/SPARK-17843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-17843. --- Resolution: Fixed Assignee: Vinayak Joshi Fix Version/s: 2.2.0

[jira] [Commented] (SPARK-18085) Better History Server scalability for many / large applications

[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15667226#comment-15667226 ] Thomas Graves commented on SPARK-18085: --- I think at a high level its fine. You men

[jira] [Closed] (SPARK-3374) Spark on Yarn remove deprecated configs for 2.0

[ https://issues.apache.org/jira/browse/SPARK-3374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves closed SPARK-3374. Resolution: Duplicate > Spark on Yarn remove deprecated configs for 2.0 > --

[jira] [Commented] (SPARK-3374) Spark on Yarn remove deprecated configs for 2.0

[ https://issues.apache.org/jira/browse/SPARK-3374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15260492#comment-15260492 ] Thomas Graves commented on SPARK-3374: -- I think these were all removed under SPARK-12

[jira] [Created] (SPARK-14963) YarnShuffleService should use YARN getRecoveryPath() for leveldb location

Thomas Graves created SPARK-14963: - Summary: YarnShuffleService should use YARN getRecoveryPath() for leveldb location Key: SPARK-14963 URL: https://issues.apache.org/jira/browse/SPARK-14963 Project:

[jira] [Updated] (SPARK-14963) YarnShuffleService should use YARN getRecoveryPath() for leveldb location

[ https://issues.apache.org/jira/browse/SPARK-14963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-14963: -- Issue Type: Improvement (was: Bug) > YarnShuffleService should use YARN getRecoveryPath() for

[jira] [Resolved] (SPARK-6735) Provide options to make maximum executor failure count ( which kills the application ) relative to a window duration or disable it.

[ https://issues.apache.org/jira/browse/SPARK-6735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-6735. -- Resolution: Fixed Fix Version/s: 2.0.0 > Provide options to make maximum executor failure

[jira] [Updated] (SPARK-6735) Provide options to make maximum executor failure count ( which kills the application ) relative to a window duration or disable it.

[ https://issues.apache.org/jira/browse/SPARK-6735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-6735: - Assignee: Saisai Shao > Provide options to make maximum executor failure count ( which kills the

[jira] [Commented] (SPARK-1989) Exit executors faster if they get into a cycle of heavy GC

[ https://issues.apache.org/jira/browse/SPARK-1989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15262761#comment-15262761 ] Thomas Graves commented on SPARK-1989: -- Personally I don't agree with this and think

[jira] [Commented] (SPARK-14963) YarnShuffleService should use YARN getRecoveryPath() for leveldb location

[ https://issues.apache.org/jira/browse/SPARK-14963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15264010#comment-15264010 ] Thomas Graves commented on SPARK-14963: --- I'm definitely fine with it but someone el

[jira] [Commented] (SPARK-11316) coalesce doesn't handle UnionRDD with partial locality properly

[ https://issues.apache.org/jira/browse/SPARK-11316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15264205#comment-15264205 ] Thomas Graves commented on SPARK-11316: --- Simple steps to reproduce an RDD with part

[jira] [Updated] (SPARK-4224) Support group acls

[ https://issues.apache.org/jira/browse/SPARK-4224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-4224: - Target Version/s: 2.0.0 > Support group acls > -- > > Key: SPARK-4

[jira] [Updated] (SPARK-4224) Support group acls

[ https://issues.apache.org/jira/browse/SPARK-4224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-4224: - Assignee: Dhruve Ashar > Support group acls > -- > > Key: SPARK-42

[jira] [Resolved] (SPARK-4224) Support group acls

[ https://issues.apache.org/jira/browse/SPARK-4224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-4224. -- Resolution: Fixed Fix Version/s: 2.0.0 > Support group acls > -- > >

[jira] [Created] (SPARK-15121) Improve logging of external shuffle handler

Thomas Graves created SPARK-15121: - Summary: Improve logging of external shuffle handler Key: SPARK-15121 URL: https://issues.apache.org/jira/browse/SPARK-15121 Project: Spark Issue Type: Imp

[jira] [Updated] (SPARK-15121) Improve logging of external shuffle handler

[ https://issues.apache.org/jira/browse/SPARK-15121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-15121: -- Assignee: (was: Thomas Graves) > Improve logging of external shuffle handler >

[jira] [Commented] (SPARK-14963) YarnShuffleService should use YARN getRecoveryPath() for leveldb location

[ https://issues.apache.org/jira/browse/SPARK-14963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15272466#comment-15272466 ] Thomas Graves commented on SPARK-14963: --- [~jerryshao] I think the other pr is being

[jira] [Created] (SPARK-15178) Remove LazyFileRegion

Thomas Graves created SPARK-15178: - Summary: Remove LazyFileRegion Key: SPARK-15178 URL: https://issues.apache.org/jira/browse/SPARK-15178 Project: Spark Issue Type: Improvement Com

[jira] [Updated] (SPARK-14963) YarnShuffleService should use YARN getRecoveryPath() for leveldb location

[ https://issues.apache.org/jira/browse/SPARK-14963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-14963: -- Assignee: Saisai Shao > YarnShuffleService should use YARN getRecoveryPath() for leveldb locati

[jira] [Resolved] (SPARK-14963) YarnShuffleService should use YARN getRecoveryPath() for leveldb location

[ https://issues.apache.org/jira/browse/SPARK-14963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-14963. --- Resolution: Fixed Fix Version/s: 2.1.0 > YarnShuffleService should use YARN getRecover

[jira] [Commented] (SPARK-4924) Factor out code to launch Spark applications into a separate library

[ https://issues.apache.org/jira/browse/SPARK-4924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15284483#comment-15284483 ] Thomas Graves commented on SPARK-4924: -- [~javadba] If you have ideas on improving th

[jira] [Created] (SPARK-15405) YARN uploading the same __spark_conf__.zip twice

Thomas Graves created SPARK-15405: - Summary: YARN uploading the same __spark_conf__.zip twice Key: SPARK-15405 URL: https://issues.apache.org/jira/browse/SPARK-15405 Project: Spark Issue Type

[jira] [Commented] (SPARK-15405) YARN uploading the same __spark_conf__.zip twice

[ https://issues.apache.org/jira/browse/SPARK-15405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15291180#comment-15291180 ] Thomas Graves commented on SPARK-15405: --- [~vanzin] I missed this when I did the re

[jira] [Created] (SPARK-15410) spark-submit --help throws exception

Thomas Graves created SPARK-15410: - Summary: spark-submit --help throws exception Key: SPARK-15410 URL: https://issues.apache.org/jira/browse/SPARK-15410 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-14331) Exceptions saving to parquetFile after join from dataframes in master

[ https://issues.apache.org/jira/browse/SPARK-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15296416#comment-15296416 ] Thomas Graves commented on SPARK-14331: --- I was trying to reproduce this to get you

[jira] [Commented] (SPARK-14331) Exceptions saving to parquetFile after join from dataframes in master

[ https://issues.apache.org/jira/browse/SPARK-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15296440#comment-15296440 ] Thomas Graves commented on SPARK-14331: --- Note I was running spark on yarn. > Excep

[jira] [Updated] (SPARK-14279) Improve the spark build to pick the version information from the pom file and add git commit information

[ https://issues.apache.org/jira/browse/SPARK-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-14279: -- Assignee: (was: Sanket Reddy) > Improve the spark build to pick the version information fro

[jira] [Commented] (SPARK-14331) Exceptions saving to parquetFile after join from dataframes in master

[ https://issues.apache.org/jira/browse/SPARK-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15296705#comment-15296705 ] Thomas Graves commented on SPARK-14331: --- that might be https://github.com/apache/sp

[jira] [Commented] (SPARK-34167) Reading parquet with Decimal(8,2) written as a Decimal64 blows up

[ https://issues.apache.org/jira/browse/SPARK-34167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17270199#comment-17270199 ] Thomas Graves commented on SPARK-34167: --- [~razajafri] can you finish detailing you

[jira] [Commented] (SPARK-27658) Catalog API to load functions

[ https://issues.apache.org/jira/browse/SPARK-27658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17281984#comment-17281984 ] Thomas Graves commented on SPARK-27658: --- can we link the SPIP to this  > Catalog

[jira] [Resolved] (SPARK-33739) Jobs committed through the S3A Magic committer don't report the bytes written

[ https://issues.apache.org/jira/browse/SPARK-33739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-33739. --- Fix Version/s: 3.2.0 Assignee: Steve Loughran Resolution: Fixed > Jobs commi

[jira] [Commented] (SPARK-34461) Collect API feedback and maybe revise some APIs

[ https://issues.apache.org/jira/browse/SPARK-34461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17286517#comment-17286517 ] Thomas Graves commented on SPARK-34461: --- So the discussions in the original SPIP a

[jira] [Commented] (SPARK-29329) maven incremental builds not working

[ https://issues.apache.org/jira/browse/SPARK-29329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17293005#comment-17293005 ] Thomas Graves commented on SPARK-29329: --- Yeah I know 4.3.0 made things better. Its

[jira] [Resolved] (SPARK-27495) SPIP: Support Stage level resource configuration and scheduling

[ https://issues.apache.org/jira/browse/SPARK-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-27495. --- Fix Version/s: 3.1.1 Target Version/s: (was: 3.2.0) Resolution: Fixed the

[jira] [Commented] (SPARK-33288) Support k8s cluster manager with stage level scheduling

[ https://issues.apache.org/jira/browse/SPARK-33288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17296060#comment-17296060 ] Thomas Graves commented on SPARK-33288: --- It looks like the version you are trying

[jira] [Commented] (SPARK-34645) [K8S] Driver pod stuck in Running state after job completes

[ https://issues.apache.org/jira/browse/SPARK-34645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17296340#comment-17296340 ] Thomas Graves commented on SPARK-34645: --- [~dongjoon] [~holden]. have you seen this

[jira] [Resolved] (SPARK-34828) YARN Shuffle Service: Support configurability of aux service name and service-specific config overrides

[ https://issues.apache.org/jira/browse/SPARK-34828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-34828. --- Fix Version/s: 3.2.0 Assignee: Erik Krogen Resolution: Fixed > YARN Shuffle

[jira] [Updated] (SPARK-36772) FinalizeShuffleMerge fails with an exception due to attempt id not matching

[ https://issues.apache.org/jira/browse/SPARK-36772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-36772: -- Target Version/s: 3.2.0 > FinalizeShuffleMerge fails with an exception due to attempt id not m

[jira] [Commented] (SPARK-35672) Spark fails to launch executors with very large user classpath lists on YARN

[ https://issues.apache.org/jira/browse/SPARK-35672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17419312#comment-17419312 ] Thomas Graves commented on SPARK-35672: --- Ok, sounds like we should revert then so

[jira] [Reopened] (SPARK-35672) Spark fails to launch executors with very large user classpath lists on YARN

[ https://issues.apache.org/jira/browse/SPARK-35672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reopened SPARK-35672: --- > Spark fails to launch executors with very large user classpath lists on YARN > ---

[jira] [Commented] (SPARK-36817) Does Apache Spark 3 support GPU usage for Spark RDDs?

[ https://issues.apache.org/jira/browse/SPARK-36817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17420737#comment-17420737 ] Thomas Graves commented on SPARK-36817: --- please refer to https://github.com/NVIDIA

[jira] [Resolved] (SPARK-36624) When application killed, sc should not exit with code 0

[ https://issues.apache.org/jira/browse/SPARK-36624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-36624. --- Fix Version/s: 3.3.0 Assignee: angerszhu Resolution: Fixed > When applicatio

[jira] [Resolved] (SPARK-36540) AM should not just finish with Success when dissconnected

[ https://issues.apache.org/jira/browse/SPARK-36540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-36540. --- Fix Version/s: 3.3.0 Assignee: angerszhu Resolution: Fixed > AM should not j

[jira] [Commented] (SPARK-37208) Support mapping Spark gpu/fpga resource types to custom YARN resource type

[ https://issues.apache.org/jira/browse/SPARK-37208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17438784#comment-17438784 ] Thomas Graves commented on SPARK-37208: --- Note, I'm working on this. > Support map

[jira] [Created] (SPARK-37208) Support mapping Spark gpu/fpga resource types to custom YARN resource type

Thomas Graves created SPARK-37208: - Summary: Support mapping Spark gpu/fpga resource types to custom YARN resource type Key: SPARK-37208 URL: https://issues.apache.org/jira/browse/SPARK-37208 Project:

[jira] [Created] (SPARK-37260) PYSPARK Arrow 3.2.0 docs link invalid

Thomas Graves created SPARK-37260: - Summary: PYSPARK Arrow 3.2.0 docs link invalid Key: SPARK-37260 URL: https://issues.apache.org/jira/browse/SPARK-37260 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-37461) yarn-client mode client's appid value is null

[ https://issues.apache.org/jira/browse/SPARK-37461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17451172#comment-17451172 ] Thomas Graves commented on SPARK-37461: --- [~angerszhuuu] please add a description t

[jira] [Updated] (SPARK-38194) Make memory overhead factor configurable

[ https://issues.apache.org/jira/browse/SPARK-38194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-38194: -- Fix Version/s: 3.3.0 > Make memory overhead factor configurable >

[jira] [Resolved] (SPARK-37618) Support cleaning up shuffle blocks from external shuffle service

[ https://issues.apache.org/jira/browse/SPARK-37618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-37618. --- Fix Version/s: 3.3.0 Assignee: Adam Binford Resolution: Fixed > Support clea

[jira] [Created] (SPARK-38677) pyspark hangs in local mode running rdd map operation

Thomas Graves created SPARK-38677: - Summary: pyspark hangs in local mode running rdd map operation Key: SPARK-38677 URL: https://issues.apache.org/jira/browse/SPARK-38677 Project: Spark Issue

[jira] [Updated] (SPARK-38677) pyspark hangs in local mode running rdd map operation

[ https://issues.apache.org/jira/browse/SPARK-38677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-38677: -- Affects Version/s: 3.3.0 > pyspark hangs in local mode running rdd map operation > ---

[jira] [Commented] (SPARK-38677) pyspark hangs in local mode running rdd map operation

[ https://issues.apache.org/jira/browse/SPARK-38677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17513536#comment-17513536 ] Thomas Graves commented on SPARK-38677: --- Note, if you kill the python.daemon proce

[jira] [Updated] (SPARK-38677) pyspark hangs in local mode running rdd map operation

[ https://issues.apache.org/jira/browse/SPARK-38677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-38677: -- Description: In spark 3.2.1 (spark 3.2.0 doesn't show this issue), pyspark will hang when run

[jira] [Commented] (SPARK-38955) from_csv can corrupt surrounding lines if a lineSep is in the data

[ https://issues.apache.org/jira/browse/SPARK-38955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17524985#comment-17524985 ] Thomas Graves commented on SPARK-38955: --- the from_csv docs point to the data sourc

[jira] [Updated] (SPARK-38955) from_csv can corrupt surrounding lines if a lineSep is in the data

[ https://issues.apache.org/jira/browse/SPARK-38955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-38955: -- Labels: corr (was: ) > from_csv can corrupt surrounding lines if a lineSep is in the data > -

[jira] [Updated] (SPARK-38955) from_csv can corrupt surrounding lines if a lineSep is in the data

[ https://issues.apache.org/jira/browse/SPARK-38955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-38955: -- Priority: Blocker (was: Major) > from_csv can corrupt surrounding lines if a lineSep is in th

[jira] [Updated] (SPARK-38955) from_csv can corrupt surrounding lines if a lineSep is in the data

[ https://issues.apache.org/jira/browse/SPARK-38955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-38955: -- Labels: (was: corr) > from_csv can corrupt surrounding lines if a lineSep is in the data > -

[jira] [Created] (SPARK-39363) fix spark.kubernetes.memoryOverheadFactor deprecation warning

Thomas Graves created SPARK-39363: - Summary: fix spark.kubernetes.memoryOverheadFactor deprecation warning Key: SPARK-39363 URL: https://issues.apache.org/jira/browse/SPARK-39363 Project: Spark

[jira] [Updated] (SPARK-39363) fix spark.kubernetes.memoryOverheadFactor deprecation warning

[ https://issues.apache.org/jira/browse/SPARK-39363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-39363: -- Description: see [https://github.com/apache/spark/pull/36744] for details.   It looks like w

[jira] [Commented] (SPARK-39363) fix spark.kubernetes.memoryOverheadFactor deprecation warning

[ https://issues.apache.org/jira/browse/SPARK-39363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17545479#comment-17545479 ] Thomas Graves commented on SPARK-39363: --- [~Kimahriman]  > fix spark.kubernetes.me

[jira] [Updated] (SPARK-39434) Provide runtime error query context when array index is out of bound

[ https://issues.apache.org/jira/browse/SPARK-39434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-39434: -- Fix Version/s: 3.4.0 > Provide runtime error query context when array index is out of bound >

[jira] [Commented] (SPARK-39107) Silent change in regexp_replace's handling of empty strings

[ https://issues.apache.org/jira/browse/SPARK-39107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17555111#comment-17555111 ] Thomas Graves commented on SPARK-39107: --- [~srowen]   I think this actually went in

[jira] [Created] (SPARK-39491) Hadoop 2.7 build fails due to org.apache.hadoop.yarn.api.records.NodeState.DECOMMISSIONING

Thomas Graves created SPARK-39491: - Summary: Hadoop 2.7 build fails due to org.apache.hadoop.yarn.api.records.NodeState.DECOMMISSIONING Key: SPARK-39491 URL: https://issues.apache.org/jira/browse/SPARK-39491

[jira] [Commented] (SPARK-34989) Improve the performance of mapChildren and withNewChildren methods

[ https://issues.apache.org/jira/browse/SPARK-34989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17319679#comment-17319679 ] Thomas Graves commented on SPARK-34989: --- I just saw this go, all the performance n

[jira] [Updated] (SPARK-35108) Pickle produces incorrect key labels for GenericRowWithSchema (data corruption)

[ https://issues.apache.org/jira/browse/SPARK-35108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-35108: -- Priority: Blocker (was: Major) > Pickle produces incorrect key labels for GenericRowWithSchem

[jira] [Updated] (SPARK-35108) Pickle produces incorrect key labels for GenericRowWithSchema (data corruption)

[ https://issues.apache.org/jira/browse/SPARK-35108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-35108: -- Labels: correctness (was: ) > Pickle produces incorrect key labels for GenericRowWithSchema (

[jira] [Commented] (SPARK-35108) Pickle produces incorrect key labels for GenericRowWithSchema (data corruption)

[ https://issues.apache.org/jira/browse/SPARK-35108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17325252#comment-17325252 ] Thomas Graves commented on SPARK-35108: --- seems like a correctness issue in some ca

[jira] [Resolved] (SPARK-34877) Add Spark AM Log link in case of master as yarn and deploy mode as client

[ https://issues.apache.org/jira/browse/SPARK-34877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-34877. --- Fix Version/s: 3.2.0 Assignee: Saurabh Chawla Resolution: Fixed > Add Spark

<    1   2   3   4   5   6   7   8   9   10   >