[jira] [Resolved] (SPARK-35881) [SQL] AQE does not support columnar execution for the final query stage

[ https://issues.apache.org/jira/browse/SPARK-35881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-35881. --- Fix Version/s: 3.3.0 3.2.0 Resolution: Fixed > [SQL] AQE does not

[jira] [Assigned] (SPARK-35881) [SQL] AQE does not support columnar execution for the final query stage

[ https://issues.apache.org/jira/browse/SPARK-35881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-35881: - Assignee: Andy Grove > [SQL] AQE does not support columnar execution for the final quer

[jira] [Commented] (SPARK-25075) Build and test Spark against Scala 2.13

[ https://issues.apache.org/jira/browse/SPARK-25075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383529#comment-17383529 ] Thomas Graves commented on SPARK-25075: --- Just wanted to check the plans for scala

[jira] [Commented] (SPARK-32333) Drop references to Master

[ https://issues.apache.org/jira/browse/SPARK-32333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17378093#comment-17378093 ] Thomas Graves commented on SPARK-32333: --- Getting back to this, now that spark 3.2

[jira] [Commented] (SPARK-33031) scheduler with blacklisting doesn't appear to pick up new executor added

[ https://issues.apache.org/jira/browse/SPARK-33031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17371592#comment-17371592 ] Thomas Graves commented on SPARK-33031: --- I do not think its resolved but haven't t

[jira] [Resolved] (SPARK-35672) Spark fails to launch executors with very large user classpath lists on YARN

[ https://issues.apache.org/jira/browse/SPARK-35672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-35672. --- Fix Version/s: 3.2.0 Assignee: Erik Krogen Resolution: Fixed > Spark fails t

[jira] [Assigned] (SPARK-35391) Memory leak in ExecutorAllocationListener breaks dynamic allocation under high load

[ https://issues.apache.org/jira/browse/SPARK-35391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-35391: - Assignee: Vasily Kolpakov > Memory leak in ExecutorAllocationListener breaks dynamic al

[jira] [Resolved] (SPARK-35391) Memory leak in ExecutorAllocationListener breaks dynamic allocation under high load

[ https://issues.apache.org/jira/browse/SPARK-35391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-35391. --- Fix Version/s: 3.1.3 3.2.0 Resolution: Fixed > Memory leak in Exec

[jira] [Assigned] (SPARK-35074) spark.jars.xxx configs should be moved to config/package.scala

[ https://issues.apache.org/jira/browse/SPARK-35074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-35074: - Assignee: dc-heros > spark.jars.xxx configs should be moved to config/package.scala > -

[jira] [Resolved] (SPARK-35074) spark.jars.xxx configs should be moved to config/package.scala

[ https://issues.apache.org/jira/browse/SPARK-35074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-35074. --- Fix Version/s: 3.2.0 Resolution: Fixed > spark.jars.xxx configs should be moved to co

[jira] [Resolved] (SPARK-35093) AQE columnar mismatch on exchange reuse

[ https://issues.apache.org/jira/browse/SPARK-35093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-35093. --- Fix Version/s: 3.2.0 3.1.2 3.0.3 Assignee: And

[jira] [Resolved] (SPARK-34472) SparkContext.addJar with an ivy path fails in cluster mode with a custom ivySettings file

[ https://issues.apache.org/jira/browse/SPARK-34472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-34472. --- Fix Version/s: 3.2.0 Assignee: Shardul Mahadik Resolution: Fixed > SparkCont

[jira] [Resolved] (SPARK-34877) Add Spark AM Log link in case of master as yarn and deploy mode as client

[ https://issues.apache.org/jira/browse/SPARK-34877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-34877. --- Fix Version/s: 3.2.0 Assignee: Saurabh Chawla Resolution: Fixed > Add Spark

[jira] [Commented] (SPARK-35108) Pickle produces incorrect key labels for GenericRowWithSchema (data corruption)

[ https://issues.apache.org/jira/browse/SPARK-35108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17325252#comment-17325252 ] Thomas Graves commented on SPARK-35108: --- seems like a correctness issue in some ca

[jira] [Updated] (SPARK-35108) Pickle produces incorrect key labels for GenericRowWithSchema (data corruption)

[ https://issues.apache.org/jira/browse/SPARK-35108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-35108: -- Labels: correctness (was: ) > Pickle produces incorrect key labels for GenericRowWithSchema (

[jira] [Updated] (SPARK-35108) Pickle produces incorrect key labels for GenericRowWithSchema (data corruption)

[ https://issues.apache.org/jira/browse/SPARK-35108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-35108: -- Priority: Blocker (was: Major) > Pickle produces incorrect key labels for GenericRowWithSchem

[jira] [Commented] (SPARK-34989) Improve the performance of mapChildren and withNewChildren methods

[ https://issues.apache.org/jira/browse/SPARK-34989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17319679#comment-17319679 ] Thomas Graves commented on SPARK-34989: --- I just saw this go, all the performance n

[jira] [Resolved] (SPARK-34828) YARN Shuffle Service: Support configurability of aux service name and service-specific config overrides

[ https://issues.apache.org/jira/browse/SPARK-34828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-34828. --- Fix Version/s: 3.2.0 Assignee: Erik Krogen Resolution: Fixed > YARN Shuffle

[jira] [Commented] (SPARK-34645) [K8S] Driver pod stuck in Running state after job completes

[ https://issues.apache.org/jira/browse/SPARK-34645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17296340#comment-17296340 ] Thomas Graves commented on SPARK-34645: --- [~dongjoon] [~holden]. have you seen this

[jira] [Commented] (SPARK-33288) Support k8s cluster manager with stage level scheduling

[ https://issues.apache.org/jira/browse/SPARK-33288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17296060#comment-17296060 ] Thomas Graves commented on SPARK-33288: --- It looks like the version you are trying

[jira] [Resolved] (SPARK-27495) SPIP: Support Stage level resource configuration and scheduling

[ https://issues.apache.org/jira/browse/SPARK-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-27495. --- Fix Version/s: 3.1.1 Target Version/s: (was: 3.2.0) Resolution: Fixed the

[jira] [Commented] (SPARK-29329) maven incremental builds not working

[ https://issues.apache.org/jira/browse/SPARK-29329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17293005#comment-17293005 ] Thomas Graves commented on SPARK-29329: --- Yeah I know 4.3.0 made things better. Its

[jira] [Commented] (SPARK-34461) Collect API feedback and maybe revise some APIs

[ https://issues.apache.org/jira/browse/SPARK-34461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17286517#comment-17286517 ] Thomas Graves commented on SPARK-34461: --- So the discussions in the original SPIP a

[jira] [Resolved] (SPARK-33739) Jobs committed through the S3A Magic committer don't report the bytes written

[ https://issues.apache.org/jira/browse/SPARK-33739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-33739. --- Fix Version/s: 3.2.0 Assignee: Steve Loughran Resolution: Fixed > Jobs commi

[jira] [Commented] (SPARK-27658) Catalog API to load functions

[ https://issues.apache.org/jira/browse/SPARK-27658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17281984#comment-17281984 ] Thomas Graves commented on SPARK-27658: --- can we link the SPIP to this  > Catalog

[jira] [Commented] (SPARK-34167) Reading parquet with Decimal(8,2) written as a Decimal64 blows up

[ https://issues.apache.org/jira/browse/SPARK-34167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17270199#comment-17270199 ] Thomas Graves commented on SPARK-34167: --- [~razajafri] can you finish detailing you

[jira] [Resolved] (SPARK-33741) Add minimum threshold speculation config

[ https://issues.apache.org/jira/browse/SPARK-33741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-33741. --- Fix Version/s: 3.2.0 Assignee: Sanket Chintapalli Resolution: Fixed > Add mi

[jira] [Commented] (SPARK-33741) Add minimum threshold speculation config

[ https://issues.apache.org/jira/browse/SPARK-33741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17260852#comment-17260852 ] Thomas Graves commented on SPARK-33741: --- probably dup of https://issues.apache.org

[jira] [Commented] (SPARK-32165) SessionState leaks SparkListener with multiple SparkSession

[ https://issues.apache.org/jira/browse/SPARK-32165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17259815#comment-17259815 ] Thomas Graves commented on SPARK-32165: --- In my opinion, I'd rather not have to go

[jira] [Commented] (SPARK-32165) SessionState leaks SparkListener with multiple SparkSession

[ https://issues.apache.org/jira/browse/SPARK-32165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17259753#comment-17259753 ] Thomas Graves commented on SPARK-32165: --- could we add a bit more detail here as to

[jira] [Commented] (SPARK-33031) scheduler with blacklisting doesn't appear to pick up new executor added

[ https://issues.apache.org/jira/browse/SPARK-33031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17258498#comment-17258498 ] Thomas Graves commented on SPARK-33031: --- ah that could be the case, but if that is

[jira] [Commented] (SPARK-33614) Fix the constant folding rule to skip it if the expression fails to execute

[ https://issues.apache.org/jira/browse/SPARK-33614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17246678#comment-17246678 ] Thomas Graves commented on SPARK-33614: --- can you add a description here with more

[jira] [Created] (SPARK-33715) Error about cores being limiting resource confusing

Thomas Graves created SPARK-33715: - Summary: Error about cores being limiting resource confusing Key: SPARK-33715 URL: https://issues.apache.org/jira/browse/SPARK-33715 Project: Spark Issue T

[jira] [Commented] (SPARK-27495) SPIP: Support Stage level resource configuration and scheduling

[ https://issues.apache.org/jira/browse/SPARK-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17245951#comment-17245951 ] Thomas Graves commented on SPARK-27495: --- the main functionality is all in, it prob

[jira] [Updated] (SPARK-33504) The application log in the Spark history server contains sensitive attributes such as password that should be redated instead of plain text

[ https://issues.apache.org/jira/browse/SPARK-33504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-33504: -- Fix Version/s: (was: 3.0.2) > The application log in the Spark history server contains sen

[jira] [Resolved] (SPARK-33504) The application log in the Spark history server contains sensitive attributes such as password that should be redated instead of plain text

[ https://issues.apache.org/jira/browse/SPARK-33504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-33504. --- Fix Version/s: 3.1.0 3.0.2 Assignee: akiyamaneko Resolutio

[jira] [Commented] (SPARK-33544) explode should not filter when used with CreateArray

[ https://issues.apache.org/jira/browse/SPARK-33544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17238371#comment-17238371 ] Thomas Graves commented on SPARK-33544: --- I'm working on a patch for this. > explo

[jira] [Created] (SPARK-33544) explode should not filter when used with CreateArray

Thomas Graves created SPARK-33544: - Summary: explode should not filter when used with CreateArray Key: SPARK-33544 URL: https://issues.apache.org/jira/browse/SPARK-33544 Project: Spark Issue

[jira] [Created] (SPARK-33462) ResourceProfile use Int for memory in ExecutorResourcesOrDefaults

Thomas Graves created SPARK-33462: - Summary: ResourceProfile use Int for memory in ExecutorResourcesOrDefaults Key: SPARK-33462 URL: https://issues.apache.org/jira/browse/SPARK-33462 Project: Spark

[jira] [Resolved] (SPARK-33288) Support k8s cluster manager with stage level scheduling

[ https://issues.apache.org/jira/browse/SPARK-33288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-33288. --- Fix Version/s: 3.1.0 Assignee: Thomas Graves Resolution: Fixed > Support k8s

[jira] [Created] (SPARK-33447) Stage level scheduling, allow specifying other spark configs via ResourceProfile

Thomas Graves created SPARK-33447: - Summary: Stage level scheduling, allow specifying other spark configs via ResourceProfile Key: SPARK-33447 URL: https://issues.apache.org/jira/browse/SPARK-33447 Pr

[jira] [Created] (SPARK-33406) k8s pyspark shell doesn't honor pyspark memory setting

Thomas Graves created SPARK-33406: - Summary: k8s pyspark shell doesn't honor pyspark memory setting Key: SPARK-33406 URL: https://issues.apache.org/jira/browse/SPARK-33406 Project: Spark Issu

[jira] [Resolved] (SPARK-31711) Register the executor source with the metrics system when running in local mode.

[ https://issues.apache.org/jira/browse/SPARK-31711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-31711. --- Fix Version/s: 3.1.0 Resolution: Fixed > Register the executor source with the metric

[jira] [Assigned] (SPARK-31711) Register the executor source with the metrics system when running in local mode.

[ https://issues.apache.org/jira/browse/SPARK-31711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-31711: - Assignee: Luca Canali > Register the executor source with the metrics system when runni

[jira] [Resolved] (SPARK-32037) Rename blacklisting feature to avoid language with racist connotation

[ https://issues.apache.org/jira/browse/SPARK-32037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-32037. --- Fix Version/s: 3.1.0 Resolution: Fixed > Rename blacklisting feature to avoid languag

[jira] [Assigned] (SPARK-32037) Rename blacklisting feature to avoid language with racist connotation

[ https://issues.apache.org/jira/browse/SPARK-32037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-32037: - Assignee: Thomas Graves > Rename blacklisting feature to avoid language with racist con

[jira] [Updated] (SPARK-32591) Add better api docs for stage level scheduling Resources

[ https://issues.apache.org/jira/browse/SPARK-32591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-32591: -- Description: A question came up when we added offheap memory to be able to set in a ResourceP

[jira] [Commented] (SPARK-33288) Support k8s cluster manager with stage level scheduling

[ https://issues.apache.org/jira/browse/SPARK-33288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17222932#comment-17222932 ] Thomas Graves commented on SPARK-33288: --- yes. Really when I say hanging it means k

[jira] [Commented] (SPARK-33288) Support k8s cluster manager with stage level scheduling

[ https://issues.apache.org/jira/browse/SPARK-33288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17222917#comment-17222917 ] Thomas Graves commented on SPARK-33288: --- note there will be a couple of caveats he

[jira] [Created] (SPARK-33288) Support k8s cluster manager with stage level scheduling

Thomas Graves created SPARK-33288: - Summary: Support k8s cluster manager with stage level scheduling Key: SPARK-33288 URL: https://issues.apache.org/jira/browse/SPARK-33288 Project: Spark Iss

[jira] [Assigned] (SPARK-33088) Enhance ExecutorPlugin API to include methods for task start and end events

[ https://issues.apache.org/jira/browse/SPARK-33088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-33088: - Assignee: Samuel Souza > Enhance ExecutorPlugin API to include methods for task start a

[jira] [Commented] (SPARK-33031) scheduler with blacklisting doesn't appear to pick up new executor added

[ https://issues.apache.org/jira/browse/SPARK-33031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17204265#comment-17204265 ] Thomas Graves commented on SPARK-33031: --- I tried this again on yarn and now it see

[jira] [Updated] (SPARK-33031) scheduler with blacklisting doesn't appear to pick up new executor added

[ https://issues.apache.org/jira/browse/SPARK-33031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-33031: -- Description: I was running a test with blacklisting  standalone mode and all the executors we

[jira] [Updated] (SPARK-33031) scheduler with blacklisting doesn't appear to pick up new executor added

[ https://issues.apache.org/jira/browse/SPARK-33031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-33031: -- Description: I was running a test with blacklisting on yarn (and standalone mode) and all the

[jira] [Updated] (SPARK-33031) scheduler with blacklisting doesn't appear to pick up new executor added

[ https://issues.apache.org/jira/browse/SPARK-33031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-33031: -- Affects Version/s: 3.0.0 > scheduler with blacklisting doesn't appear to pick up new executor

[jira] [Updated] (SPARK-33031) scheduler with blacklisting doesn't appear to pick up new executor added

[ https://issues.apache.org/jira/browse/SPARK-33031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-33031: -- Affects Version/s: (was: 3.0.0) 3.1.0 > scheduler with blacklisting

[jira] [Commented] (SPARK-33029) Standalone mode blacklist executors page UI marks driver as blacklisted

[ https://issues.apache.org/jira/browse/SPARK-33029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17204259#comment-17204259 ] Thomas Graves commented on SPARK-33029: --- NOte I filed https://issues.apache.org/ji

[jira] [Created] (SPARK-33031) scheduler with blacklisting doesn't appear to pick up new executor added

Thomas Graves created SPARK-33031: - Summary: scheduler with blacklisting doesn't appear to pick up new executor added Key: SPARK-33031 URL: https://issues.apache.org/jira/browse/SPARK-33031 Project: S

[jira] [Updated] (SPARK-33029) Standalone mode blacklist executors page UI marks driver as blacklisted

[ https://issues.apache.org/jira/browse/SPARK-33029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-33029: -- Description: I am running a spark shell on a 1 node standalone cluster.  I noticed that the e

[jira] [Updated] (SPARK-33029) Standalone mode blacklist executors page UI marks driver as blacklisted

[ https://issues.apache.org/jira/browse/SPARK-33029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-33029: -- Attachment: Screen Shot 2020-09-29 at 1.53.37 PM.png > Standalone mode blacklist executors pag

[jira] [Updated] (SPARK-33029) Standalone mode blacklist executors page UI marks driver as blacklisted

[ https://issues.apache.org/jira/browse/SPARK-33029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-33029: -- Attachment: Screen Shot 2020-09-29 at 1.52.09 PM.png > Standalone mode blacklist executors pag

[jira] [Created] (SPARK-33029) Standalone mode blacklist executors page UI marks driver as blacklisted

Thomas Graves created SPARK-33029: - Summary: Standalone mode blacklist executors page UI marks driver as blacklisted Key: SPARK-33029 URL: https://issues.apache.org/jira/browse/SPARK-33029 Project: Sp

[jira] [Commented] (SPARK-32935) File source V2: support bucketing

[ https://issues.apache.org/jira/browse/SPARK-32935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17198368#comment-17198368 ] Thomas Graves commented on SPARK-32935: --- sorry looks like my update went in same t

[jira] [Updated] (SPARK-32935) File source V2: support bucketing

[ https://issues.apache.org/jira/browse/SPARK-32935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-32935: -- Description: Datasource V2 does not currently support bucketed reads or writes similar to Dat

[jira] [Updated] (SPARK-32935) File source V2: support bucketing

[ https://issues.apache.org/jira/browse/SPARK-32935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-32935: -- Description: Datasource V2 does not currently support bucketed reads similar to Datasource V1

[jira] [Updated] (SPARK-32935) File source V2: support bucketing

[ https://issues.apache.org/jira/browse/SPARK-32935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-32935: -- Description: Datasource V2 does not currently support bucketed reads similar to Datasource V1

[jira] [Commented] (SPARK-27589) Spark file source V2

[ https://issues.apache.org/jira/browse/SPARK-27589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17198339#comment-17198339 ] Thomas Graves commented on SPARK-27589: --- thanks for confirming and filing the jira

[jira] [Commented] (SPARK-27589) Spark file source V2

[ https://issues.apache.org/jira/browse/SPARK-27589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197669#comment-17197669 ] Thomas Graves commented on SPARK-27589: --- I'm guessing my question got missed - doe

[jira] [Commented] (SPARK-27589) Spark file source V2

[ https://issues.apache.org/jira/browse/SPARK-27589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197143#comment-17197143 ] Thomas Graves commented on SPARK-27589: --- somewhat related, I was looking through t

[jira] [Commented] (SPARK-32898) totalExecutorRunTimeMs is too big

[ https://issues.apache.org/jira/browse/SPARK-32898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17196972#comment-17196972 ] Thomas Graves commented on SPARK-32898: --- [~linhongliu-db] can you please provide m

[jira] [Updated] (SPARK-32894) Timestamp cast in exernal orc table

[ https://issues.apache.org/jira/browse/SPARK-32894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-32894: -- Summary: Timestamp cast in exernal orc table (was: Timestamp cast in exernal ocr table) > Ti

[jira] [Updated] (SPARK-32635) When pyspark.sql.functions.lit() function is used with dataframe cache, it returns wrong result

[ https://issues.apache.org/jira/browse/SPARK-32635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-32635: -- Labels: correct (was: ) > When pyspark.sql.functions.lit() function is used with dataframe ca

[jira] [Updated] (SPARK-32635) When pyspark.sql.functions.lit() function is used with dataframe cache, it returns wrong result

[ https://issues.apache.org/jira/browse/SPARK-32635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-32635: -- Labels: correctness (was: correct) > When pyspark.sql.functions.lit() function is used with d

[jira] [Updated] (SPARK-32635) When pyspark.sql.functions.lit() function is used with dataframe cache, it returns wrong result

[ https://issues.apache.org/jira/browse/SPARK-32635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-32635: -- Priority: Blocker (was: Major) > When pyspark.sql.functions.lit() function is used with dataf

[jira] [Commented] (SPARK-32037) Rename blacklisting feature to avoid language with racist connotation

[ https://issues.apache.org/jira/browse/SPARK-32037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17195531#comment-17195531 ] Thomas Graves commented on SPARK-32037: --- it is a good point about blocklist being

[jira] [Commented] (SPARK-32411) GPU Cluster Fail

[ https://issues.apache.org/jira/browse/SPARK-32411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17194485#comment-17194485 ] Thomas Graves commented on SPARK-32411: --- [~chitralverma] if you are still having a

[jira] [Created] (SPARK-32824) The error is confusing when resource .amount not provided

Thomas Graves created SPARK-32824: - Summary: The error is confusing when resource .amount not provided Key: SPARK-32824 URL: https://issues.apache.org/jira/browse/SPARK-32824 Project: Spark

[jira] [Created] (SPARK-32823) Standalone Master UI resources in use wrong

Thomas Graves created SPARK-32823: - Summary: Standalone Master UI resources in use wrong Key: SPARK-32823 URL: https://issues.apache.org/jira/browse/SPARK-32823 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-32823) Standalone Master UI resources in use wrong

[ https://issues.apache.org/jira/browse/SPARK-32823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17192256#comment-17192256 ] Thomas Graves commented on SPARK-32823: --- I'm looking into this. > Standalone Mast

[jira] [Commented] (SPARK-32037) Rename blacklisting feature to avoid language with racist connotation

[ https://issues.apache.org/jira/browse/SPARK-32037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17184059#comment-17184059 ] Thomas Graves commented on SPARK-32037: --- I started a thread on dev to get feedback

[jira] [Commented] (SPARK-32333) Drop references to Master

[ https://issues.apache.org/jira/browse/SPARK-32333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17184051#comment-17184051 ] Thomas Graves commented on SPARK-32333: --- I send email to the dev list to get feedb

[jira] [Commented] (SPARK-32672) Data corruption in some cached compressed boolean columns

[ https://issues.apache.org/jira/browse/SPARK-32672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17181867#comment-17181867 ] Thomas Graves commented on SPARK-32672: --- [~cltlfcjin]  Please do not be changing p

[jira] [Commented] (SPARK-32672) Data corruption in some cached compressed boolean columns

[ https://issues.apache.org/jira/browse/SPARK-32672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17181468#comment-17181468 ] Thomas Graves commented on SPARK-32672: --- [~cloud_fan] [~ruifengz] > Data corrupti

[jira] [Updated] (SPARK-32672) Data corruption in some cached compressed boolean columns

[ https://issues.apache.org/jira/browse/SPARK-32672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-32672: -- Affects Version/s: 3.0.1 > Data corruption in some cached compressed boolean columns > ---

[jira] [Updated] (SPARK-32672) Data corruption in some cached compressed boolean columns

[ https://issues.apache.org/jira/browse/SPARK-32672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-32672: -- Labels: correctness (was: ) > Data corruption in some cached compressed boolean columns > ---

[jira] [Comment Edited] (SPARK-32640) Spark 3.1 log(NaN) returns null instead of NaN

[ https://issues.apache.org/jira/browse/SPARK-32640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180772#comment-17180772 ] Thomas Graves edited comment on SPARK-32640 at 8/19/20, 7:14 PM: -

[jira] [Comment Edited] (SPARK-32640) Spark 3.1 log(NaN) returns null instead of NaN

[ https://issues.apache.org/jira/browse/SPARK-32640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180772#comment-17180772 ] Thomas Graves edited comment on SPARK-32640 at 8/19/20, 7:14 PM: -

[jira] [Commented] (SPARK-32640) Spark 3.1 log(NaN) returns null instead of NaN

[ https://issues.apache.org/jira/browse/SPARK-32640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180772#comment-17180772 ] Thomas Graves commented on SPARK-32640: --- hmm, interesting, this is how my test was

[jira] [Commented] (SPARK-32640) Spark 3.1 log(NaN) returns null instead of NaN

[ https://issues.apache.org/jira/browse/SPARK-32640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180533#comment-17180533 ] Thomas Graves commented on SPARK-32640: --- The problem is with 3.1.0 not, 3.0.0. The

[jira] [Updated] (SPARK-32640) Spark 3.1 log(NaN) returns null instead of NaN

[ https://issues.apache.org/jira/browse/SPARK-32640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-32640: -- Description: I was testing Spark 3.1.0 and I noticed that if you take the log(NaN) it now ret

[jira] [Created] (SPARK-32640) Spark 3.1 log(NaN) returns null instead of NaN

Thomas Graves created SPARK-32640: - Summary: Spark 3.1 log(NaN) returns null instead of NaN Key: SPARK-32640 URL: https://issues.apache.org/jira/browse/SPARK-32640 Project: Spark Issue Type:

[jira] [Created] (SPARK-32591) Add better api docs for stage level scheduling Resources

Thomas Graves created SPARK-32591: - Summary: Add better api docs for stage level scheduling Resources Key: SPARK-32591 URL: https://issues.apache.org/jira/browse/SPARK-32591 Project: Spark Is

[jira] [Commented] (SPARK-32037) Rename blacklisting feature to avoid language with racist connotation

[ https://issues.apache.org/jira/browse/SPARK-32037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17170817#comment-17170817 ] Thomas Graves commented on SPARK-32037: --- allowlist and blocklist have been used by

[jira] [Commented] (SPARK-32333) Drop references to Master

[ https://issues.apache.org/jira/browse/SPARK-32333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17170451#comment-17170451 ] Thomas Graves commented on SPARK-32333: --- What about renaming Master to be Applicat

[jira] [Updated] (SPARK-32332) AQE doesn't adequately allow for Columnar Processing extension

[ https://issues.apache.org/jira/browse/SPARK-32332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-32332: -- Fix Version/s: 3.0.1 > AQE doesn't adequately allow for Columnar Processing extension > -

[jira] [Commented] (SPARK-27589) Spark file source V2

[ https://issues.apache.org/jira/browse/SPARK-27589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17168133#comment-17168133 ] Thomas Graves commented on SPARK-27589: --- Hey, I see most of the sources are still

[jira] [Resolved] (SPARK-32332) AQE doesn't adequately allow for Columnar Processing extension

[ https://issues.apache.org/jira/browse/SPARK-32332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-32332. --- Fix Version/s: 3.1.0 Assignee: Wenchen Fan Resolution: Fixed > AQE doesn't a

[jira] [Resolved] (SPARK-30322) Add stage level scheduling docs

[ https://issues.apache.org/jira/browse/SPARK-30322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-30322. --- Fix Version/s: 3.1.0 Assignee: Thomas Graves Resolution: Fixed > Add stage l

[jira] [Updated] (SPARK-30322) Add stage level scheduling docs

[ https://issues.apache.org/jira/browse/SPARK-30322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-30322: -- Description: Add stage level scheduling docs. > Add stage level scheduling docs >

[jira] [Commented] (SPARK-32470) Remove task result size check for shuffle map stage

[ https://issues.apache.org/jira/browse/SPARK-32470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17167233#comment-17167233 ] Thomas Graves commented on SPARK-32470: --- Please add a description to the Jira as t

<    1   2   3   4   5   6   7   8   9   10   >