[jira] [Commented] (SPARK-22148) TaskSetManager.abortIfCompletelyBlacklisted should not abort when all current executors are blacklisted but dynamic allocation is enabled

2020-04-13 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-22148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17082736#comment-17082736 ] Erik Krogen commented on SPARK-22148: - For future folks: the JIRA created for the issue is

[jira] [Commented] (SPARK-31418) Blacklisting feature aborts Spark job without retrying for max num retries in case of Dynamic allocation

2020-04-21 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17089179#comment-17089179 ] Erik Krogen commented on SPARK-31418: - PR was posted by [~vsowrirajan] here:

[jira] [Commented] (SPARK-32037) Rename blacklisting feature to avoid language with racist connotation

2020-09-14 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17195542#comment-17195542 ] Erik Krogen commented on SPARK-32037: - Thanks for continuing to push this forward [~tgraves]! Let me

[jira] [Commented] (SPARK-33138) unify temp view and permanent view behaviors

2020-10-14 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17214011#comment-17214011 ] Erik Krogen commented on SPARK-33138: - Hi [~leanken], thanks for starting this effort. It looks very

[jira] [Created] (SPARK-33185) YARN: Print direct links to driver logs alongside application report in cluster mode

2020-10-19 Thread Erik Krogen (Jira)
Erik Krogen created SPARK-33185: --- Summary: YARN: Print direct links to driver logs alongside application report in cluster mode Key: SPARK-33185 URL: https://issues.apache.org/jira/browse/SPARK-33185

[jira] [Updated] (SPARK-33185) YARN: Print direct links to driver logs alongside application report in cluster mode

2020-10-19 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated SPARK-33185: Description: Currently when run in {{cluster}} mode on YARN, the Spark {{yarn.Client}} will

[jira] [Commented] (SPARK-32944) Avoid push down Filter through Project when it will hurts performance

2020-09-21 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17199456#comment-17199456 ] Erik Krogen commented on SPARK-32944: - [~viirya] I'm curious if you can provide some examples of

[jira] [Created] (SPARK-33214) HiveExternalCatalogVersionsSuite shouldn't use or delete hard-coded /tmp directory

2020-10-21 Thread Erik Krogen (Jira)
Erik Krogen created SPARK-33214: --- Summary: HiveExternalCatalogVersionsSuite shouldn't use or delete hard-coded /tmp directory Key: SPARK-33214 URL: https://issues.apache.org/jira/browse/SPARK-33214

[jira] [Commented] (SPARK-33138) unify temp view and permanent view behaviors

2020-10-27 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17221506#comment-17221506 ] Erik Krogen commented on SPARK-33138: - I see, thanks for clarifying [~cloud_fan]. > unify temp

[jira] [Commented] (SPARK-32037) Rename blacklisting feature to avoid language with racist connotation

2020-06-22 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17142426#comment-17142426 ] Erik Krogen commented on SPARK-32037: - Thanks for the suggestions [~H4ml3t]! * *quarantined* to me

[jira] [Created] (SPARK-32036) Remove references to "blacklist"/"whitelist" language (outside of blacklisting feature)

2020-06-19 Thread Erik Krogen (Jira)
Erik Krogen created SPARK-32036: --- Summary: Remove references to "blacklist"/"whitelist" language (outside of blacklisting feature) Key: SPARK-32036 URL: https://issues.apache.org/jira/browse/SPARK-32036

[jira] [Commented] (SPARK-32037) Rename blacklisting feature to avoid language with racist connotation

2020-06-19 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17140876#comment-17140876 ] Erik Krogen commented on SPARK-32037: - +1 from me, I agree that this feature is basically a health

[jira] [Updated] (SPARK-32037) Rename blacklisting feature to avoid language with racist connotation

2020-06-19 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated SPARK-32037: Description: As per [discussion on the Spark dev

[jira] [Updated] (SPARK-32036) Remove references to "blacklist"/"whitelist" language (outside of blacklisting feature)

2020-06-19 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated SPARK-32036: Description: As per [discussion on the Spark dev

[jira] [Created] (SPARK-32037) Rename blacklisting feature to avoid language with racist connotation

2020-06-19 Thread Erik Krogen (Jira)
Erik Krogen created SPARK-32037: --- Summary: Rename blacklisting feature to avoid language with racist connotation Key: SPARK-32037 URL: https://issues.apache.org/jira/browse/SPARK-32037 Project: Spark

[jira] [Updated] (SPARK-32334) Investigate commonizing Columnar and Row data transformations

2020-07-17 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated SPARK-32334: Description: We introduced more Columnar Support with SPARK-27396. With that we recognized that

[jira] [Commented] (SPARK-32037) Rename blacklisting feature to avoid language with racist connotation

2020-07-16 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17159318#comment-17159318 ] Erik Krogen commented on SPARK-32037: - I think it might be helpful to frame the discussion around

[jira] [Updated] (SPARK-33726) Duplicate field names causes wrong answers during aggregation

2020-12-09 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated SPARK-33726: Labels: correctness (was: ) > Duplicate field names causes wrong answers during aggregation >

[jira] [Commented] (SPARK-33772) Build and Run Spark on JDK17

2020-12-14 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17249088#comment-17249088 ] Erik Krogen commented on SPARK-33772: - It is very weird to see JDK versions bumping up by 6 whole

[jira] [Commented] (SPARK-23862) Spark ExpressionEncoder should support java enum type in scala

2020-12-18 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17252049#comment-17252049 ] Erik Krogen commented on SPARK-23862: - I'm going to take up this work. > Spark ExpressionEncoder

[jira] [Comment Edited] (SPARK-23862) Spark ExpressionEncoder should support java enum type in scala

2020-12-18 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17252049#comment-17252049 ] Erik Krogen edited comment on SPARK-23862 at 12/18/20, 11:45 PM: - I'm

[jira] [Reopened] (SPARK-33185) YARN: Print direct links to driver logs alongside application report in cluster mode

2020-11-20 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen reopened SPARK-33185: - > YARN: Print direct links to driver logs alongside application report in > cluster mode >

[jira] [Commented] (SPARK-33185) YARN: Print direct links to driver logs alongside application report in cluster mode

2020-11-20 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17236466#comment-17236466 ] Erik Krogen commented on SPARK-33185: - I found that the existing logic doesn't work properly in

[jira] [Created] (SPARK-34133) [AVRO] Respect case sensitivity when performing Catalyst-to-Avro field matching and enhance error messages

2021-01-15 Thread Erik Krogen (Jira)
Erik Krogen created SPARK-34133: --- Summary: [AVRO] Respect case sensitivity when performing Catalyst-to-Avro field matching and enhance error messages Key: SPARK-34133 URL:

[jira] [Updated] (SPARK-34133) [AVRO] Respect case sensitivity when performing Catalyst-to-Avro field matching

2021-01-20 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated SPARK-34133: Summary: [AVRO] Respect case sensitivity when performing Catalyst-to-Avro field matching (was:

[jira] [Created] (SPARK-34182) [AVRO] Improve error messages when matching Catalyst-to-Avro schemas

2021-01-20 Thread Erik Krogen (Jira)
Erik Krogen created SPARK-34182: --- Summary: [AVRO] Improve error messages when matching Catalyst-to-Avro schemas Key: SPARK-34182 URL: https://issues.apache.org/jira/browse/SPARK-34182 Project: Spark

[jira] [Updated] (SPARK-34133) [AVRO] Respect case sensitivity when performing Catalyst-to-Avro field matching

2021-01-20 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated SPARK-34133: Description: Spark SQL is normally case-insensitive (by default), but currently when

[jira] [Updated] (SPARK-34182) [AVRO] Improve error messages when matching Catalyst-to-Avro schemas

2021-01-20 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated SPARK-34182: Issue Type: Improvement (was: Bug) > [AVRO] Improve error messages when matching

[jira] [Created] (SPARK-34231) [AVRO][TEST] AvroSuite has test failure when run from IDE due to bad loading of resource file

2021-01-25 Thread Erik Krogen (Jira)
Erik Krogen created SPARK-34231: --- Summary: [AVRO][TEST] AvroSuite has test failure when run from IDE due to bad loading of resource file Key: SPARK-34231 URL: https://issues.apache.org/jira/browse/SPARK-34231

[jira] [Commented] (SPARK-7768) Make user-defined type (UDT) API public

2021-01-22 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-7768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17270235#comment-17270235 ] Erik Krogen commented on SPARK-7768: [~metasim] you might want to start a discussion on

[jira] [Commented] (SPARK-34344) Have functionality to trace back Spark SQL queries from the application ID that got submitted on YARN

2021-02-03 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17278233#comment-17278233 ] Erik Krogen commented on SPARK-34344: - This sounds the like "SQL" tab on the SHS UI, added in 2.0.0

[jira] [Commented] (SPARK-34344) Have functionality to trace back Spark SQL queries from the application ID that got submitted on YARN

2021-02-03 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17278242#comment-17278242 ] Erik Krogen commented on SPARK-34344: - I see, thanks for the clarification. You want the original

[jira] [Commented] (SPARK-34344) Have functionality to trace back Spark SQL queries from the application ID that got submitted on YARN

2021-02-03 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17278173#comment-17278173 ] Erik Krogen commented on SPARK-34344: - [~arpan3189] can you elaborate what you mean here? > Have

[jira] [Comment Edited] (SPARK-34344) Have functionality to trace back Spark SQL queries from the application ID that got submitted on YARN

2021-02-03 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17278173#comment-17278173 ] Erik Krogen edited comment on SPARK-34344 at 2/3/21, 4:48 PM: -- [~arpan3189]

[jira] [Commented] (SPARK-27589) Spark file source V2

2021-01-27 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17273154#comment-17273154 ] Erik Krogen commented on SPARK-27589: - [~Gengliang.Wang] are you or anyone else planning to work on

[jira] [Commented] (SPARK-35744) Performance degradation in avro SpecificRecordBuilders

2021-06-14 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17363029#comment-17363029 ] Erik Krogen commented on SPARK-35744: - [~steven.aerts] can you elaborate on where you're using

[jira] [Commented] (SPARK-35667) spark.speculation causes incorrect query results with TRANSFORM

2021-06-07 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17358707#comment-17358707 ] Erik Krogen commented on SPARK-35667: - fyi [~vsowrirajan] [~ron8hu] > spark.speculation causes

[jira] [Created] (SPARK-35672) Spark fails to launch executors with very large user classpath lists on YARN

2021-06-07 Thread Erik Krogen (Jira)
Erik Krogen created SPARK-35672: --- Summary: Spark fails to launch executors with very large user classpath lists on YARN Key: SPARK-35672 URL: https://issues.apache.org/jira/browse/SPARK-35672 Project:

[jira] [Commented] (SPARK-35715) Option "--files" with local:// prefix is not honoured for Spark on kubernetes

2021-06-10 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17361086#comment-17361086 ] Erik Krogen commented on SPARK-35715: - Not sure about k8s, but at least for YARN this is expected --

[jira] [Commented] (SPARK-35744) Performance degradation in avro SpecificRecordBuilders

2021-06-21 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17366892#comment-17366892 ] Erik Krogen commented on SPARK-35744: - [~steven.aerts] going a bit off topic from this JIRA, but out

[jira] [Commented] (SPARK-35672) Spark fails to launch executors with very large user classpath lists on YARN

2021-06-25 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17369564#comment-17369564 ] Erik Krogen commented on SPARK-35672: - #32810 went into master. Put up #33090 for branch-3.1 >

[jira] [Commented] (SPARK-35817) Queries against wide Avro tables can be slow

2021-06-18 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17365762#comment-17365762 ] Erik Krogen commented on SPARK-35817: - Thanks for catching this [~bersprockets]! I will be happy to

[jira] [Comment Edited] (SPARK-35817) Queries against wide Avro tables can be slow

2021-06-18 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17365762#comment-17365762 ] Erik Krogen edited comment on SPARK-35817 at 6/18/21, 10:09 PM: Thanks

[jira] [Updated] (SPARK-35668) Use "concurrency" setting on Github Actions

2021-06-08 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated SPARK-35668: Description: We are using

[jira] [Commented] (SPARK-35321) Spark 3.x can't talk to HMS 1.2.x and lower due to get_all_functions Thrift API missing

2021-05-05 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17339911#comment-17339911 ] Erik Krogen commented on SPARK-35321: - Gotcha. Yeah, agreed that if it's unnecessary we may as well

[jira] [Commented] (SPARK-35321) Spark 3.x can't talk to HMS 1.2.x and lower due to get_all_functions Thrift API missing

2021-05-05 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17339904#comment-17339904 ] Erik Krogen commented on SPARK-35321: - Isn't this what the {{IsolatedClientLoader}} is for? You

[jira] [Updated] (SPARK-35259) ExternalBlockHandler metrics have misleading unit in the name

2021-06-28 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated SPARK-35259: Description: Today {{ExternalBlockHandler}} exposes a few {{Timer}} metrics: {code} // Time

[jira] [Created] (SPARK-35918) Consolidate logic between AvroSerializer/AvroDeserializer for schema mismatch handling and error messages

2021-06-28 Thread Erik Krogen (Jira)
Erik Krogen created SPARK-35918: --- Summary: Consolidate logic between AvroSerializer/AvroDeserializer for schema mismatch handling and error messages Key: SPARK-35918 URL:

[jira] [Commented] (SPARK-32333) Drop references to Master

2021-07-12 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17379239#comment-17379239 ] Erik Krogen commented on SPARK-32333: - +1 on leader from my side > Drop references to Master >

[jira] [Comment Edited] (SPARK-35957) Cannot convert Avro schema to catalyst type because schema at path is not compatible

2021-07-12 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17379278#comment-17379278 ] Erik Krogen edited comment on SPARK-35957 at 7/12/21, 5:10 PM: --- Based on

[jira] [Comment Edited] (SPARK-35957) Cannot convert Avro schema to catalyst type because schema at path is not compatible

2021-07-12 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17379278#comment-17379278 ] Erik Krogen edited comment on SPARK-35957 at 7/12/21, 5:10 PM: --- Based on

[jira] [Commented] (SPARK-35957) Cannot convert Avro schema to catalyst type because schema at path is not compatible

2021-07-12 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17379278#comment-17379278 ] Erik Krogen commented on SPARK-35957: - Based on the discussion in [the linked Hudi

[jira] [Updated] (SPARK-35259) ExternalBlockHandler metrics have misleading unit in the name

2021-04-28 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated SPARK-35259: Description: Today {{ExternalBlockHandler}} exposes a few {{Timer}} metrics: {code} // Time

[jira] [Updated] (SPARK-35259) ExternalBlockHandler metrics have misleading unit in the name

2021-04-28 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated SPARK-35259: Description: Today {{ExternalBlockHandler}} exposes a few {{Timer}} metrics: {code} // Time

[jira] [Created] (SPARK-35263) Refactor ShuffleBlockFetcherIteratorSuite to reduce duplicated code

2021-04-28 Thread Erik Krogen (Jira)
Erik Krogen created SPARK-35263: --- Summary: Refactor ShuffleBlockFetcherIteratorSuite to reduce duplicated code Key: SPARK-35263 URL: https://issues.apache.org/jira/browse/SPARK-35263 Project: Spark

[jira] [Commented] (SPARK-35259) ExternalBlockHandler metrics have misleading unit in the name

2021-04-28 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17334932#comment-17334932 ] Erik Krogen commented on SPARK-35259: - I have a PR for this but it is based on the PR for

[jira] [Created] (SPARK-35258) Enhance ESS ExternalBlockHandler with additional block rate-based metrics and histograms

2021-04-28 Thread Erik Krogen (Jira)
Erik Krogen created SPARK-35258: --- Summary: Enhance ESS ExternalBlockHandler with additional block rate-based metrics and histograms Key: SPARK-35258 URL: https://issues.apache.org/jira/browse/SPARK-35258

[jira] [Updated] (SPARK-35259) ExternalBlockHandler metrics have misleading unit in the name

2021-04-28 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated SPARK-35259: Summary: ExternalBlockHandler metrics have misleading unit in the name (was:

[jira] [Created] (SPARK-35259) ExternalBlockHandler metrics have incorrect unit in the name

2021-04-28 Thread Erik Krogen (Jira)
Erik Krogen created SPARK-35259: --- Summary: ExternalBlockHandler metrics have incorrect unit in the name Key: SPARK-35259 URL: https://issues.apache.org/jira/browse/SPARK-35259 Project: Spark

[jira] [Created] (SPARK-34378) Support extra optional Avro fields in AvroSerializer

2021-02-05 Thread Erik Krogen (Jira)
Erik Krogen created SPARK-34378: --- Summary: Support extra optional Avro fields in AvroSerializer Key: SPARK-34378 URL: https://issues.apache.org/jira/browse/SPARK-34378 Project: Spark Issue

[jira] [Commented] (SPARK-34378) Support extra optional Avro fields in AvroSerializer

2021-02-05 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17279839#comment-17279839 ] Erik Krogen commented on SPARK-34378: - Internally we build this feature on top of SPARK-34365, so I

[jira] [Commented] (SPARK-34336) Use GenericData as Avro serialization data model can improve Avro write/read performance

2021-02-02 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17277488#comment-17277488 ] Erik Krogen commented on SPARK-34336: - Thanks for bringing this up [~Baohe Zhang], I came across PR

[jira] [Created] (SPARK-34828) YARN Shuffle Service: Support configurability of aux service name and service-specific config overrides

2021-03-22 Thread Erik Krogen (Jira)
Erik Krogen created SPARK-34828: --- Summary: YARN Shuffle Service: Support configurability of aux service name and service-specific config overrides Key: SPARK-34828 URL:

[jira] [Updated] (SPARK-34752) Upgrade Jetty to 9.3.37 to fix CVE-2020-27223

2021-03-15 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated SPARK-34752: Description: Another day, another Jetty CVE :)  Our internal build tools are complaining about

[jira] [Updated] (SPARK-34752) Upgrade Jetty to 9.3.37 to fix CVE-2020-27223

2021-03-15 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated SPARK-34752: Description: Another day, another Jetty CVE :)  Our internal build tools are complaining about

[jira] [Created] (SPARK-34752) Upgrade Jetty to 9.3.37 to fix CVE-2020-27223

2021-03-15 Thread Erik Krogen (Jira)
Erik Krogen created SPARK-34752: --- Summary: Upgrade Jetty to 9.3.37 to fix CVE-2020-27223 Key: SPARK-34752 URL: https://issues.apache.org/jira/browse/SPARK-34752 Project: Spark Issue Type:

[jira] [Updated] (SPARK-34752) Upgrade Jetty to 9.4.37 to fix CVE-2020-27223

2021-03-15 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated SPARK-34752: Summary: Upgrade Jetty to 9.4.37 to fix CVE-2020-27223 (was: Upgrade Jetty to 9.3.37 to fix

[jira] [Updated] (SPARK-34752) Upgrade Jetty to 9.4.37 to fix CVE-2020-27223

2021-03-15 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated SPARK-34752: Description: Another day, another Jetty CVE :)  Our internal build tools are complaining about

[jira] [Commented] (SPARK-34624) Filter non-jar dependencies from ivy/maven coordinates

2021-03-04 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17295650#comment-17295650 ] Erik Krogen commented on SPARK-34624: - Thanks for reporting this [~shardulm]! One question for you:

[jira] [Comment Edited] (SPARK-34624) Filter non-jar dependencies from ivy/maven coordinates

2021-03-04 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17295650#comment-17295650 ] Erik Krogen edited comment on SPARK-34624 at 3/5/21, 12:01 AM: --- Thanks for

[jira] [Created] (SPARK-35106) HadoopMapReduceCommitProtocol performs bad rename when dynamic partition overwrite is used

2021-04-16 Thread Erik Krogen (Jira)
Erik Krogen created SPARK-35106: --- Summary: HadoopMapReduceCommitProtocol performs bad rename when dynamic partition overwrite is used Key: SPARK-35106 URL: https://issues.apache.org/jira/browse/SPARK-35106

[jira] [Commented] (SPARK-34455) Deprecate spark.sql.legacy.replaceDatabricksSparkAvro.enabled

2021-02-17 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286056#comment-17286056 ] Erik Krogen commented on SPARK-34455: - +1 it is time to get around to this! > Deprecate

[jira] [Updated] (SPARK-34365) Support configurable Avro schema field matching for positional or by-name

2021-02-04 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated SPARK-34365: Description: When reading an Avro dataset (using the dataset's schema or by overriding it with

[jira] [Commented] (SPARK-34365) Support configurable Avro schema field matching for positional or by-name

2021-02-04 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17279073#comment-17279073 ] Erik Krogen commented on SPARK-34365: - I plan to post a PR for this in the next few days, unless I

[jira] [Created] (SPARK-34365) Support configurable Avro schema field matching for positional or by-name

2021-02-04 Thread Erik Krogen (Jira)
Erik Krogen created SPARK-34365: --- Summary: Support configurable Avro schema field matching for positional or by-name Key: SPARK-34365 URL: https://issues.apache.org/jira/browse/SPARK-34365 Project:

[jira] [Commented] (SPARK-32333) Drop references to Master

2021-08-27 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17405904#comment-17405904 ] Erik Krogen commented on SPARK-32333: - Personally, tackling publicly-facing things is my highest

[jira] [Commented] (SPARK-36673) Incorrect Unions of struct with mismatched field name case

2021-09-09 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17412841#comment-17412841 ] Erik Krogen commented on SPARK-36673: - >From the Scaladoc for {{union}}: {code} * Also as

[jira] [Commented] (SPARK-35957) Cannot convert Avro schema to catalyst type because schema at path is not compatible

2021-07-13 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17379978#comment-17379978 ] Erik Krogen commented on SPARK-35957: - [~jkdll] would it be possible for you to try against the

[jira] [Commented] (SPARK-36134) jackson-databind RCE vulnerability [Need to upgrade to 2.9.3.1]

2021-07-14 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17380695#comment-17380695 ] Erik Krogen commented on SPARK-36134: - Jackson is already 2.12.3 (from

[jira] [Reopened] (SPARK-28266) data duplication when `path` serde property is present

2021-07-13 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen reopened SPARK-28266: - Re-opening this issue based on [~shardulm]'s example above demonstrating that this is indeed a

[jira] [Commented] (SPARK-36416) Add SQL metrics to AdaptiveSparkPlanExec for BHJs and Skew joins

2021-08-04 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17393360#comment-17393360 ] Erik Krogen commented on SPARK-36416: - +1 this would be very helpful! > Add SQL metrics to

[jira] [Commented] (SPARK-33828) SQL Adaptive Query Execution QA

2021-10-18 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17430124#comment-17430124 ] Erik Krogen commented on SPARK-33828: - [~dongjoon] as you mentioned above, this epic was initially

[jira] [Comment Edited] (SPARK-37027) Fix behavior inconsistent in Hive table when ‘path’ is provided in SERDEPROPERTIES

2021-10-18 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17430125#comment-17430125 ] Erik Krogen edited comment on SPARK-37027 at 10/18/21, 6:10 PM:

[jira] [Commented] (SPARK-37027) Fix behavior inconsistent in Hive table when ‘path’ is provided in SERDEPROPERTIES

2021-10-18 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17430125#comment-17430125 ] Erik Krogen commented on SPARK-37027: - [~yuzhousun] actually this is already fixed by SPARK-28266 in

[jira] [Commented] (SPARK-33828) SQL Adaptive Query Execution QA

2021-10-19 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17430798#comment-17430798 ] Erik Krogen commented on SPARK-33828: - Thanks [~dongjoon]! > SQL Adaptive Query Execution QA >

[jira] [Commented] (SPARK-37043) Cancel all running job after AQE plan finished

2021-10-19 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17430797#comment-17430797 ] Erik Krogen commented on SPARK-37043: - [~ulysses] any concerns if I make this a sub-task of

[jira] [Commented] (SPARK-36905) Reading Hive view without explicit column names fails in Spark

2021-09-30 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17422962#comment-17422962 ] Erik Krogen commented on SPARK-36905: - [~shardulm] is important here that the view is from Hive? Can

[jira] [Updated] (SPARK-36810) Handle HDFS read inconsistencies on Spark when observer Namenode is used

2021-09-20 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated SPARK-36810: Summary: Handle HDFS read inconsistencies on Spark when observer Namenode is used (was: Handle

[jira] [Updated] (SPARK-36810) Handle HDSF read inconsistencies on Spark when observer Namenode is used

2021-09-20 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated SPARK-36810: Description: In short, with HDFS HA and with the use of [Observer

[jira] [Commented] (SPARK-36905) Reading Hive view without explicit column names fails in Spark

2021-10-04 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17424183#comment-17424183 ] Erik Krogen commented on SPARK-36905: - cc also [~maropu] [~viirya] [~csun] > Reading Hive view

[jira] [Commented] (SPARK-35672) Spark fails to launch executors with very large user classpath lists on YARN

2021-09-27 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17421016#comment-17421016 ] Erik Krogen commented on SPARK-35672: - Re-submitted at [PR

[jira] [Commented] (SPARK-35672) Spark fails to launch executors with very large user classpath lists on YARN

2021-09-24 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17419830#comment-17419830 ] Erik Krogen commented on SPARK-35672: - Thanks [~petertoth] [~hyukjin.kwon] [~Gengliang.Wang] for

[jira] [Commented] (SPARK-37166) SPIP: Storage Partitioned Join

2021-11-01 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17436959#comment-17436959 ] Erik Krogen commented on SPARK-37166: - [~csun] can you link the doc here? > SPIP: Storage

[jira] [Created] (SPARK-37121) TestUtils.isPythonVersionAtLeast38 returns incorrect results

2021-10-26 Thread Erik Krogen (Jira)
Erik Krogen created SPARK-37121: --- Summary: TestUtils.isPythonVersionAtLeast38 returns incorrect results Key: SPARK-37121 URL: https://issues.apache.org/jira/browse/SPARK-37121 Project: Spark

[jira] [Updated] (SPARK-37043) Cancel all running job after AQE plan finished

2021-10-25 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated SPARK-37043: Parent: SPARK-37063 Issue Type: Sub-task (was: Improvement) > Cancel all running job

[jira] [Commented] (SPARK-37043) Cancel all running job after AQE plan finished

2021-10-25 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17433834#comment-17433834 ] Erik Krogen commented on SPARK-37043: - Converted to subtask of SPARK-37063. > Cancel all running

[jira] [Updated] (SPARK-37621) ClassCastException when trying to persist the result of a join between two Iceberg tables

2021-12-13 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated SPARK-37621: Description: I am gettin an error when I try to persist the results on a Join operation. Note

[jira] [Commented] (SPARK-36134) jackson-databind RCE vulnerability

2021-07-16 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17382128#comment-17382128 ] Erik Krogen commented on SPARK-36134: - Whoops, must have missed the 3.1.2 release :) Thanks for

[jira] [Commented] (SPARK-36134) jackson-databind RCE vulnerability

2021-07-15 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17381417#comment-17381417 ] Erik Krogen commented on SPARK-36134: - 3.1.2 doesn't exist yet, the only release in the 3.1 line is

[jira] [Commented] (SPARK-38245) Avro Complex Union Type return `member$I`

2022-02-22 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17496337#comment-17496337 ] Erik Krogen commented on SPARK-38245: - FWIW, though I don't have context for when this logic was

  1   2   >