[jira] [Commented] (SPARK-41551) Improve/complete PathOutputCommitProtocol support for dynamic partitioning

2022-12-20 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17649775#comment-17649775 ] Steve Loughran commented on SPARK-41551: So there's an interesting little "feature" of

[jira] [Commented] (SPARK-41392) spark builds against hadoop trunk/3.4.0-SNAPSHOT fail in scala-maven plugin

2022-12-06 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17643774#comment-17643774 ] Steve Loughran commented on SPARK-41392: may relate to the bouncy castle 1.68 update of

[jira] [Created] (SPARK-41392) spark builds against hadoop trunk/3.4.0-SNAPSHOT fail in scala-maven plugin

2022-12-05 Thread Steve Loughran (Jira)
Steve Loughran created SPARK-41392: -- Summary: spark builds against hadoop trunk/3.4.0-SNAPSHOT fail in scala-maven plugin Key: SPARK-41392 URL: https://issues.apache.org/jira/browse/SPARK-41392

[jira] [Created] (SPARK-41551) Improve/complete PathOutputCommitProtocol support for dynamic partitioning

2022-12-16 Thread Steve Loughran (Jira)
Steve Loughran created SPARK-41551: -- Summary: Improve/complete PathOutputCommitProtocol support for dynamic partitioning Key: SPARK-41551 URL: https://issues.apache.org/jira/browse/SPARK-41551

[jira] [Commented] (SPARK-41392) spark builds against hadoop trunk/3.4.0-SNAPSHOT fail in scala-maven plugin

2022-12-05 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17643492#comment-17643492 ] Steve Loughran commented on SPARK-41392: MBP m1 with {code} uname -a Darwin stevel-MBP16

[jira] [Commented] (SPARK-41599) Memory leak in FileSystem.CACHE when submitting apps to secure cluster using InProcessLauncher

2023-01-18 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17678189#comment-17678189 ] Steve Loughran commented on SPARK-41599: well, the challenge there becomes "not changing that

[jira] [Commented] (SPARK-40034) PathOutputCommitters to work with dynamic partition overwrite

2023-01-19 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17678814#comment-17678814 ] Steve Loughran commented on SPARK-40034: Note that these changes aren't sufficient. The hadoop

[jira] [Created] (SPARK-42537) Remove obsolete/superfluous imports in spark-hadoop-cloud module

2023-02-23 Thread Steve Loughran (Jira)
Steve Loughran created SPARK-42537: -- Summary: Remove obsolete/superfluous imports in spark-hadoop-cloud module Key: SPARK-42537 URL: https://issues.apache.org/jira/browse/SPARK-42537 Project: Spark

[jira] [Commented] (SPARK-42537) Remove obsolete/superfluous imports in spark-hadoop-cloud module

2023-02-23 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692617#comment-17692617 ] Steve Loughran commented on SPARK-42537: FYI +[~dannycjones]. I'm getting build issues related

[jira] [Commented] (SPARK-40034) PathOutputCommitters to work with dynamic partition overwrite

2023-03-01 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17694969#comment-17694969 ] Steve Loughran commented on SPARK-40034: thanks for the update. I will get that new pr done

[jira] [Commented] (SPARK-43170) The spark sql like statement is pushed down to parquet for execution, but the data cannot be queried

2023-04-28 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717611#comment-17717611 ] Steve Loughran commented on SPARK-43170: FWIW, using S3 URLs

[jira] [Commented] (SPARK-44124) Upgrade AWS SDK to v2

2023-07-31 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17749264#comment-17749264 ] Steve Loughran commented on SPARK-44124: we are soon to move hadoop trunk up to SDK v2,

[jira] [Commented] (SPARK-44116) Utilize Hadoop vectorized APIs

2023-07-31 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17749266#comment-17749266 ] Steve Loughran commented on SPARK-44116: If this gets into the libraries, you don't need

[jira] [Commented] (SPARK-44042) SPIP: PySpark Test Framework

2023-06-21 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17735646#comment-17735646 ] Steve Loughran commented on SPARK-44042: * you can create an independent git repo for this (ASF

[jira] [Commented] (SPARK-41599) Memory leak in FileSystem.CACHE when submitting apps to secure cluster using InProcessLauncher

2023-06-16 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17733454#comment-17733454 ] Steve Loughran commented on SPARK-41599: correct. remember, all the source of hadoop is there

[jira] [Created] (SPARK-47008) Spark to support S3 Express One Zone Storage

2024-02-08 Thread Steve Loughran (Jira)
Steve Loughran created SPARK-47008: -- Summary: Spark to support S3 Express One Zone Storage Key: SPARK-47008 URL: https://issues.apache.org/jira/browse/SPARK-47008 Project: Spark Issue Type:

[jira] [Commented] (SPARK-46247) Invalid bucket file error when reading from bucketed table created with PathOutputCommitProtocol

2024-01-18 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17808227#comment-17808227 ] Steve Loughran commented on SPARK-46247: why is the file invalid? any more stack trace? # try

[jira] [Updated] (SPARK-46793) Revert S3A endpoint fixup logic of SPARK-35878

2024-01-22 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-46793: --- Summary: Revert S3A endpoint fixup logic of SPARK-35878 (was: Revert region fixup logic of

[jira] [Created] (SPARK-46793) Revert region fixup logic of SPARK-35878

2024-01-22 Thread Steve Loughran (Jira)
Steve Loughran created SPARK-46793: -- Summary: Revert region fixup logic of SPARK-35878 Key: SPARK-46793 URL: https://issues.apache.org/jira/browse/SPARK-46793 Project: Spark Issue Type:

[jira] [Commented] (SPARK-45404) Support AWS_ENDPOINT_URL env variable

2024-01-22 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17809536#comment-17809536 ] Steve Loughran commented on SPARK-45404: Just saw this while working on SPARK-35878. If you

[jira] [Commented] (SPARK-47008) Spark to support S3 Express One Zone Storage

2024-05-13 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17846014#comment-17846014 ] Steve Loughran commented on SPARK-47008: yes, that looks like it. real PITA this feature, though

[jira] [Commented] (SPARK-48123) Provide a constant table schema for querying structured logs

2024-05-07 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17844245#comment-17844245 ] Steve Loughran commented on SPARK-48123: this doesn't handle nested stack traces. I seem to have

[jira] [Commented] (SPARK-38330) Certificate doesn't match any of the subject alternative names: [*.s3.amazonaws.com, s3.amazonaws.com]

2024-03-21 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17829521#comment-17829521 ] Steve Loughran commented on SPARK-38330: [~jpanda] a bit late but your problem is the WONTFIX

[jira] [Commented] (SPARK-41392) spark builds against hadoop trunk/3.4.0-SNAPSHOT fail in scala-maven plugin

2024-03-01 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17822572#comment-17822572 ] Steve Loughran commented on SPARK-41392: expect an official release this week; this pr will

[jira] [Commented] (SPARK-41392) spark builds against hadoop trunk/3.4.0-SNAPSHOT fail in scala-maven plugin

2024-02-28 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17821694#comment-17821694 ] Steve Loughran commented on SPARK-41392: Hadoop 3.4.0 RC2 exhibits this; spark needs its patches

[jira] [Updated] (SPARK-41392) spark builds against hadoop trunk/3.4.0-SNAPSHOT fail in scala-maven plugin

2024-02-28 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-41392: --- Priority: Major (was: Minor) > spark builds against hadoop trunk/3.4.0-SNAPSHOT fail in

[jira] [Commented] (SPARK-44970) Spark History File Uploads Can Fail on S3

2024-05-29 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17850475#comment-17850475 ] Steve Loughran commented on SPARK-44970: correct. file is only saved on close(). The incomplete

[jira] [Commented] (SPARK-48571) Reduce the number of accesses to S3 object storage

2024-06-11 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17854030#comment-17854030 ] Steve Loughran commented on SPARK-48571: The hadoop openFile() code came with HADOOP-15229 ;

<    4   5   6   7   8   9