[jira] [Commented] (SPARK-24492) Endless attempted task when TaskCommitDenied exception writing to S3A

[ https://issues.apache.org/jira/browse/SPARK-24492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16507113#comment-16507113 ] Steve Loughran commented on SPARK-24492: well, you've got a consistency problem

[jira] [Commented] (SPARK-24476) java.net.SocketTimeoutException: Read timed out under jets3t while running the Spark Structured Streaming

[ https://issues.apache.org/jira/browse/SPARK-24476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16513933#comment-16513933 ] Steve Loughran commented on SPARK-24476: * Use S3A, as S3n is unsupported and de

[jira] [Resolved] (SPARK-26284) Spark History server object vs file storage behavior difference

[ https://issues.apache.org/jira/browse/SPARK-26284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved SPARK-26284. Resolution: Invalid > Spark History server object vs file storage behavior difference > --

[jira] [Commented] (SPARK-26284) Spark History server object vs file storage behavior difference

[ https://issues.apache.org/jira/browse/SPARK-26284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16725958#comment-16725958 ] Steve Loughran commented on SPARK-26284: This is how object stores work. PUTs /

[jira] [Commented] (SPARK-2984) FileNotFoundException on _temporary directory

[ https://issues.apache.org/jira/browse/SPARK-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16732388#comment-16732388 ] Steve Loughran commented on SPARK-2984: --- Gaurav, if this has returned in a 2.x vers

[jira] [Commented] (SPARK-23534) Spark run on Hadoop 3.0.0

[ https://issues.apache.org/jira/browse/SPARK-23534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16767077#comment-16767077 ] Steve Loughran commented on SPARK-23534: bq. am curious to know if hadoop3 offer

[jira] [Comment Edited] (SPARK-21797) spark cannot read partitioned data in S3 that are partly in glacier

[ https://issues.apache.org/jira/browse/SPARK-21797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16201917#comment-16201917 ] Steve Loughran edited comment on SPARK-21797 at 2/14/19 3:54 PM: -

[jira] [Commented] (SPARK-25766) AMCredentialRenewer can leak FS clients

[ https://issues.apache.org/jira/browse/SPARK-25766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16768747#comment-16768747 ] Steve Loughran commented on SPARK-25766: sounds good, though I should check. The

[jira] [Commented] (SPARK-19111) S3 Mesos history upload fails silently if too large

[ https://issues.apache.org/jira/browse/SPARK-19111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15821773#comment-15821773 ] Steve Loughran commented on SPARK-19111: Just realised one more thing If the all

[jira] [Updated] (SPARK-11353) Writing to S3 buckets, which only support AWS4-HMAC-SHA256 fails with s3n URLs

[ https://issues.apache.org/jira/browse/SPARK-11353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-11353: --- Summary: Writing to S3 buckets, which only support AWS4-HMAC-SHA256 fails with s3n URLs (was

[jira] [Resolved] (SPARK-11353) Writing to S3 buckets, which only support AWS4-HMAC-SHA256 fails with s3n URLs

[ https://issues.apache.org/jira/browse/SPARK-11353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved SPARK-11353. Resolution: Duplicate This is a duplicate of SPARK-13044; that's transitive a WONTFIX due t

[jira] [Commented] (SPARK-19111) S3 Mesos history upload fails silently if too large

[ https://issues.apache.org/jira/browse/SPARK-19111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15848403#comment-15848403 ] Steve Loughran commented on SPARK-19111: Charles, you might also want to keep an

[jira] [Comment Edited] (SPARK-19407) defaultFS is used FileSystem.get instead of getting it from uri scheme

[ https://issues.apache.org/jira/browse/SPARK-19407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15851290#comment-15851290 ] Steve Loughran edited comment on SPARK-19407 at 2/3/17 10:17 AM: --

[jira] [Commented] (SPARK-19407) defaultFS is used FileSystem.get instead of getting it from uri scheme

[ https://issues.apache.org/jira/browse/SPARK-19407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15851290#comment-15851290 ] Steve Loughran commented on SPARK-19407: Yes, looks like {{StreamMetadata.read()

[jira] [Commented] (SPARK-19715) Option to Strip Paths in FileSource

[ https://issues.apache.org/jira/browse/SPARK-19715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15882995#comment-15882995 ] Steve Loughran commented on SPARK-19715: This is a silly question, but has the si

[jira] [Updated] (SPARK-14561) History Server does not see new logs in S3

[ https://issues.apache.org/jira/browse/SPARK-14561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-14561: --- Component/s: Spark Core > History Server does not see new logs in S3 > --

[jira] [Commented] (SPARK-19715) Option to Strip Paths in FileSource

[ https://issues.apache.org/jira/browse/SPARK-19715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15884230#comment-15884230 ] Steve Loughran commented on SPARK-19715: OK. I'd recommend going twith Path.getUR

[jira] [Comment Edited] (SPARK-19715) Option to Strip Paths in FileSource

[ https://issues.apache.org/jira/browse/SPARK-19715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15884230#comment-15884230 ] Steve Loughran edited comment on SPARK-19715 at 2/25/17 1:24 PM: --

[jira] [Created] (SPARK-19739) SparkHadoopUtil.appendS3AndSparkHadoopConfigurations to propagate full set of AWS env vars

Steve Loughran created SPARK-19739: -- Summary: SparkHadoopUtil.appendS3AndSparkHadoopConfigurations to propagate full set of AWS env vars Key: SPARK-19739 URL: https://issues.apache.org/jira/browse/SPARK-19739

[jira] [Updated] (SPARK-7481) Add spark-cloud module to pull in object store support

[ https://issues.apache.org/jira/browse/SPARK-7481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-7481: -- Description: To keep the s3n classpath right, to add s3a, swift & azure, the dependencies of sp

[jira] [Updated] (SPARK-7481) Add spark-cloud module to pull in object store support

[ https://issues.apache.org/jira/browse/SPARK-7481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-7481: -- Affects Version/s: (was: 1.3.1) 2.1.0 > Add spark-cloud module to pul

[jira] [Updated] (SPARK-7481) Add spark-hadoop-cloud module to pull in object store support

[ https://issues.apache.org/jira/browse/SPARK-7481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-7481: -- Summary: Add spark-hadoop-cloud module to pull in object store support (was: Add spark-cloud mo

[jira] [Commented] (SPARK-6951) History server slow startup if the event log directory is large

[ https://issues.apache.org/jira/browse/SPARK-6951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15890040#comment-15890040 ] Steve Loughran commented on SPARK-6951: --- Having been downstream of YARN timeline ser

[jira] [Commented] (SPARK-20799) Unable to infer schema for ORC/Parquet on S3N when secrets are in the URL

[ https://issues.apache.org/jira/browse/SPARK-20799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16611995#comment-16611995 ] Steve Loughran commented on SPARK-20799: Update: Hadoop 3.3+ will remove all sup

[jira] [Commented] (SPARK-20153) Support Multiple aws credentials in order to access multiple Hive on S3 table in spark application

[ https://issues.apache.org/jira/browse/SPARK-20153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16612001#comment-16612001 ] Steve Loughran commented on SPARK-20153: bq. Amazon EMR does not currently suppo

[jira] [Commented] (SPARK-25480) Dynamic partitioning + saveAsTable with multiple partition columns create empty directory

[ https://issues.apache.org/jira/browse/SPARK-25480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16622265#comment-16622265 ] Steve Loughran commented on SPARK-25480: * Does this only happen with EMR S3? *

[jira] [Commented] (SPARK-24523) InterruptedException when closing SparkContext

[ https://issues.apache.org/jira/browse/SPARK-24523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16623357#comment-16623357 ] Steve Loughran commented on SPARK-24523: This sounds like a near replica of (HAD

[jira] [Commented] (SPARK-25480) Dynamic partitioning + saveAsTable with multiple partition columns create empty directory

[ https://issues.apache.org/jira/browse/SPARK-25480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16623367#comment-16623367 ] Steve Loughran commented on SPARK-25480: OK, two suggestions * can you replicat

[jira] [Created] (SPARK-25766) AMCredentialRenewer can leak FS clients

Steve Loughran created SPARK-25766: -- Summary: AMCredentialRenewer can leak FS clients Key: SPARK-25766 URL: https://issues.apache.org/jira/browse/SPARK-25766 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-19143) API in Spark for distributing new delegation tokens (Improve delegation token handling in secure clusters)

[ https://issues.apache.org/jira/browse/SPARK-19143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656476#comment-16656476 ] Steve Loughran commented on SPARK-19143: I'm looking at this as I add DTs into S

[jira] [Comment Edited] (SPARK-21725) spark thriftserver insert overwrite table partition select

[ https://issues.apache.org/jira/browse/SPARK-21725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656631#comment-16656631 ] Steve Loughran edited comment on SPARK-21725 at 10/19/18 10:48 AM: ---

[jira] [Commented] (SPARK-21725) spark thriftserver insert overwrite table partition select

[ https://issues.apache.org/jira/browse/SPARK-21725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656631#comment-16656631 ] Steve Loughran commented on SPARK-21725: bq. can we fix it on the Hadoop side?

[jira] [Commented] (SPARK-19143) API in Spark for distributing new delegation tokens (Improve delegation token handling in secure clusters)

[ https://issues.apache.org/jira/browse/SPARK-19143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657214#comment-16657214 ] Steve Loughran commented on SPARK-19143: thanks. when I get that S3a patch in s

[jira] [Commented] (SPARK-25855) Don't use Erasure Coding for event log files

[ https://issues.apache.org/jira/browse/SPARK-25855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16667270#comment-16667270 ] Steve Loughran commented on SPARK-25855: thx for the mention. yes, looking @ str

[jira] [Commented] (SPARK-25966) "EOF Reached the end of stream with bytes left to read" while reading/writing to Parquets

[ https://issues.apache.org/jira/browse/SPARK-25966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679611#comment-16679611 ] Steve Loughran commented on SPARK-25966: bq. It looks to me like a problem in c

[jira] [Commented] (SPARK-10063) Remove DirectParquetOutputCommitter

[ https://issues.apache.org/jira/browse/SPARK-10063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15490023#comment-15490023 ] Steve Loughran commented on SPARK-10063: Amazon EMR's s3 is its own codebase; afr

[jira] [Commented] (SPARK-17593) list files on s3 very slow

[ https://issues.apache.org/jira/browse/SPARK-17593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15503583#comment-15503583 ] Steve Loughran commented on SPARK-17593: Sean is right: this is primarily S3, or

[jira] [Commented] (SPARK-17593) list files on s3 very slow

[ https://issues.apache.org/jira/browse/SPARK-17593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15503663#comment-15503663 ] Steve Loughran commented on SPARK-17593: Looking at the dir tree, anything you co

[jira] [Resolved] (SPARK-17259) Hadoop 2.7 profile to depend on Hadoop 2.7.3

[ https://issues.apache.org/jira/browse/SPARK-17259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved SPARK-17259. Resolution: Duplicate > Hadoop 2.7 profile to depend on Hadoop 2.7.3 >

[jira] [Commented] (SPARK-15343) NoClassDefFoundError when initializing Spark with YARN

[ https://issues.apache.org/jira/browse/SPARK-15343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15535594#comment-15535594 ] Steve Loughran commented on SPARK-15343: Jo, this is an eternal problem with the

[jira] [Commented] (SPARK-10063) Remove DirectParquetOutputCommitter

[ https://issues.apache.org/jira/browse/SPARK-10063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15548557#comment-15548557 ] Steve Loughran commented on SPARK-10063: Looking at the git logs to see which cod

[jira] [Commented] (SPARK-15343) NoClassDefFoundError when initializing Spark with YARN

[ https://issues.apache.org/jira/browse/SPARK-15343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15568180#comment-15568180 ] Steve Loughran commented on SPARK-15343: this is a tough problem with Hadoop core

[jira] [Commented] (SPARK-15343) NoClassDefFoundError when initializing Spark with YARN

[ https://issues.apache.org/jira/browse/SPARK-15343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15568187#comment-15568187 ] Steve Loughran commented on SPARK-15343: There is a very quick fix here, to stop

[jira] [Commented] (SPARK-14561) History Server does not see new logs in S3

[ https://issues.apache.org/jira/browse/SPARK-14561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15572401#comment-15572401 ] Steve Loughran commented on SPARK-14561: To clarify: it's not changes in existing

[jira] [Commented] (SPARK-9004) Add s3 bytes read/written metrics

[ https://issues.apache.org/jira/browse/SPARK-9004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15572414#comment-15572414 ] Steve Loughran commented on SPARK-9004: --- HADOOP-13605 added a whole new set of count

[jira] [Commented] (SPARK-12571) AWS credentials not available for read.parquet in SQLContext

[ https://issues.apache.org/jira/browse/SPARK-12571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15572442#comment-15572442 ] Steve Loughran commented on SPARK-12571: Means the credentials aren't at the far

[jira] [Commented] (SPARK-8437) Using directory path without wildcard for filename slow for large number of files with wholeTextFiles and binaryFiles

[ https://issues.apache.org/jira/browse/SPARK-8437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15572468#comment-15572468 ] Steve Loughran commented on SPARK-8437: --- Just came across by way of comments in the

[jira] [Updated] (SPARK-7481) Add spark-cloud module to pull in object store support; test

[ https://issues.apache.org/jira/browse/SPARK-7481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-7481: -- Summary: Add spark-cloud module to pull in object store support; test (was: Add spark-cloud mod

[jira] [Commented] (SPARK-7481) Add spark-cloud module to pull in object store support; test

[ https://issues.apache.org/jira/browse/SPARK-7481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15582210#comment-15582210 ] Steve Loughran commented on SPARK-7481: --- For anyone watching this; the code is prett

[jira] [Commented] (SPARK-5925) YARN - Spark progress bar stucks at 10% but after finishing shows 100%

[ https://issues.apache.org/jira/browse/SPARK-5925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15584969#comment-15584969 ] Steve Loughran commented on SPARK-5925: --- looking at this, I'm confused about what I'

[jira] [Commented] (SPARK-23308) ignoreCorruptFiles should not ignore retryable IOException

[ https://issues.apache.org/jira/browse/SPARK-23308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354784#comment-16354784 ] Steve Loughran commented on SPARK-23308: bq. I have not heard this come up before

[jira] [Comment Edited] (SPARK-23308) ignoreCorruptFiles should not ignore retryable IOException

[ https://issues.apache.org/jira/browse/SPARK-23308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354784#comment-16354784 ] Steve Loughran edited comment on SPARK-23308 at 2/7/18 12:26 AM: --

[jira] [Commented] (SPARK-23308) ignoreCorruptFiles should not ignore retryable IOException

[ https://issues.apache.org/jira/browse/SPARK-23308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16357266#comment-16357266 ] Steve Loughran commented on SPARK-23308: bq. Other option would be creating a spe

[jira] [Commented] (SPARK-23308) ignoreCorruptFiles should not ignore retryable IOException

[ https://issues.apache.org/jira/browse/SPARK-23308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16357387#comment-16357387 ] Steve Loughran commented on SPARK-23308: HADOOP-15216 covers S3A handling this fa

[jira] [Commented] (SPARK-23308) ignoreCorruptFiles should not ignore retryable IOException

[ https://issues.apache.org/jira/browse/SPARK-23308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16360809#comment-16360809 ] Steve Loughran commented on SPARK-23308: BTW bq I should get at least ~82k part

[jira] [Comment Edited] (SPARK-23308) ignoreCorruptFiles should not ignore retryable IOException

[ https://issues.apache.org/jira/browse/SPARK-23308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16360809#comment-16360809 ] Steve Loughran edited comment on SPARK-23308 at 2/14/18 11:27 AM: -

[jira] [Commented] (SPARK-23308) ignoreCorruptFiles should not ignore retryable IOException

[ https://issues.apache.org/jira/browse/SPARK-23308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16365495#comment-16365495 ] Steve Loughran commented on SPARK-23308: I'm going to recommend this is closed as

[jira] [Commented] (SPARK-23420) Datasource loading not handling paths with regex chars.

[ https://issues.apache.org/jira/browse/SPARK-23420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16369330#comment-16369330 ] Steve Loughran commented on SPARK-23420: Can I note that if there's a colon in th

[jira] [Commented] (SPARK-11182) HDFS Delegation Token will be expired when calling "UserGroupInformation.getCurrentUser.addCredentials" in HA mode

[ https://issues.apache.org/jira/browse/SPARK-11182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16369914#comment-16369914 ] Steve Loughran commented on SPARK-11182: bug is in HDFS; been fixed in 2.8.2+ wit

[jira] [Commented] (SPARK-16996) Hive ACID delta files not seen

[ https://issues.apache.org/jira/browse/SPARK-16996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16380370#comment-16380370 ] Steve Loughran commented on SPARK-16996: Like I said, Spark is trouble; we've jus

[jira] [Commented] (SPARK-23652) Spark Connection with S3

[ https://issues.apache.org/jira/browse/SPARK-23652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395428#comment-16395428 ] Steve Loughran commented on SPARK-23652: Don't use the s3:// connector which ship

[jira] [Created] (SPARK-23654) cut jets3t as a dependency of spark-core

Steve Loughran created SPARK-23654: -- Summary: cut jets3t as a dependency of spark-core Key: SPARK-23654 URL: https://issues.apache.org/jira/browse/SPARK-23654 Project: Spark Issue Type: Impr

[jira] [Updated] (SPARK-23652) Verify error when using ASF s3:// connector.

[ https://issues.apache.org/jira/browse/SPARK-23652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-23652: --- Summary: Verify error when using ASF s3:// connector. (was: Spark Connection with S3) > Ver

[jira] [Updated] (SPARK-23652) Verify error when using ASF s3:// connector. & Jetty 0.9.4

[ https://issues.apache.org/jira/browse/SPARK-23652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-23652: --- Summary: Verify error when using ASF s3:// connector. & Jetty 0.9.4 (was: Verify error when

[jira] [Updated] (SPARK-23652) Verify error when using ASF s3:// connector. & Jetty 0.9.4

[ https://issues.apache.org/jira/browse/SPARK-23652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-23652: --- Priority: Minor (was: Critical) > Verify error when using ASF s3:// connector. & Jetty 0.9.4

[jira] [Commented] (SPARK-23652) Verify error when using ASF s3:// connector. & Jetty 0.9.4

[ https://issues.apache.org/jira/browse/SPARK-23652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395449#comment-16395449 ] Steve Loughran commented on SPARK-23652: this stack trace is just HADOOP-11086; t

[jira] [Commented] (SPARK-23654) cut jets3t as a dependency of spark-core

[ https://issues.apache.org/jira/browse/SPARK-23654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395451#comment-16395451 ] Steve Loughran commented on SPARK-23654: SPARK-22634 highights that the spark-had

[jira] [Updated] (SPARK-23654) cut jets3t as a dependency of spark-core; exclude it from hadoop-cloud module as incompatible

[ https://issues.apache.org/jira/browse/SPARK-23654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-23654: --- Summary: cut jets3t as a dependency of spark-core; exclude it from hadoop-cloud module as inc

[jira] [Resolved] (SPARK-23652) Verify error when using ASF s3:// connector. & Jetty 0.9.4

[ https://issues.apache.org/jira/browse/SPARK-23652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved SPARK-23652. Resolution: Duplicate > Verify error when using ASF s3:// connector. & Jetty 0.9.4 > --

[jira] [Commented] (SPARK-23654) cut jets3t as a dependency of spark-core; exclude it from hadoop-cloud module as incompatible

[ https://issues.apache.org/jira/browse/SPARK-23654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16398653#comment-16398653 ] Steve Loughran commented on SPARK-23654: # In Hadoop 3.x anyone trying to create

[jira] [Commented] (SPARK-22634) Update Bouncy castle dependency

[ https://issues.apache.org/jira/browse/SPARK-22634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16398659#comment-16398659 ] Steve Loughran commented on SPARK-22634: moving to jets3t 0.9.4 breaks the (legac

[jira] [Created] (SPARK-23681) Move OrcFileFormat switch to using hadoop.mapreduce classes

Steve Loughran created SPARK-23681: -- Summary: Move OrcFileFormat switch to using hadoop.mapreduce classes Key: SPARK-23681 URL: https://issues.apache.org/jira/browse/SPARK-23681 Project: Spark

[jira] [Updated] (SPARK-23681) Switch OrcFileFormat to using newer hadoop.mapreduce output classes

[ https://issues.apache.org/jira/browse/SPARK-23681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-23681: --- Summary: Switch OrcFileFormat to using newer hadoop.mapreduce output classes (was: Move OrcF

[jira] [Updated] (SPARK-23681) Switch OrcFileFormat to newer hadoop.mapreduce output classes

[ https://issues.apache.org/jira/browse/SPARK-23681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-23681: --- Summary: Switch OrcFileFormat to newer hadoop.mapreduce output classes (was: Switch OrcFileF

[jira] [Created] (SPARK-23683) FileCommitProtocol.instantiate to require 3-arg constructor for dynamic partition overwrite

Steve Loughran created SPARK-23683: -- Summary: FileCommitProtocol.instantiate to require 3-arg constructor for dynamic partition overwrite Key: SPARK-23683 URL: https://issues.apache.org/jira/browse/SPARK-23683

[jira] [Commented] (SPARK-21962) Distributed Tracing in Spark

[ https://issues.apache.org/jira/browse/SPARK-21962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16401791#comment-16401791 ] Steve Loughran commented on SPARK-21962: HTrace instrumentation would be the stra

[jira] [Updated] (SPARK-23654) Cut jets3t as a dependency of spark-core; exclude it from hadoop-cloud module as incompatible

[ https://issues.apache.org/jira/browse/SPARK-23654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-23654: --- Summary: Cut jets3t as a dependency of spark-core; exclude it from hadoop-cloud module as inc

[jira] [Comment Edited] (SPARK-22513) Provide build profile for hadoop 2.8

[ https://issues.apache.org/jira/browse/SPARK-22513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16413685#comment-16413685 ] Steve Loughran edited comment on SPARK-22513 at 3/26/18 11:11 AM: -

[jira] [Commented] (SPARK-22513) Provide build profile for hadoop 2.8

[ https://issues.apache.org/jira/browse/SPARK-22513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16413685#comment-16413685 ] Steve Loughran commented on SPARK-22513: API wise, everything compiled against 2.

[jira] [Comment Edited] (SPARK-22513) Provide build profile for hadoop 2.8

[ https://issues.apache.org/jira/browse/SPARK-22513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16413685#comment-16413685 ] Steve Loughran edited comment on SPARK-22513 at 3/26/18 11:10 AM: -

[jira] [Comment Edited] (SPARK-22513) Provide build profile for hadoop 2.8

[ https://issues.apache.org/jira/browse/SPARK-22513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16413685#comment-16413685 ] Steve Loughran edited comment on SPARK-22513 at 3/26/18 11:10 AM: -

[jira] [Comment Edited] (SPARK-22513) Provide build profile for hadoop 2.8

[ https://issues.apache.org/jira/browse/SPARK-22513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16413685#comment-16413685 ] Steve Loughran edited comment on SPARK-22513 at 3/26/18 11:13 AM: -

[jira] [Commented] (SPARK-22513) Provide build profile for hadoop 2.8

[ https://issues.apache.org/jira/browse/SPARK-22513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16415320#comment-16415320 ] Steve Loughran commented on SPARK-22513: bq. So I guess at the summary level Sean

[jira] [Created] (SPARK-23807) Add Hadoop 3 profile with relevant POM fix ups, cloud-storage artifacts and binding

Steve Loughran created SPARK-23807: -- Summary: Add Hadoop 3 profile with relevant POM fix ups, cloud-storage artifacts and binding Key: SPARK-23807 URL: https://issues.apache.org/jira/browse/SPARK-23807

[jira] [Commented] (SPARK-23807) Add Hadoop 3 profile with relevant POM fix ups, cloud-storage artifacts and binding

[ https://issues.apache.org/jira/browse/SPARK-23807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16419017#comment-16419017 ] Steve Loughran commented on SPARK-23807: yes, this profile is part of the hadoop

[jira] [Updated] (SPARK-23807) Add Hadoop 3 profile with relevant POM fix ups, cloud-storage artifacts and binding

[ https://issues.apache.org/jira/browse/SPARK-23807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-23807: --- Issue Type: Sub-task (was: Improvement) Parent: SPARK-23534 > Add Hadoop 3 profile w

[jira] [Commented] (SPARK-23534) Spark run on Hadoop 3.0.0

[ https://issues.apache.org/jira/browse/SPARK-23534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16419053#comment-16419053 ] Steve Loughran commented on SPARK-23534: [~nchammas] bq. Cloudera still ships 2.

[jira] [Updated] (SPARK-23807) Add Hadoop 3 profile with relevant POM fix ups

[ https://issues.apache.org/jira/browse/SPARK-23807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-23807: --- Summary: Add Hadoop 3 profile with relevant POM fix ups (was: Add Hadoop 3 profile with rele

[jira] [Commented] (SPARK-22919) Bump Apache httpclient versions

[ https://issues.apache.org/jira/browse/SPARK-22919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16431198#comment-16431198 ] Steve Loughran commented on SPARK-22919: going to highlight this appears to break

[jira] [Commented] (SPARK-23966) Refactoring all checkpoint file writing logic in a common interface

[ https://issues.apache.org/jira/browse/SPARK-23966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437201#comment-16437201 ] Steve Loughran commented on SPARK-23966: w.r.t FileContext.rename vs FileSystem.r

[jira] [Created] (SPARK-23977) Add committer binding to Hadoop 3.1 PathOutputCommitter Mechanism

Steve Loughran created SPARK-23977: -- Summary: Add committer binding to Hadoop 3.1 PathOutputCommitter Mechanism Key: SPARK-23977 URL: https://issues.apache.org/jira/browse/SPARK-23977 Project: Spark

[jira] [Updated] (SPARK-23977) Add commit protocol binding to Hadoop 3.1 PathOutputCommitter mechanism

[ https://issues.apache.org/jira/browse/SPARK-23977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-23977: --- Summary: Add commit protocol binding to Hadoop 3.1 PathOutputCommitter mechanism (was: Add c

[jira] [Commented] (SPARK-21074) Parquet files are read fully even though only count() is requested

[ https://issues.apache.org/jira/browse/SPARK-21074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16281901#comment-16281901 ] Steve Loughran commented on SPARK-21074: Is there any update on this? # I'd lik

[jira] [Commented] (SPARK-18294) Implement commit protocol to support `mapred` package's committer

[ https://issues.apache.org/jira/browse/SPARK-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16292595#comment-16292595 ] Steve Loughran commented on SPARK-18294: Following up on this, one question: Why

[jira] [Commented] (SPARK-7755) MetadataCache.refresh does not take into account _SUCCESS

[ https://issues.apache.org/jira/browse/SPARK-7755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16313803#comment-16313803 ] Steve Loughran commented on SPARK-7755: --- I concur with [~hyukjin.kwon] here: if inco

[jira] [Resolved] (SPARK-18883) FileNotFoundException on _temporary directory

[ https://issues.apache.org/jira/browse/SPARK-18883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved SPARK-18883. Resolution: Won't Fix I'm going to close as a WONTFIX, because the solution is "don't use

[jira] [Commented] (SPARK-23050) Structured Streaming with S3 file source duplicates data because of eventual consistency.

[ https://issues.apache.org/jira/browse/SPARK-23050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324745#comment-16324745 ] Steve Loughran commented on SPARK-23050: this s3n is the amazon EMR closed source

[jira] [Commented] (SPARK-23050) Structured Streaming with S3 file source duplicates data because of eventual consistency.

[ https://issues.apache.org/jira/browse/SPARK-23050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16326330#comment-16326330 ] Steve Loughran commented on SPARK-23050: Quick review of the code Yes, there's p

[jira] [Comment Edited] (SPARK-23050) Structured Streaming with S3 file source duplicates data because of eventual consistency.

[ https://issues.apache.org/jira/browse/SPARK-23050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16326330#comment-16326330 ] Steve Loughran edited comment on SPARK-23050 at 1/15/18 3:24 PM: --

[jira] [Commented] (SPARK-6305) Add support for log4j 2.x to Spark

[ https://issues.apache.org/jira/browse/SPARK-6305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16327144#comment-16327144 ] Steve Loughran commented on SPARK-6305: --- It'll be related to HADOOP-12956 , HDFS-128

<    1   2   3   4   5   6   7   8   9   >