[GitHub] [spark] dongjoon-hyun commented on pull request #32518: [SPARK-35383][CORE] Improve s3a magic committer support by inferring missing configs

2022-02-24 Thread GitBox
dongjoon-hyun commented on pull request #32518: URL: https://github.com/apache/spark/pull/32518#issuecomment-1050425417 Hi, @itayB . If you are using Apache Spark 3.2.1 with Hadoop 3.3.1, you don't need to the first one. However, you still needs the Parquet recommendation. -- This is

[GitHub] [spark] dongjoon-hyun commented on pull request #32518: [SPARK-35383][CORE] Improve s3a magic committer support by inferring missing configs

2021-05-12 Thread GitBox
dongjoon-hyun commented on pull request #32518: URL: https://github.com/apache/spark/pull/32518#issuecomment-840019899 Although AppVeyor build failed due to timeout, Jenkins passed. Merged to master. Thank you, @dbtsai , @HyukjinKwon , @steveloughran . This is a part of efforts to give

[GitHub] [spark] dongjoon-hyun commented on pull request #32518: [SPARK-35383][CORE] Improve s3a magic committer support by inferring missing configs

2021-05-12 Thread GitBox
dongjoon-hyun commented on pull request #32518: URL: https://github.com/apache/spark/pull/32518#issuecomment-839992094 Here are my thoughts. > Be aware that https://issues.apache.org/jira/browse/HADOOP-17483 turns the magic committer on everywhere Of course, we've been

[GitHub] [spark] dongjoon-hyun commented on pull request #32518: [SPARK-35383][CORE] Improve s3a magic committer support by inferring missing configs

2021-05-12 Thread GitBox
dongjoon-hyun commented on pull request #32518: URL: https://github.com/apache/spark/pull/32518#issuecomment-839980053 Thank you so much for review and comments, @steveloughran ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] dongjoon-hyun commented on pull request #32518: [SPARK-35383][CORE] Improve s3a magic committer support by inferring missing configs

2021-05-12 Thread GitBox
dongjoon-hyun commented on pull request #32518: URL: https://github.com/apache/spark/pull/32518#issuecomment-839841587 > LGTM. Should we eventually do this in Hadoop, cc @steveloughran and @dongjoon-hyun ? Thank you for review, @dbtsai . The following two are Spark configurations.

[GitHub] [spark] dongjoon-hyun commented on pull request #32518: [SPARK-35383][CORE] Improve s3a magic committer support by inferring missing configs

2021-05-11 Thread GitBox
dongjoon-hyun commented on pull request #32518: URL: https://github.com/apache/spark/pull/32518#issuecomment-839470621 Hi, @steveloughran . Could you review this PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use