steveloughran commented on pull request #2530: URL: https://github.com/apache/hadoop/pull/2530#issuecomment-742788785
Matching spark PR is https://github.com/apache/spark/pull/30714 I've now done downstream testing in spark local mode, verifying from the log that the attribute was read. ``` 2020-12-10 14:56:16,760 [Executor task launch worker for task 3] INFO magic.MagicCommitTracker (MagicCommitTracker.java:aboutToComplete(134)) - Uncommitted data pending to file s3a://stevel-london/cloud-integration/DELAY_LISTING_ME/S3ACommitDataframeSuite/dataframe-committer/committer-magic-parquet/parquet/__magic/job-f718ab6a-27df-40fe-8b25-a8d83cd25dca/tasks/attempt_202012101456156213593579150946569_0005_m_000000_3/__base/year=2017/month=1/part-00000-fe9c22b3-8639-451e-9843-7498f2ea5f41.c000.snappy.parquet; commit metadata for 1 parts in cloud-integration/DELAY_LISTING_ME/S3ACommitDataframeSuite/dataframe-committer/committer-magic-parquet/parquet/__magic/job-f718ab6a-27df-40fe-8b25-a8d83cd25dca/tasks/attempt_202012101456156213593579150946569_0005_m_000000_3/__base/year=2017/month=1/part-00000-fe9c22b3-8639-451e-9843-7498f2ea5f41.c000.snappy.parquet.pending. sixe: 6286 byte(s) 2020-12-10 14:56:17,513 [Executor task launch worker for task 3] INFO s3a.S3ABlockOutputStream (S3ABlockOutputStream.java:close(388)) - File cloud-integration/DELAY_LISTING_ME/S3ACommitDataframeSuite/dataframe-committer/committer-magic-parquet/parquet/year=2017/month=1/part-00000-fe9c22b3-8639-451e-9843-7498f2ea5f41.c000.snappy.parquet will be visible when the job is committed 2020-12-10 14:56:17,705 [Executor task launch worker for task 3] DEBUG datasources.BasicWriteTaskStatsTracker (Logging.scala:logDebug(61)) - File Length statistics for s3a://stevel-london/cloud-integration/DELAY_LISTING_ME/S3ACommitDataframeSuite/dataframe-committer/committer-magic-parquet/parquet/__magic/job-f718ab6a-27df-40fe-8b25-a8d83cd25dca/tasks/attempt_202012101456156213593579150946569_0005_m_000000_3/__base/year=2017/month=1/part-00000-fe9c22b3-8639-451e-9843-7498f2ea5f41.c000.snappy.parquet retrieved from XAttr: 6286 ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
