Apache Hadoop qbt Report: branch-3.3+JDK8 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/111/ No changes -1 overall The following subsystems voted -1: blanks pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-common-project/hadoop-common/src/test/resources/xml/external-dtd.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml Failed junit tests : hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService cc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/111/artifact/out/results-compile-cc-root.txt [48K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/111/artifact/out/results-compile-javac-root.txt [380K] blanks: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/111/artifact/out/blanks-eol.txt [14M] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/111/artifact/out/blanks-tabs.txt [2.0M] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/111/artifact/out/results-checkstyle-root.txt [14M] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/111/artifact/out/results-pathlen.txt [16K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/111/artifact/out/results-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/111/artifact/out/results-shellcheck.txt [20K] xml: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/111/artifact/out/xml.txt [28K] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/111/artifact/out/results-javadoc-javadoc-root.txt [972K] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/111/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [736K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/111/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt [96K] Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK11 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/498/ [May 31, 2023, 6:26:31 AM] (github) HDFS-17026. RBF: NamenodeHeartbeatService should update JMX report with configurable frequency. (#5691). Contributed by hchaverri. [May 31, 2023, 10:40:07 AM] (github) HDFS-17000. Fix faulty loop condition in TestDFSStripedOutputStreamUpdatePipeline (#5699). Contributed by Marcono1234. [May 31, 2023, 5:20:19 PM] (github) HDFS-17027. RBF: Adds auto-msync support for clients connecting to routers. (#5693) [Jun 1, 2023, 2:53:48 AM] (github) HDFS-16996. Fix flaky testFsCloseAfterClusterShutdown in TestFileCreation (#5697). Contributed by Nishtha Shah. [Jun 1, 2023, 3:02:38 AM] (github) HDFS-17031. Reduce some repeated codes in RouterRpcServer. (#5701). Contributed by Chengwei Wang. -1 overall The following subsystems voted -1: blanks hadolint mvnsite pathlen spotbugs unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-common-project/hadoop-common/src/test/resources/xml/external-dtd.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml spotbugs : module:hadoop-hdfs-project/hadoop-hdfs Redundant nullcheck of oldLock, which is known to be non-null in org.apache.hadoop.hdfs.server.datanode.DataStorage.isPreUpgradableLayout(Storage$StorageDirectory)) Redundant null check at DataStorage.java:is known to be non-null in org.apache.hadoop.hdfs.server.datanode.DataStorage.isPreUpgradableLayout(Storage$StorageDirectory)) Redundant null check at DataStorage.java:[line 695] Redundant nullcheck of metaChannel, which is known to be non-null in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MappableBlockLoader.verifyChecksum(long, FileInputStream, FileChannel, String) Redundant null check at MappableBlockLoader.java:is known to be non-null in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MappableBlockLoader.verifyChecksum(long, FileInputStream, FileChannel, String) Redundant null check at MappableBlockLoader.java:[line 138] Redundant nullcheck of blockChannel, which is known to be non-null in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MemoryMappableBlockLoader.load(long, FileInputStream, FileInputStream, String, ExtendedBlockId) Redundant null check at MemoryMappableBlockLoader.java:is known to be non-null in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MemoryMappableBlockLoader.load(long, FileInputStream, FileInputStream, String, ExtendedBlockId) Redundant null check at MemoryMappableBlockLoader.java:[line 75] Redundant nullcheck of blockChannel, which is known to be non-null in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.NativePmemMappableBlockLoader.load(long, FileInputStream, FileInputStream, String, ExtendedBlockId) Redundant null check at NativePmemMappableBlockLoader.java:is known to be non-null in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.NativePmemMappableBlockLoader.load(long, FileInputStream, FileInputStream, String, ExtendedBlockId) Redundant null check at NativePmemMappableBlockLoader.java:[line 85] Redundant nullcheck of metaChannel, which is known to be non-null in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.NativePmemMappableBlockLoader.verifyChecksumAndMapBlock(NativeIO$POSIX$$PmemMappedRegion,, long, FileInputStream, FileChannel, String) Redundant null check at NativePmemMappableBlockLoader.java:is known to be non-null in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.NativePmemMappableBlockLoader.verifyChecksumAndMapBlock(NativeIO$POSIX$$PmemMappedRegion,, long, FileInputStream, FileChannel, String) Redundant null check at NativePmemMappableBlockLoader.java:[line 130] org.apache.hadoop.hdfs.server.namenode.top.window.RollingWindowManager$UserCounts doesn't override java.util.ArrayList.equals(Object) At
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1244/ [May 31, 2023, 6:26:31 AM] (github) HDFS-17026. RBF: NamenodeHeartbeatService should update JMX report with configurable frequency. (#5691). Contributed by hchaverri. [May 31, 2023, 10:40:07 AM] (github) HDFS-17000. Fix faulty loop condition in TestDFSStripedOutputStreamUpdatePipeline (#5699). Contributed by Marcono1234. [May 31, 2023, 5:20:19 PM] (github) HDFS-17027. RBF: Adds auto-msync support for clients connecting to routers. (#5693) -1 overall The following subsystems voted -1: blanks hadolint pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-common-project/hadoop-common/src/test/resources/xml/external-dtd.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml Failed junit tests : hadoop.hdfs.server.namenode.ha.TestObserverNode hadoop.mapreduce.v2.TestUberAM hadoop.mapreduce.v2.TestMRJobsWithProfiler hadoop.mapreduce.v2.TestMRJobs cc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1244/artifact/out/results-compile-cc-root.txt [96K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1244/artifact/out/results-compile-javac-root.txt [12K] blanks: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1244/artifact/out/blanks-eol.txt [14M] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1244/artifact/out/blanks-tabs.txt [2.0M] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1244/artifact/out/results-checkstyle-root.txt [13M] hadolint: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1244/artifact/out/results-hadolint.txt [20K] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1244/artifact/out/results-pathlen.txt [16K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1244/artifact/out/results-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1244/artifact/out/results-shellcheck.txt [24K] xml: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1244/artifact/out/xml.txt [24K] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1244/artifact/out/results-javadoc-javadoc-root.txt [244K] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1244/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [452K] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1244/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt [72K] Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: [DISCUSS] new repo/sub-project, fs-api-shim
For the case in question, ORC can extend its shim module to do exactly this. Let me take a look at the async ORC patch to see what needs to happen. BTW, Steve I'll be at Buzzwords for your talk. It may make sense to have such a library, but ORC already has that mechanism. .. Owen On Thu, Jun 1, 2023 at 2:09 PM Ayush Saxena wrote: > That sounds good > > On Thu, 1 Jun 2023 at 18:09, Steve Loughran > wrote: > > > hadoop-api-shim ? > > > > On Thu, 1 Jun 2023 at 04:07, Ayush Saxena wrote: > > > > > +1, for the new repo. > > > > > > The name sounds fine, but good if we have scope of having “hadoop-” > > > prefix, we have that for almost all of the subprojects/modules > > > > > > Can hadoop-shims or hadoop-shims-api or something on similar lines > work? > > > > > > -Ayush > > > > > > > On 01-Jun-2023, at 1:18 AM, Steve Loughran > > > > > > wrote: > > > > > > > > I want to create a new repository to put a shim library to allow > > > previous > > > > > > > releases to access the more recent hadoop filesystem APIs -currently > > the > > > > open source implementations of parquet, ORC can't use vectored io, in > > > > particular, even though we can in Cloudera. Providing a shim opens > them > > > up > > > > to all *and* gets the APIs more broadly stressed/tested. > > > > This needs to be in its own repository, not just for rapid initial > > > release, > > > > but because it is designed to be built as old a version of hadoop we > > can > > > > reasonably support, which IMO means hadoop 3.1.0+. I know parquet > still > > > > wants to build against 2.8.x, but to claim support for hadoop 2 means > > > > "build and test on java7", which is unrealistic in 2023. > > > > > > > > Initial WiP implementation, which works with 3.1.0 and tests against > > > others > > > > https://github.com/steveloughran/fs-api-shim > > > > the complexity is about testing this -I have contract tests which > then > > > need > > > > to be executed on every supported hadoop release, which will need a > > > > separate module for each one. > > > > > > > > I can create the repo easily enough, just would like approval. And is > > the > > > > name OK? > > > > > > > > steve > > > > > >
[jira] [Resolved] (HADOOP-18755) openFile builder new optLong() methods break hbase-filesystem
[ https://issues.apache.org/jira/browse/HADOOP-18755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-18755. - Fix Version/s: 3.4.0 3.3.9 Resolution: Fixed > openFile builder new optLong() methods break hbase-filesystem > - > > Key: HADOOP-18755 > URL: https://issues.apache.org/jira/browse/HADOOP-18755 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 3.3.9 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.9 > > > The new methods of HADOOP-18724 break builds of hbase filesystem because > HBASE-26483 added an implementation of the interface, which, > because it lacks the complete methods, doesn't compile any more. > It also marked the (deprecated) methods as final in the base implementation, > which is also a source of pain. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: [DISCUSS] new repo/sub-project, fs-api-shim
That sounds good On Thu, 1 Jun 2023 at 18:09, Steve Loughran wrote: > hadoop-api-shim ? > > On Thu, 1 Jun 2023 at 04:07, Ayush Saxena wrote: > > > +1, for the new repo. > > > > The name sounds fine, but good if we have scope of having “hadoop-” > > prefix, we have that for almost all of the subprojects/modules > > > > Can hadoop-shims or hadoop-shims-api or something on similar lines work? > > > > -Ayush > > > > > On 01-Jun-2023, at 1:18 AM, Steve Loughran > > > wrote: > > > > > > I want to create a new repository to put a shim library to allow > > previous > > > > > releases to access the more recent hadoop filesystem APIs -currently > the > > > open source implementations of parquet, ORC can't use vectored io, in > > > particular, even though we can in Cloudera. Providing a shim opens them > > up > > > to all *and* gets the APIs more broadly stressed/tested. > > > This needs to be in its own repository, not just for rapid initial > > release, > > > but because it is designed to be built as old a version of hadoop we > can > > > reasonably support, which IMO means hadoop 3.1.0+. I know parquet still > > > wants to build against 2.8.x, but to claim support for hadoop 2 means > > > "build and test on java7", which is unrealistic in 2023. > > > > > > Initial WiP implementation, which works with 3.1.0 and tests against > > others > > > https://github.com/steveloughran/fs-api-shim > > > the complexity is about testing this -I have contract tests which then > > need > > > to be executed on every supported hadoop release, which will need a > > > separate module for each one. > > > > > > I can create the repo easily enough, just would like approval. And is > the > > > name OK? > > > > > > steve > > >
[jira] [Created] (HADOOP-18758) build/release a shaded version of the AWS SDK
Steve Loughran created HADOOP-18758: --- Summary: build/release a shaded version of the AWS SDK Key: HADOOP-18758 URL: https://issues.apache.org/jira/browse/HADOOP-18758 Project: Hadoop Common Issue Type: Sub-task Components: build, fs/s3 Affects Versions: 3.4.0 Reporter: Steve Loughran we need a shaded version of the AWS SDK to stay in control of our dependencies. Either we somehow get the AWS dev team to do it, or we do it ourselves in the hadoop-thirdparty repository. Doing it ourselves would actually give us a bit more control as we have the possibility of making a lighter weight release. It would reduce the control of people downstream though, as they wouldn't be able to "just" drop in a new JAR -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: [DISCUSS] new repo/sub-project, fs-api-shim
hadoop-api-shim ? On Thu, 1 Jun 2023 at 04:07, Ayush Saxena wrote: > +1, for the new repo. > > The name sounds fine, but good if we have scope of having “hadoop-” > prefix, we have that for almost all of the subprojects/modules > > Can hadoop-shims or hadoop-shims-api or something on similar lines work? > > -Ayush > > > On 01-Jun-2023, at 1:18 AM, Steve Loughran > wrote: > > > > I want to create a new repository to put a shim library to allow > previous > > > releases to access the more recent hadoop filesystem APIs -currently the > > open source implementations of parquet, ORC can't use vectored io, in > > particular, even though we can in Cloudera. Providing a shim opens them > up > > to all *and* gets the APIs more broadly stressed/tested. > > This needs to be in its own repository, not just for rapid initial > release, > > but because it is designed to be built as old a version of hadoop we can > > reasonably support, which IMO means hadoop 3.1.0+. I know parquet still > > wants to build against 2.8.x, but to claim support for hadoop 2 means > > "build and test on java7", which is unrealistic in 2023. > > > > Initial WiP implementation, which works with 3.1.0 and tests against > others > > https://github.com/steveloughran/fs-api-shim > > the complexity is about testing this -I have contract tests which then > need > > to be executed on every supported hadoop release, which will need a > > separate module for each one. > > > > I can create the repo easily enough, just would like approval. And is the > > name OK? > > > > steve >
[jira] [Created] (HADOOP-18757) S3A Committer finalyzing the commits in a single thread
Moditha Hewasinghage created HADOOP-18757:
-
Summary: S3A Committer finalyzing the commits in a single thread
Key: HADOOP-18757
URL: https://issues.apache.org/jira/browse/HADOOP-18757
Project: Hadoop Common
Issue Type: Bug
Components: fs/s3
Affects Versions: 3.3.5
Reporter: Moditha Hewasinghage
S3A Committer is being bottle-necked on the driver when finalizing the commits.
It seems like only a single thread is being used to finalize the commit. In the
experiment we are saving 36,000 files ending committing for almost 2 hours each
file taking 0.1 - 0.5 seconds while all the executors stay idle while the
driver commits. I have attached the driver log snippets to support this theory
in comparison to spark 3.4.0.
The most likely reason is the usage of
[ThreadPoolExecutor]([https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ThreadPoolExecutor.html])
in
[https://github.com/apache/hadoop/blob/706d88266abcee09ed78fbaa0ad5f74d818ab0e9/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/impl/CommitContext.java#L239-L244]
where an LinkedBlockingQueue is used with a corePoolSize of 0 which ends up
with a single thread and no new threads created.
>From the documentation
{code:java}
Unbounded queues. Using an unbounded queue (for example a LinkedBlockingQueue
without a predefined capacity) will cause new tasks to wait in the queue when
all corePoolSize threads are busy. Thus, no more than corePoolSize threads will
ever be created. (And the value of the maximumPoolSize therefore doesn't have
any effect.) This may be appropriate when each task is completely independent
of others, so tasks cannot affect each others execution; for example, in a web
page server. While this style of queuing can be useful in smoothing out
transient bursts of requests, it admits the possibility of unbounded work queue
growth when commands continue to arrive on average faster than they can be
processed.{code}
Magic Committer spark 3.5.0-SNAPSHOT
{code:java}
2023-05-26 15:35:04,852 DEBUG impl.CommitContext: creating thread pool of size
32
2023-05-26 15:35:04,922 INFO yarn.YarnAllocator: Driver requested a total
number of 0 executor(s) for resource profile id: 0.
2023-05-26 15:35:07,910 INFO commit.AbstractS3ACommitter: Starting: committing
the output of 36000 task(s)
2023-05-26 15:35:07,914 INFO commit.AbstractS3ACommitter: Starting: Loading and
committing files in pendingset
s3:date=20230520/__magic/job-b9167b1a-35a5-448
e-8022-31e0f7457354/00/task_202305261454174742717183892533843_0031_m_00.pendingset
2023-05-26 15:35:07,941 DEBUG files.PersistentCommitData: Reading commit data
from file
s3:date=20230520/__magic/job-b9167b1a-35a5-448e-8022-31e0f7457354/00/
task_202305261454174742717183892533843_0031_m_00.pendingset
2023-05-26 15:35:08,036 DEBUG impl.CommitContext: creating thread pool of size
32
2023-05-26 15:35:08,037 DEBUG impl.CommitOperations: Committing single commit
SinglePendingCommit
{version=2,
uri='s3:date=20230520/__magic/job-b9167b1a-35a5-
448e-8022-31e0f7457354/00/tasks/attempt_202305261454174742717183892533843_0031_m_00_1920855/__base/rawRequestType=imp_dsp/part-0-b196e04e-c87b-43e1-97e0-7ad34ef0bedf.c000.zstd.parquet',
destination='/date=20230520/rawRequestType=imp_dsp/part-0-b196e04e-c87b-43e1-97e0-7ad34ef0bedf.c000.zstd.parquet',
uploadId='SCp78J9aYOmnPwrtfd5a.Q6B9Zu6olQw3eZcpyt.W.BCu6.M6fz54nlTe2ZYUDicSskFgPocsjlVXAbfiPn3Xu
26MzXNdWcg5j_kBGg9iWpeoWh4K21gt7bbKetML95MXAck15yP.VGFeOleoAspUg--',
created=1685113555232, saved=1685113555232, size=110636173, date='Fri May 26
15:05:55 UTC 2023', jobId='b9167b1a-35a5-448e-8022-31e0f7457354',
taskId='attemp t_202305261454174742717183892533843_0031_m_00_1920855',
notes='',
etags=[64b9d14a89890cf7f931ff05074f6b6a,7a8c22d95d85736806205ad0f84094a5,82f55a1b0595b38420c6b55cb2763a46,212a3105afcf4a9045730f885a622e0a,b563ec91932a66498
e8e9d717b625961,0425b7ae17a8a2af80734f0adf4debe9,496e9fbc08a583657fd29779dcd848d6,72bff33d090699baacb6a6171760a675,b2c6b9b76256416d304392bfbabca382,1a8f63d960cf50589b83b407ddfc77c7,d79cf49f16cd61fb00faf67034268529,e3b7ea2bd0ff
f2e20a89eccd0ebbeb4a,8edc254a94902fee2884350b4a89f223,f0858853a4542b562c6973e28ee40c19]}
2023-05-26 15:35:08,037 INFO impl.CommitOperations: Starting: Committing file
/date=20230520/rawRequestType=imp_dsp/part-0-b196e04e-c87b-43e1-97e0-7ad34ef0bedf.c000.zstd.parquet
s
ize 110636173
2023-05-26 15:35:08,198 DEBUG impl.CommitOperations: Successful commit of file
length 110636173
2023-05-26 15:35:08,199 INFO impl.CommitOperations: Committing file
/date=20230520/rawRequestType=imp_dsp/part-0-b196e04e-c87b-43e1-97e0-7ad34ef0bedf.c000.zstd.parquet
size 110636
173: duration 0:00.162s
2023-05-26 15:35:08,208 INFO commit.AbstractS3ACommitter: Loading and
committing files in pendingset
