zhangyue19921010 opened a new pull request #4964: URL: https://github.com/apache/hudi/pull/4964
https://issues.apache.org/jira/browse/HUDI-2747 ## What is the purpose of the pull request Fix hudi cli metadata commands. Currently when running hudi cli metadata commands locally, it complains about spark master, e.g., yarn master not found. The commands need to take in spark related configs so that they can run locally and in cluster. ## Brief change log Let's take `metadata list-partitions` as an example, Before this patch ``` 67411 [Spring Shell] INFO org.apache.spark.SparkContext - Running Spark version 2.4.4 67442 [Spring Shell] INFO org.apache.spark.SparkContext - Submitted application: HoodieCLI 67515 [Spring Shell] INFO org.apache.spark.SecurityManager - Changing view acls to: yuezhang 67515 [Spring Shell] INFO org.apache.spark.SecurityManager - Changing modify acls to: yuezhang 67516 [Spring Shell] INFO org.apache.spark.SecurityManager - Changing view acls groups to: 67516 [Spring Shell] INFO org.apache.spark.SecurityManager - Changing modify acls groups to: 67517 [Spring Shell] INFO org.apache.spark.SecurityManager - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yuezhang); groups with view permissions: Set(); users with modify permissions: Set(yuezhang); groups with modify permissions: Set() 67571 [Spring Shell] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.output.compression.codec is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.codec 67571 [Spring Shell] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress 67571 [Spring Shell] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.output.compression.type is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.type 67823 [Spring Shell] INFO org.apache.spark.util.Utils - Successfully started service 'sparkDriver' on port 56009. 67851 [Spring Shell] INFO org.apache.spark.SparkEnv - Registering MapOutputTracker 67877 [Spring Shell] INFO org.apache.spark.SparkEnv - Registering BlockManagerMaster 67881 [Spring Shell] INFO org.apache.spark.storage.BlockManagerMasterEndpoint - Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information 67881 [Spring Shell] INFO org.apache.spark.storage.BlockManagerMasterEndpoint - BlockManagerMasterEndpoint up 67896 [Spring Shell] INFO org.apache.spark.storage.DiskBlockManager - Created local directory at /private/var/folders/61/77xdhf3x0x9g3t_vdd1c9_nwr4wznp/T/blockmgr-956db4c2-91a4-4287-913c-69feff340328 67917 [Spring Shell] INFO org.apache.spark.storage.memory.MemoryStore - MemoryStore started with capacity 2004.6 MB 67970 [Spring Shell] INFO org.apache.spark.SparkEnv - Registering OutputCommitCoordinator 68082 [Spring Shell] INFO org.spark_project.jetty.util.log - Logging initialized @68799ms 68158 [Spring Shell] INFO org.spark_project.jetty.server.Server - jetty-9.3.z-SNAPSHOT, build timestamp: 2019-02-16T00:53:49+08:00, git hash: eb70b240169fcf1abbd86af36482d1c49826fa0b 68175 [Spring Shell] INFO org.spark_project.jetty.server.Server - Started @68893ms 68200 [Spring Shell] INFO org.spark_project.jetty.server.AbstractConnector - Started ServerConnector@6ce10da9{HTTP/1.1,[http/1.1]}{0.0.0.0:4040} 68200 [Spring Shell] INFO org.apache.spark.util.Utils - Successfully started service 'SparkUI' on port 4040. 68226 [Spring Shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@701a506a{/jobs,null,AVAILABLE,@Spark} 68226 [Spring Shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@43834940{/jobs/json,null,AVAILABLE,@Spark} 68227 [Spring Shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@7bab57ed{/jobs/job,null,AVAILABLE,@Spark} 68227 [Spring Shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@1b91c8aa{/jobs/job/json,null,AVAILABLE,@Spark} 68228 [Spring Shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@42814bd1{/stages,null,AVAILABLE,@Spark} 68228 [Spring Shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@63347853{/stages/json,null,AVAILABLE,@Spark} 68228 [Spring Shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@58b14e8c{/stages/stage,null,AVAILABLE,@Spark} 68229 [Spring Shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@45d10a0b{/stages/stage/json,null,AVAILABLE,@Spark} 68230 [Spring Shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@1746e913{/stages/pool,null,AVAILABLE,@Spark} 68230 [Spring Shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@5ab68f0a{/stages/pool/json,null,AVAILABLE,@Spark} 68230 [Spring Shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@678c3f74{/storage,null,AVAILABLE,@Spark} 68231 [Spring Shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@2ac0e0dc{/storage/json,null,AVAILABLE,@Spark} 68231 [Spring Shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@aee8ae6{/storage/rdd,null,AVAILABLE,@Spark} 68231 [Spring Shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@644e8a6e{/storage/rdd/json,null,AVAILABLE,@Spark} 68232 [Spring Shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@623f7e0c{/environment,null,AVAILABLE,@Spark} 68232 [Spring Shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@4f54a497{/environment/json,null,AVAILABLE,@Spark} 68232 [Spring Shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@3113ae21{/executors,null,AVAILABLE,@Spark} 68233 [Spring Shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@219e5f97{/executors/json,null,AVAILABLE,@Spark} 68233 [Spring Shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@38488ab6{/executors/threadDump,null,AVAILABLE,@Spark} 68233 [Spring Shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@52d2e134{/executors/threadDump/json,null,AVAILABLE,@Spark} 68243 [Spring Shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@7e9c3fe9{/static,null,AVAILABLE,@Spark} 68244 [Spring Shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@5d2fa317{/,null,AVAILABLE,@Spark} 68245 [Spring Shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@1c5344fd{/api,null,AVAILABLE,@Spark} 68246 [Spring Shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@447ff00f{/jobs/job/kill,null,AVAILABLE,@Spark} 68247 [Spring Shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@55a01d8e{/stages/stage/kill,null,AVAILABLE,@Spark} 68250 [Spring Shell] INFO org.apache.spark.ui.SparkUI - Bound SparkUI to 0.0.0.0, and started at http://172.24.10.26:4040 68283 [Spring Shell] ERROR org.apache.spark.SparkContext - Error initializing SparkContext. org.apache.spark.SparkException: Could not parse Master URL: 'yarn' at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2784) at org.apache.spark.SparkContext.<init>(SparkContext.scala:493) at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58) at org.apache.hudi.cli.utils.SparkUtil.initJavaSparkConf(SparkUtil.java:117) at org.apache.hudi.cli.commands.MetadataCommand.initJavaSparkContext(MetadataCommand.java:367) at org.apache.hudi.cli.commands.MetadataCommand.listPartitions(MetadataCommand.java:201) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:216) at org.springframework.shell.core.SimpleExecutionStrategy.invoke(SimpleExecutionStrategy.java:68) at org.springframework.shell.core.SimpleExecutionStrategy.execute(SimpleExecutionStrategy.java:59) at org.springframework.shell.core.AbstractShell.executeCommand(AbstractShell.java:134) at org.springframework.shell.core.JLineShell.promptLoop(JLineShell.java:533) at org.springframework.shell.core.JLineShell.run(JLineShell.java:179) at java.lang.Thread.run(Thread.java:748) 68302 [Spring Shell] INFO org.spark_project.jetty.server.AbstractConnector - Stopped Spark@6ce10da9{HTTP/1.1,[http/1.1]}{0.0.0.0:4040} 68304 [Spring Shell] INFO org.apache.spark.ui.SparkUI - Stopped Spark web UI at http://172.24.10.26:4040 68318 [dispatcher-event-loop-8] INFO org.apache.spark.MapOutputTrackerMasterEndpoint - MapOutputTrackerMasterEndpoint stopped! 68331 [Spring Shell] INFO org.apache.spark.storage.memory.MemoryStore - MemoryStore cleared 68332 [Spring Shell] INFO org.apache.spark.storage.BlockManager - BlockManager stopped 68346 [Spring Shell] INFO org.apache.spark.storage.BlockManagerMaster - BlockManagerMaster stopped 68347 [Spring Shell] WARN org.apache.spark.metrics.MetricsSystem - Stopping a MetricsSystem that is not running 68355 [dispatcher-event-loop-1] INFO org.apache.spark.scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint - OutputCommitCoordinator stopped! 68364 [Spring Shell] INFO org.apache.spark.SparkContext - Successfully stopped SparkContext 68364 [Spring Shell] ERROR org.springframework.shell.core.SimpleExecutionStrategy - Command failed java.lang.reflect.UndeclaredThrowableException 68364 [Spring Shell] WARN org.springframework.shell.core.JLineShellComponent.exceptions - java.lang.reflect.UndeclaredThrowableException at org.springframework.util.ReflectionUtils.rethrowRuntimeException(ReflectionUtils.java:315) at org.springframework.util.ReflectionUtils.handleInvocationTargetException(ReflectionUtils.java:295) at org.springframework.util.ReflectionUtils.handleReflectionException(ReflectionUtils.java:279) at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:219) at org.springframework.shell.core.SimpleExecutionStrategy.invoke(SimpleExecutionStrategy.java:68) at org.springframework.shell.core.SimpleExecutionStrategy.execute(SimpleExecutionStrategy.java:59) at org.springframework.shell.core.AbstractShell.executeCommand(AbstractShell.java:134) at org.springframework.shell.core.JLineShell.promptLoop(JLineShell.java:533) at org.springframework.shell.core.JLineShell.run(JLineShell.java:179) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.spark.SparkException: Could not parse Master URL: 'yarn' at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2784) at org.apache.spark.SparkContext.<init>(SparkContext.scala:493) at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58) at org.apache.hudi.cli.utils.SparkUtil.initJavaSparkConf(SparkUtil.java:117) at org.apache.hudi.cli.commands.MetadataCommand.initJavaSparkContext(MetadataCommand.java:367) at org.apache.hudi.cli.commands.MetadataCommand.listPartitions(MetadataCommand.java:201) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:216) ``` After this patch, we can use `metadata list-partitions --sparkMaster local[*]` ``` 97028 [Spring Shell] WARN org.apache.spark.SparkContext - Another SparkContext is being constructed (or threw an exception in its constructor). This may indicate an error, since only one SparkContext may be running in this JVM (see SPARK-2243). The other SparkContext was created at: org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58) org.apache.hudi.cli.utils.SparkUtil.initJavaSparkConf(SparkUtil.java:117) org.apache.hudi.cli.commands.MetadataCommand.initJavaSparkContext(MetadataCommand.java:367) org.apache.hudi.cli.commands.MetadataCommand.listPartitions(MetadataCommand.java:201) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:216) org.springframework.shell.core.SimpleExecutionStrategy.invoke(SimpleExecutionStrategy.java:68) org.springframework.shell.core.SimpleExecutionStrategy.execute(SimpleExecutionStrategy.java:59) org.springframework.shell.core.AbstractShell.executeCommand(AbstractShell.java:134) org.springframework.shell.core.JLineShell.promptLoop(JLineShell.java:533) org.springframework.shell.core.JLineShell.run(JLineShell.java:179) java.lang.Thread.run(Thread.java:748) 97028 [Spring Shell] INFO org.apache.spark.SparkContext - Running Spark version 2.4.4 97029 [Spring Shell] INFO org.apache.spark.SparkContext - Submitted application: HoodieCLI 97030 [Spring Shell] INFO org.apache.spark.SecurityManager - Changing view acls to: yuezhang 97030 [Spring Shell] INFO org.apache.spark.SecurityManager - Changing modify acls to: yuezhang 97030 [Spring Shell] INFO org.apache.spark.SecurityManager - Changing view acls groups to: 97030 [Spring Shell] INFO org.apache.spark.SecurityManager - Changing modify acls groups to: 97030 [Spring Shell] INFO org.apache.spark.SecurityManager - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yuezhang); groups with view permissions: Set(); users with modify permissions: Set(yuezhang); groups with modify permissions: Set() 97062 [Spring Shell] INFO org.apache.spark.util.Utils - Successfully started service 'sparkDriver' on port 56031. 97064 [Spring Shell] INFO org.apache.spark.SparkEnv - Registering MapOutputTracker 97065 [Spring Shell] INFO org.apache.spark.SparkEnv - Registering BlockManagerMaster 97065 [Spring Shell] INFO org.apache.spark.storage.BlockManagerMasterEndpoint - Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information 97065 [Spring Shell] INFO org.apache.spark.storage.BlockManagerMasterEndpoint - BlockManagerMasterEndpoint up 97066 [Spring Shell] INFO org.apache.spark.storage.DiskBlockManager - Created local directory at /private/var/folders/61/77xdhf3x0x9g3t_vdd1c9_nwr4wznp/T/blockmgr-3f3b2949-3a00-49d7-a600-f0f585e9e7fc 97066 [Spring Shell] INFO org.apache.spark.storage.memory.MemoryStore - MemoryStore started with capacity 2004.6 MB 97067 [Spring Shell] INFO org.apache.spark.SparkEnv - Registering OutputCommitCoordinator 97074 [Spring Shell] INFO org.spark_project.jetty.server.Server - jetty-9.3.z-SNAPSHOT, build timestamp: 2019-02-16T00:53:49+08:00, git hash: eb70b240169fcf1abbd86af36482d1c49826fa0b 97075 [Spring Shell] INFO org.spark_project.jetty.server.Server - Started @97793ms 97075 [Spring Shell] INFO org.spark_project.jetty.server.AbstractConnector - Started ServerConnector@53404266{HTTP/1.1,[http/1.1]}{0.0.0.0:4040} 97075 [Spring Shell] INFO org.apache.spark.util.Utils - Successfully started service 'SparkUI' on port 4040. 97076 [Spring Shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@25c98e66{/jobs,null,AVAILABLE,@Spark} 97076 [Spring Shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@58ad3d9b{/jobs/json,null,AVAILABLE,@Spark} 97076 [Spring Shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@1ba0d88{/jobs/job,null,AVAILABLE,@Spark} 97076 [Spring Shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@50383cab{/jobs/job/json,null,AVAILABLE,@Spark} 97077 [Spring Shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@396b3274{/stages,null,AVAILABLE,@Spark} 97077 [Spring Shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@81b8127{/stages/json,null,AVAILABLE,@Spark} 97077 [Spring Shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@599c0be1{/stages/stage,null,AVAILABLE,@Spark} 97077 [Spring Shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@48d0a91d{/stages/stage/json,null,AVAILABLE,@Spark} 97078 [Spring Shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@28c8cae1{/stages/pool,null,AVAILABLE,@Spark} 97078 [Spring Shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@7c8e7466{/stages/pool/json,null,AVAILABLE,@Spark} 97078 [Spring Shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@4574ab15{/storage,null,AVAILABLE,@Spark} 97078 [Spring Shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@8216ae7{/storage/json,null,AVAILABLE,@Spark} 97079 [Spring Shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@665036c0{/storage/rdd,null,AVAILABLE,@Spark} 97079 [Spring Shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@69f24328{/storage/rdd/json,null,AVAILABLE,@Spark} 97079 [Spring Shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@9be6100{/environment,null,AVAILABLE,@Spark} 97079 [Spring Shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@acb4482{/environment/json,null,AVAILABLE,@Spark} 97080 [Spring Shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@6d4a952d{/executors,null,AVAILABLE,@Spark} 97080 [Spring Shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@165a70af{/executors/json,null,AVAILABLE,@Spark} 97080 [Spring Shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@6c16b505{/executors/threadDump,null,AVAILABLE,@Spark} 97080 [Spring Shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@3ca90351{/executors/threadDump/json,null,AVAILABLE,@Spark} 97081 [Spring Shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@5731a14{/static,null,AVAILABLE,@Spark} 97081 [Spring Shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@40e9cd1d{/,null,AVAILABLE,@Spark} 97081 [Spring Shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@198ee698{/api,null,AVAILABLE,@Spark} 97082 [Spring Shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@7077c974{/jobs/job/kill,null,AVAILABLE,@Spark} 97082 [Spring Shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@7ef86342{/stages/stage/kill,null,AVAILABLE,@Spark} 97082 [Spring Shell] INFO org.apache.spark.ui.SparkUI - Bound SparkUI to 0.0.0.0, and started at http://172.24.10.26:4040 97179 [Spring Shell] INFO org.apache.spark.executor.Executor - Starting executor ID driver on host localhost 97254 [Spring Shell] INFO org.apache.spark.util.Utils - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 56032. 97254 [Spring Shell] INFO org.apache.spark.network.netty.NettyBlockTransferService - Server created on 172.24.10.26:56032 97256 [Spring Shell] INFO org.apache.spark.storage.BlockManager - Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy 97284 [Spring Shell] INFO org.apache.spark.storage.BlockManagerMaster - Registering BlockManager BlockManagerId(driver, 172.24.10.26, 56032, None) 97285 [dispatcher-event-loop-10] INFO org.apache.spark.storage.BlockManagerMasterEndpoint - Registering block manager 172.24.10.26:56032 with 2004.6 MB RAM, BlockManagerId(driver, 172.24.10.26, 56032, None) 97288 [Spring Shell] INFO org.apache.spark.storage.BlockManagerMaster - Registered BlockManager BlockManagerId(driver, 172.24.10.26, 56032, None) 97289 [Spring Shell] INFO org.apache.spark.storage.BlockManager - Initialized BlockManager: BlockManagerId(driver, 172.24.10.26, 56032, None) 97416 [Spring Shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@71d6123f{/metrics/json,null,AVAILABLE,@Spark} 97536 [Spring Shell] INFO org.apache.hudi.common.table.HoodieTableMetaClient - Loading HoodieTableMetaClient from /Users/yuezhang/tmp/hudiAfTable/forecast_agg 97537 [Spring Shell] INFO org.apache.hudi.common.table.HoodieTableConfig - Loading table properties from /Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/hoodie.properties 97538 [Spring Shell] INFO org.apache.hudi.common.table.HoodieTableMetaClient - Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from /Users/yuezhang/tmp/hudiAfTable/forecast_agg 97538 [Spring Shell] INFO org.apache.hudi.common.table.HoodieTableMetaClient - Loading HoodieTableMetaClient from /Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata 97539 [Spring Shell] INFO org.apache.hudi.common.table.HoodieTableConfig - Loading table properties from /Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/.hoodie/hoodie.properties 97540 [Spring Shell] INFO org.apache.hudi.common.table.HoodieTableMetaClient - Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from /Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata 97543 [Spring Shell] INFO org.apache.hudi.metadata.HoodieTableMetadataUtil - Loading latest merged file slices for metadata table partition files 97566 [Spring Shell] INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline - Loaded instants upto : Option{val=[20220214171053821__deltacommit__COMPLETED]} 97572 [Spring Shell] INFO org.apache.hudi.common.table.view.AbstractTableFileSystemView - Took 3 ms to read 0 instants, 0 replaced file groups 97587 [Spring Shell] INFO org.apache.hudi.common.util.ClusteringUtils - Found 0 files in pending clustering operations 97588 [Spring Shell] INFO org.apache.hudi.common.table.view.AbstractTableFileSystemView - Building file system view for partition (files) 97600 [Spring Shell] INFO org.apache.hudi.common.table.view.AbstractTableFileSystemView - addFilesToView: NumFiles=12, NumFileGroups=1, FileGroupsCreationTime=8, StoreTimeTaken=1 97611 [Spring Shell] INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline - Loaded instants upto : Option{val=[20220214171053821__commit__COMPLETED]} 97743 [Spring Shell] INFO org.apache.hudi.common.table.HoodieTableMetaClient - Loading HoodieTableMetaClient from /Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata 97744 [Spring Shell] INFO org.apache.hudi.common.table.HoodieTableConfig - Loading table properties from /Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/.hoodie/hoodie.properties 97745 [Spring Shell] INFO org.apache.hudi.common.table.HoodieTableMetaClient - Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from /Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata 97760 [Spring Shell] INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline - Loaded instants upto : Option{val=[20220214171053821__deltacommit__COMPLETED]} 97768 [Spring Shell] INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader - Scanning log file HoodieLogFile{pathStr='file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.1_0-0-0', fileLen=-1} 97774 [Spring Shell] INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader - Reading a delete block from file file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.1_0-0-0 97778 [Spring Shell] INFO org.apache.hudi.common.table.log.HoodieLogFormatReader - Moving to the next reader for logfile HoodieLogFile{pathStr='file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.1_0-14-29', fileLen=-1} 97778 [Spring Shell] INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader - Scanning log file HoodieLogFile{pathStr='file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.1_0-14-29', fileLen=-1} 97783 [Spring Shell] INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader - Reading a data block from file file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.1_0-14-29 at instant 00000000000000 97784 [Spring Shell] INFO org.apache.hudi.common.table.log.HoodieLogFormatReader - Moving to the next reader for logfile HoodieLogFile{pathStr='file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.2_0-27-40', fileLen=-1} 97785 [Spring Shell] INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader - Scanning log file HoodieLogFile{pathStr='file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.2_0-27-40', fileLen=-1} 97785 [Spring Shell] INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader - Reading a data block from file file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.2_0-27-40 at instant 20220214165727173 97785 [Spring Shell] INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader - Number of remaining logblocks to merge 2 97842 [Spring Shell] INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader - Number of remaining logblocks to merge 1 97861 [Spring Shell] INFO org.apache.hadoop.hbase.io.hfile.CacheConfig - Allocating LruBlockCache size=1.42 GB, blockSize=64 KB 97875 [Spring Shell] INFO org.apache.hadoop.hbase.io.hfile.CacheConfig - Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=1567280, freeSize=1525578832, maxSize=1527146112, heapSize=1567280, minSize=1450788736, minFactor=0.95, multiSize=725394368, multiFactor=0.5, singleSize=362697184, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 98021 [Spring Shell] INFO org.apache.hadoop.io.compress.CodecPool - Got brand-new decompressor [.gz] 98029 [Spring Shell] INFO org.apache.hadoop.io.compress.CodecPool - Got brand-new decompressor [.gz] 98030 [Spring Shell] INFO org.apache.hadoop.io.compress.CodecPool - Got brand-new decompressor [.gz] 98096 [Spring Shell] INFO org.apache.hadoop.io.compress.CodecPool - Got brand-new decompressor [.gz] 98123 [Spring Shell] INFO org.apache.hudi.common.util.collection.ExternalSpillableMap - Estimated Payload size => 328 98143 [Spring Shell] INFO org.apache.hudi.common.table.log.HoodieLogFormatReader - Moving to the next reader for logfile HoodieLogFile{pathStr='file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.3_0-11-20', fileLen=-1} 98143 [Spring Shell] INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader - Scanning log file HoodieLogFile{pathStr='file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.3_0-11-20', fileLen=-1} 98144 [Spring Shell] INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader - Reading a data block from file file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.3_0-11-20 at instant 20220214170139680 98144 [Spring Shell] INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader - Number of remaining logblocks to merge 1 98155 [Spring Shell] INFO org.apache.hadoop.hbase.io.hfile.CacheConfig - Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=1567280, freeSize=1525578832, maxSize=1527146112, heapSize=1567280, minSize=1450788736, minFactor=0.95, multiSize=725394368, multiFactor=0.5, singleSize=362697184, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 98155 [Spring Shell] INFO org.apache.hadoop.io.compress.CodecPool - Got brand-new decompressor [.gz] 98155 [Spring Shell] INFO org.apache.hadoop.io.compress.CodecPool - Got brand-new decompressor [.gz] 98156 [Spring Shell] INFO org.apache.hadoop.io.compress.CodecPool - Got brand-new decompressor [.gz] 98156 [Spring Shell] INFO org.apache.hadoop.io.compress.CodecPool - Got brand-new decompressor [.gz] 98159 [Spring Shell] INFO org.apache.hudi.common.table.log.HoodieLogFormatReader - Moving to the next reader for logfile HoodieLogFile{pathStr='file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.4_0-11-20', fileLen=-1} 98159 [Spring Shell] INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader - Scanning log file HoodieLogFile{pathStr='file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.4_0-11-20', fileLen=-1} 98160 [Spring Shell] INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader - Reading a data block from file file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.4_0-11-20 at instant 20220214170233074 98160 [Spring Shell] INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader - Number of remaining logblocks to merge 1 98170 [Spring Shell] INFO org.apache.hadoop.hbase.io.hfile.CacheConfig - Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=1567280, freeSize=1525578832, maxSize=1527146112, heapSize=1567280, minSize=1450788736, minFactor=0.95, multiSize=725394368, multiFactor=0.5, singleSize=362697184, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 98171 [Spring Shell] INFO org.apache.hadoop.io.compress.CodecPool - Got brand-new decompressor [.gz] 98171 [Spring Shell] INFO org.apache.hadoop.io.compress.CodecPool - Got brand-new decompressor [.gz] 98171 [Spring Shell] INFO org.apache.hadoop.io.compress.CodecPool - Got brand-new decompressor [.gz] 98171 [Spring Shell] INFO org.apache.hadoop.io.compress.CodecPool - Got brand-new decompressor [.gz] 98173 [Spring Shell] INFO org.apache.hudi.common.table.log.HoodieLogFormatReader - Moving to the next reader for logfile HoodieLogFile{pathStr='file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.5_0-11-20', fileLen=-1} 98174 [Spring Shell] INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader - Scanning log file HoodieLogFile{pathStr='file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.5_0-11-20', fileLen=-1} 98174 [Spring Shell] INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader - Reading a data block from file file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.5_0-11-20 at instant 20220214170320009 98174 [Spring Shell] INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader - Number of remaining logblocks to merge 1 98185 [Spring Shell] INFO org.apache.hadoop.hbase.io.hfile.CacheConfig - Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=1567280, freeSize=1525578832, maxSize=1527146112, heapSize=1567280, minSize=1450788736, minFactor=0.95, multiSize=725394368, multiFactor=0.5, singleSize=362697184, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 98185 [Spring Shell] INFO org.apache.hadoop.io.compress.CodecPool - Got brand-new decompressor [.gz] 98185 [Spring Shell] INFO org.apache.hadoop.io.compress.CodecPool - Got brand-new decompressor [.gz] 98185 [Spring Shell] INFO org.apache.hadoop.io.compress.CodecPool - Got brand-new decompressor [.gz] 98186 [Spring Shell] INFO org.apache.hadoop.io.compress.CodecPool - Got brand-new decompressor [.gz] 98188 [Spring Shell] INFO org.apache.hudi.common.table.log.HoodieLogFormatReader - Moving to the next reader for logfile HoodieLogFile{pathStr='file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.6_0-11-20', fileLen=-1} 98189 [Spring Shell] INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader - Scanning log file HoodieLogFile{pathStr='file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.6_0-11-20', fileLen=-1} 98189 [Spring Shell] INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader - Reading a data block from file file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.6_0-11-20 at instant 20220214170407096 98189 [Spring Shell] INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader - Number of remaining logblocks to merge 1 98200 [Spring Shell] INFO org.apache.hadoop.hbase.io.hfile.CacheConfig - Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=1567280, freeSize=1525578832, maxSize=1527146112, heapSize=1567280, minSize=1450788736, minFactor=0.95, multiSize=725394368, multiFactor=0.5, singleSize=362697184, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 98200 [Spring Shell] INFO org.apache.hadoop.io.compress.CodecPool - Got brand-new decompressor [.gz] 98200 [Spring Shell] INFO org.apache.hadoop.io.compress.CodecPool - Got brand-new decompressor [.gz] 98200 [Spring Shell] INFO org.apache.hadoop.io.compress.CodecPool - Got brand-new decompressor [.gz] 98200 [Spring Shell] INFO org.apache.hadoop.io.compress.CodecPool - Got brand-new decompressor [.gz] 98203 [Spring Shell] INFO org.apache.hudi.common.table.log.HoodieLogFormatReader - Moving to the next reader for logfile HoodieLogFile{pathStr='file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.7_0-11-20', fileLen=-1} 98204 [Spring Shell] INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader - Scanning log file HoodieLogFile{pathStr='file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.7_0-11-20', fileLen=-1} 98204 [Spring Shell] INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader - Reading a data block from file file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.7_0-11-20 at instant 20220214170451797 98204 [Spring Shell] INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader - Number of remaining logblocks to merge 1 98218 [Spring Shell] INFO org.apache.hadoop.hbase.io.hfile.CacheConfig - Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=1567280, freeSize=1525578832, maxSize=1527146112, heapSize=1567280, minSize=1450788736, minFactor=0.95, multiSize=725394368, multiFactor=0.5, singleSize=362697184, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 98218 [Spring Shell] INFO org.apache.hadoop.io.compress.CodecPool - Got brand-new decompressor [.gz] 98218 [Spring Shell] INFO org.apache.hadoop.io.compress.CodecPool - Got brand-new decompressor [.gz] 98218 [Spring Shell] INFO org.apache.hadoop.io.compress.CodecPool - Got brand-new decompressor [.gz] 98219 [Spring Shell] INFO org.apache.hadoop.io.compress.CodecPool - Got brand-new decompressor [.gz] 98224 [Spring Shell] INFO org.apache.hudi.common.table.log.HoodieLogFormatReader - Moving to the next reader for logfile HoodieLogFile{pathStr='file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.8_0-11-20', fileLen=-1} 98225 [Spring Shell] INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader - Scanning log file HoodieLogFile{pathStr='file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.8_0-11-20', fileLen=-1} 98225 [Spring Shell] INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader - Reading a data block from file file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.8_0-11-20 at instant 20220214170547258 98225 [Spring Shell] INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader - Number of remaining logblocks to merge 1 98238 [Spring Shell] INFO org.apache.hadoop.hbase.io.hfile.CacheConfig - Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=1567280, freeSize=1525578832, maxSize=1527146112, heapSize=1567280, minSize=1450788736, minFactor=0.95, multiSize=725394368, multiFactor=0.5, singleSize=362697184, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 98238 [Spring Shell] INFO org.apache.hadoop.io.compress.CodecPool - Got brand-new decompressor [.gz] 98238 [Spring Shell] INFO org.apache.hadoop.io.compress.CodecPool - Got brand-new decompressor [.gz] 98238 [Spring Shell] INFO org.apache.hadoop.io.compress.CodecPool - Got brand-new decompressor [.gz] 98238 [Spring Shell] INFO org.apache.hadoop.io.compress.CodecPool - Got brand-new decompressor [.gz] 98241 [Spring Shell] INFO org.apache.hudi.common.table.log.HoodieLogFormatReader - Moving to the next reader for logfile HoodieLogFile{pathStr='file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.9_0-11-20', fileLen=-1} 98241 [Spring Shell] INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader - Scanning log file HoodieLogFile{pathStr='file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.9_0-11-20', fileLen=-1} 98241 [Spring Shell] INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader - Reading a data block from file file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.9_0-11-20 at instant 20220214170815557 98241 [Spring Shell] INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader - Number of remaining logblocks to merge 1 98252 [Spring Shell] INFO org.apache.hadoop.hbase.io.hfile.CacheConfig - Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=1567280, freeSize=1525578832, maxSize=1527146112, heapSize=1567280, minSize=1450788736, minFactor=0.95, multiSize=725394368, multiFactor=0.5, singleSize=362697184, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 98252 [Spring Shell] INFO org.apache.hadoop.io.compress.CodecPool - Got brand-new decompressor [.gz] 98253 [Spring Shell] INFO org.apache.hadoop.io.compress.CodecPool - Got brand-new decompressor [.gz] 98253 [Spring Shell] INFO org.apache.hadoop.io.compress.CodecPool - Got brand-new decompressor [.gz] 98253 [Spring Shell] INFO org.apache.hadoop.io.compress.CodecPool - Got brand-new decompressor [.gz] 98255 [Spring Shell] INFO org.apache.hudi.common.table.log.HoodieLogFormatReader - Moving to the next reader for logfile HoodieLogFile{pathStr='file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.10_0-11-20', fileLen=-1} 98256 [Spring Shell] INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader - Scanning log file HoodieLogFile{pathStr='file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.10_0-11-20', fileLen=-1} 98256 [Spring Shell] INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader - Reading a data block from file file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.10_0-11-20 at instant 20220214171053821 98256 [Spring Shell] INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader - Number of remaining logblocks to merge 1 98267 [Spring Shell] INFO org.apache.hadoop.hbase.io.hfile.CacheConfig - Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=1567280, freeSize=1525578832, maxSize=1527146112, heapSize=1567280, minSize=1450788736, minFactor=0.95, multiSize=725394368, multiFactor=0.5, singleSize=362697184, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 98267 [Spring Shell] INFO org.apache.hadoop.io.compress.CodecPool - Got brand-new decompressor [.gz] 98267 [Spring Shell] INFO org.apache.hadoop.io.compress.CodecPool - Got brand-new decompressor [.gz] 98267 [Spring Shell] INFO org.apache.hadoop.io.compress.CodecPool - Got brand-new decompressor [.gz] 98267 [Spring Shell] INFO org.apache.hadoop.io.compress.CodecPool - Got brand-new decompressor [.gz] 98268 [Spring Shell] INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader - Merging the final data blocks 98268 [Spring Shell] INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader - Number of remaining logblocks to merge 1 98279 [Spring Shell] INFO org.apache.hadoop.hbase.io.hfile.CacheConfig - Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=1567280, freeSize=1525578832, maxSize=1527146112, heapSize=1567280, minSize=1450788736, minFactor=0.95, multiSize=725394368, multiFactor=0.5, singleSize=362697184, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 98279 [Spring Shell] INFO org.apache.hadoop.io.compress.CodecPool - Got brand-new decompressor [.gz] 98279 [Spring Shell] INFO org.apache.hadoop.io.compress.CodecPool - Got brand-new decompressor [.gz] 98279 [Spring Shell] INFO org.apache.hadoop.io.compress.CodecPool - Got brand-new decompressor [.gz] 98280 [Spring Shell] INFO org.apache.hadoop.io.compress.CodecPool - Got brand-new decompressor [.gz] 98281 [Spring Shell] INFO org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner - Number of log files scanned => 11 98281 [Spring Shell] INFO org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner - MaxMemoryInBytes allowed for compaction => 1073741824 98281 [Spring Shell] INFO org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner - Number of entries in MemoryBasedMap in ExternalSpillableMap => 2 98281 [Spring Shell] INFO org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner - Total size in bytes of MemoryBasedMap in ExternalSpillableMap => 656 98281 [Spring Shell] INFO org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner - Number of entries in BitCaskDiskMap in ExternalSpillableMap => 0 98281 [Spring Shell] INFO org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner - Size of file spilled to disk => 0 98282 [Spring Shell] INFO org.apache.hudi.metadata.HoodieBackedTableMetadata - Opened 11 metadata log files (dataset instant=20220214171053821, metadata instant=20220214171053821) in 677 ms 98294 [Spring Shell] INFO org.apache.hudi.metadata.BaseTableMetadata - Listed partitions from metadata: #partitions=1 98302 [Spring Shell] INFO org.springframework.shell.core.JLineShellComponent - ╔═══════════╗ ║ partition ║ ╠═══════════╣ ║ 20210623 ║ ╚═══════════╝ ``` ## Verify this pull request *(Please pick either of the following options)* This pull request is a trivial rework / code cleanup without any test coverage. *(or)* This pull request is already covered by existing tests, such as *(please describe tests)*. (or) This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end.* - *Added HoodieClientWriteTest to verify the change.* - *Manually verified the change by running a job locally.* ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
