alberttwong opened a new issue, #11909: URL: https://github.com/apache/hudi/issues/11909
Hudi CLI conf is hard coded to /opt/hudi/packaging/hudi-cli-bundle/conf/hudi-defaults.conf A clear and concise description of the problem. **To Reproduce** Steps to reproduce the behavior: ``` export HOODIE_ENV_fs_DOT_s3a_DOT_access_DOT_key=admin export HOODIE_ENV_fs_DOT_s3a_DOT_secret_DOT_key=password export HOODIE_ENV_fs_DOT_s3a_DOT_endpoint=http://minio:9000 export HOODIE_ENV_fs_DOT_s3a_DOT_aws_DOT_credentials_DOT_provider=org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider export CLIENT_JAR=/opt/hudicli/hadoop-aws-2.10.2.jar:/opt/hudicli/aws-java-sdk-bundle-1.11.271.jar export SPARK_BUNDLE_JAR=/opt/hudicli/hudi-spark3.4-bundle_2.12-0.15.0.jar export CLI_BUNDLE_JAR=/opt/hudicli/hudi-cli-bundle_2.12-0.15.0.jar export HUDI_CONF_DIR=/opt/hudicli/conf/ ``` ``` hudi:stock_ticks_mor->compaction schedule --hoodieConfigs hoodie.compact.inline.max.delta.commits=1 17824 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:02 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 17916 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:02 INFO SparkMain: Invoking SparkMain: COMPACT_SCHEDULE 17933 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:02 INFO SparkContext: Running Spark version 3.4.3 17947 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:02 INFO ResourceUtils: ============================================================== 17947 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:02 INFO ResourceUtils: No custom resources configured for spark.driver. 17947 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:02 INFO ResourceUtils: ============================================================== 17948 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:02 INFO SparkContext: Submitted application: hoodie-cli-COMPACT_SCHEDULE 17960 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:02 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: , offHeap -> name: offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0) 17966 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:02 INFO ResourceProfile: Limiting resource is cpu 17966 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:02 INFO ResourceProfileManager: Added ResourceProfile id: 0 17999 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:02 INFO SecurityManager: Changing view acls to: root 17999 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:02 INFO SecurityManager: Changing modify acls to: root 17999 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:02 INFO SecurityManager: Changing view acls groups to: 17999 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:02 INFO SecurityManager: Changing modify acls groups to: 17999 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:02 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: root; groups with view permissions: EMPTY; users with modify permissions: root; groups with modify permissions: EMPTY 18013 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:02 INFO deprecation: mapred.output.compression.codec is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.codec 18013 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:02 INFO deprecation: mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress 18013 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:02 INFO deprecation: mapred.output.compression.type is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.type 18117 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:02 INFO Utils: Successfully started service 'sparkDriver' on port 35369. 18134 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:02 INFO SparkEnv: Registering MapOutputTracker 18175 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:02 INFO SparkEnv: Registering BlockManagerMaster 18184 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:02 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information 18185 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:02 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up 18187 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:02 INFO SparkEnv: Registering BlockManagerMasterHeartbeat 18198 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:02 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-cf72b92e-9610-4f37-b6ca-8ee223b8c424 18208 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:02 INFO MemoryStore: MemoryStore started with capacity 366.3 MiB 18218 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:02 INFO SparkEnv: Registering OutputCommitCoordinator 18297 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:03 INFO JettyUtils: Start Jetty 0.0.0.0:4040 for SparkUI 18329 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:03 INFO Utils: Successfully started service 'SparkUI' on port 4040. 18349 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:03 INFO SparkContext: Added JAR file:///opt/hudicli/hudi-spark3.4-bundle_2.12-0.15.0.jar at spark://openjdk8:35369/jars/hudi-spark3.4-bundle_2.12-0.15.0.jar with timestamp 1725643082650 18349 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:03 INFO SparkContext: Added JAR file:/opt/hudicli/hudi-cli-bundle_2.12-0.15.0.jar at spark://openjdk8:35369/jars/hudi-cli-bundle_2.12-0.15.0.jar with timestamp 1725643082650 18385 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:03 INFO Executor: Starting executor ID driver on host openjdk8 18388 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:03 INFO Executor: Starting executor with user classpath (userClassPathFirst = false): '' 18394 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:03 INFO Executor: Fetching spark://openjdk8:35369/jars/hudi-spark3.4-bundle_2.12-0.15.0.jar with timestamp 1725643082650 18419 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:03 INFO TransportClientFactory: Successfully created connection to openjdk8/172.18.0.13:35369 after 13 ms (0 ms spent in bootstraps) 18424 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:03 INFO Utils: Fetching spark://openjdk8:35369/jars/hudi-spark3.4-bundle_2.12-0.15.0.jar to /tmp/spark-96bb40be-d352-44ce-b5c3-89a836585bbd/userFiles-1f0c1f46-fb5b-4c20-9f13-61aa8423fe7b/fetchFileTemp8256267113323046445.tmp 18606 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:03 INFO Executor: Adding file:/tmp/spark-96bb40be-d352-44ce-b5c3-89a836585bbd/userFiles-1f0c1f46-fb5b-4c20-9f13-61aa8423fe7b/hudi-spark3.4-bundle_2.12-0.15.0.jar to class loader 18607 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:03 INFO Executor: Fetching spark://openjdk8:35369/jars/hudi-cli-bundle_2.12-0.15.0.jar with timestamp 1725643082650 18607 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:03 INFO Utils: Fetching spark://openjdk8:35369/jars/hudi-cli-bundle_2.12-0.15.0.jar to /tmp/spark-96bb40be-d352-44ce-b5c3-89a836585bbd/userFiles-1f0c1f46-fb5b-4c20-9f13-61aa8423fe7b/fetchFileTemp6763625618385495819.tmp 18677 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:03 INFO Executor: Adding file:/tmp/spark-96bb40be-d352-44ce-b5c3-89a836585bbd/userFiles-1f0c1f46-fb5b-4c20-9f13-61aa8423fe7b/hudi-cli-bundle_2.12-0.15.0.jar to class loader 18680 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:03 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 35467. 18681 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:03 INFO NettyBlockTransferService: Server created on openjdk8:35467 18681 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:03 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy 18684 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:03 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, openjdk8, 35467, None) 18686 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:03 INFO BlockManagerMasterEndpoint: Registering block manager openjdk8:35467 with 366.3 MiB RAM, BlockManagerId(driver, openjdk8, 35467, None) 18688 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:03 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, openjdk8, 35467, None) 18688 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:03 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, openjdk8, 35467, None) 18805 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:03 INFO HadoopFSUtils: Picking up value for hoodie env var : HOODIE_ENV_fs_DOT_s3a_DOT_endpoint 18805 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:03 INFO HadoopFSUtils: Picking up value for hoodie env var : HOODIE_ENV_fs_DOT_s3a_DOT_access_DOT_key 18805 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:03 INFO HadoopFSUtils: Picking up value for hoodie env var : HOODIE_ENV_fs_DOT_s3a_DOT_aws_DOT_credentials_DOT_provider 18805 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:03 INFO HadoopFSUtils: Picking up value for hoodie env var : HOODIE_ENV_fs_DOT_s3a_DOT_secret_DOT_key 18815 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:03 INFO HadoopFSUtils: Picking up value for hoodie env var : HOODIE_ENV_fs_DOT_s3a_DOT_endpoint 18819 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:03 INFO HadoopFSUtils: Picking up value for hoodie env var : HOODIE_ENV_fs_DOT_s3a_DOT_access_DOT_key 18819 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:03 INFO HadoopFSUtils: Picking up value for hoodie env var : HOODIE_ENV_fs_DOT_s3a_DOT_aws_DOT_credentials_DOT_provider 18819 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:03 INFO HadoopFSUtils: Picking up value for hoodie env var : HOODIE_ENV_fs_DOT_s3a_DOT_secret_DOT_key 18820 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:03 ERROR DFSPropertiesConfiguration: Error reading in properties from dfs from file file:/opt/hudi/packaging/hudi-cli-bundle/conf/hudi-defaults.conf 18820 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:03 INFO SparkContext: SparkContext is stopping with exitCode 0. 18826 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:03 INFO SparkUI: Stopped Spark web UI at http://openjdk8:4040 18834 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:03 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 18838 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:03 INFO MemoryStore: MemoryStore cleared 18838 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:03 INFO BlockManager: BlockManager stopped 18842 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:03 INFO BlockManagerMaster: BlockManagerMaster stopped 18843 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:03 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 18851 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - 24/09/06 17:18:03 INFO SparkContext: Successfully stopped SparkContext 18851 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - Exception in thread "main" java.lang.ExceptionInInitializerError 18851 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - at org.apache.hudi.utilities.UtilHelpers.buildProperties(UtilHelpers.java:277) 18852 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - at org.apache.hudi.utilities.HoodieCompactor.<init>(HoodieCompactor.java:67) 18852 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - at org.apache.hudi.cli.commands.SparkMain.compact(SparkMain.java:366) 18852 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - at org.apache.hudi.cli.commands.SparkMain.main(SparkMain.java:176) 18852 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 18852 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 18852 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 18852 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - at java.lang.reflect.Method.invoke(Method.java:498) 18852 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) 18852 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1020) 18852 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192) 18852 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215) 18852 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91) 18852 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1111) 18852 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1120) 18852 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 18852 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - Caused by: org.apache.hudi.exception.HoodieIOException: Cannot read properties from dfs from file file:/opt/hudi/packaging/hudi-cli-bundle/conf/hudi-defaults.conf 18852 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - at org.apache.hudi.common.config.DFSPropertiesConfiguration.addPropsFromFile(DFSPropertiesConfiguration.java:170) 18852 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - at org.apache.hudi.common.config.DFSPropertiesConfiguration.loadGlobalProps(DFSPropertiesConfiguration.java:122) 18852 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - at org.apache.hudi.common.config.DFSPropertiesConfiguration.<clinit>(DFSPropertiesConfiguration.java:71) 18852 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - ... 16 more 18852 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - Caused by: java.io.FileNotFoundException: File file:/opt/hudi/packaging/hudi-cli-bundle/conf/hudi-defaults.conf does not exist 18852 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:779) 18852 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:1100) 18852 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:769) 18852 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:462) 18852 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:160) 18852 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:372) 18852 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:976) 18853 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - at org.apache.hudi.storage.hadoop.HoodieHadoopStorage.open(HoodieHadoopStorage.java:148) 18853 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - at org.apache.hudi.common.config.DFSPropertiesConfiguration.addPropsFromFile(DFSPropertiesConfiguration.java:165) 18853 [Thread-5] INFO org.apache.hudi.cli.utils.InputStreamConsumer [] - ... 18 more ``` **Expected behavior** A clear and concise description of what you expected to happen. **Environment Description** * Hudi version : * Spark version : * Hive version : * Hadoop version : * Storage (HDFS/S3/GCS..) : * Running on Docker? (yes/no) : **Additional context** Add any other context about the problem here. **Stacktrace** ```Add the stacktrace of the error.``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
