Machos65 opened a new issue, #13184: URL: https://github.com/apache/hudi/issues/13184
**_Tips before filing an issue_** - Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)? - Join the mailing list to engage in conversations and get faster support at [email protected]. - If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly. **Describe the problem you faced** I was trying to ingest data from kafka source which is in avro schema and whenever i run the submit i get this error 25/04/20 23:35:39 INFO SparkContext: Successfully stopped SparkContext Exception in thread "main" java.io.IOException: Could not load source class org.apache.hudi.utilities.sources.AvroKafkaSource at org.apache.hudi.utilities.UtilHelpers.createSource(UtilHelpers.java:165) at org.apache.hudi.utilities.streamer.StreamSync.<init>(StreamSync.java:321) at org.apache.hudi.utilities.streamer.HoodieStreamer$StreamSyncService.<init>(HoodieStreamer.java:768) at org.apache.hudi.utilities.streamer.HoodieStreamer.<init>(HoodieStreamer.java:176) at org.apache.hudi.utilities.streamer.HoodieStreamer.<init>(HoodieStreamer.java:151) at org.apache.hudi.utilities.streamer.HoodieStreamer.<init>(HoodieStreamer.java:137) at org.apache.hudi.utilities.streamer.HoodieStreamer.main(HoodieStreamer.java:646) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1034) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:199) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:222) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1125) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1134) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: org.apache.hudi.exception.HoodieException: Unable to instantiate class org.apache.hudi.utilities.sources.AvroKafkaSource at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:75) at org.apache.hudi.utilities.UtilHelpers.createSource(UtilHelpers.java:158) ... 18 more Caused by: java.lang.NoSuchMethodException: org.apache.hudi.utilities.sources.AvroKafkaSource.<init>(org.apache.hudi.common.config.TypedProperties,org.apache.spark.api.java.JavaSparkContext,org.apache.spark.sql.SparkSession,org.apache.hudi.utilities.schema.SchemaProvider) at java.base/java.lang.Class.getConstructor0(Class.java:3585) at java.base/java.lang.Class.getConstructor(Class.java:2271) at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:73) ... 19 more 25/04/20 23:35:39 INFO ShutdownHookManager: Shutdown hook called 25/04/20 23:35:39 INFO ShutdownHookManager: Deleting directory /tmp/spark-29e276dc-643e-4c4e-87f2-2d7382bd13fb 25/04/20 23:35:39 INFO ShutdownHookManager: Deleting directory /tmp/spark-e1869b95-56b2-4fc7-af42-b0e6b858c57c A clear and concise description of the problem. HOW can i overcome that one as am stuck there am using hudi streamer **To Reproduce** Steps to reproduce the behavior: 1. setup kafka debezium postgress to ingest data and the topics are created succesful with no problem 2. this is my kafka.properties for hudi streamer hoodie.datasource.write.recordkey.field=employee_id hoodie.datasource.write.partitionpath.field=country hoodie.datasource.write.precombine.field=ts auto.offset.reset=earliest kafka.bootstrap.servers=localhost:9092 hoodie.streamer.source.kafka.topic=dbserver1.public.employees hoodie.streamer.source.kafka.value.deserializer.class=org.apache.hudi.utilities.deser.KafkaAvroSchemaDeserializer hoodie.streamer.schemaprovider.registry.url=http://localhost:8082/subjects/dbserver1.public.employees-value/versions/latest schema.registry.url=http://localhost:8082 hoodie.streamer.schemaprovider.registry.schemaconverter= 3. here is my submit job spark-submit \ --class org.apache.hudi.utilities.streamer.HoodieStreamer \ --packages org.apache.hudi:hudi-spark3.5-bundle_2.12:1.0.1 \ --properties-file spark-config.properties \ --master 'local[*]' \ --executor-memory 1g \ /opt/spark/jars/hudi-utilities-slim-bundle_2.12-1.0.1.jar \ --table-type COPY_ON_WRITE \ --op UPSERT \ --source-ordering-field ts \ --source-class org.apache.hudi.utilities.sources.AvroKafkaSource \ --target-base-path file:///home/data/hoodie/employe \ --target-table employees \ --schemaprovider-class org.apache.hudi.utilities.schema.SchemaRegistryProvider \ --props hudi_tbl.properties 4. **Expected behavior** To generat hudi tables A clear and concise description of what you expected to happen. **Environment Description** * Hudi version : 1.0.1 * Spark version :3.5 * Hive version : (not syncing to hive as am perfoming test ) * Hadoop version :(same like hive ) * Storage (HDFS/S3/GCS..) : local storage * Running on Docker? (yes/no) : no **Additional context** Add any other context about the problem here. **Stacktrace** ```Add the stacktrace of the error.``` org.apache.hudi:hudi-spark3.5-bundle_2.12:1.0.1 --properties-file spark-config.properties --master 'local[*]' --executor-memory 1g /opt/spark/jars/hudi-utilities-slim-bundle_2.12-1.0.1.jar --table-type COPY_ON_WRITE --op UPSERT --source-ordering-field ts --source-class org.apache.hudi.utilities.sources.AvroKafkaSource --target-base-path file:///home/data/hoodie/employe --target-table employees --schemaprovider-class org.apache.hudi.utilities.schema.SchemaRegistryProvider --props hudi_tbl.properties 25/04/20 23:35:25 WARN Utils: Your hostname, ICTWKS-FP002 resolves to a loopback address: 127.0.1.1; using 10.255.255.254 instead (on interface lo) 25/04/20 23:35:25 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address :: loading settings :: url = jar:file:/opt/spark/jars/ivy-2.5.1.jar!/org/apache/ivy/core/settings/ivysettings.xml Ivy Default Cache set to: /root/.ivy2/cache The jars for the packages stored in: /root/.ivy2/jars org.apache.hudi#hudi-spark3.5-bundle_2.12 added as a dependency :: resolving dependencies :: org.apache.spark#spark-submit-parent-437a3ba5-d207-4661-9eaf-271eb586aef7;1.0 confs: [default] found org.apache.hudi#hudi-spark3.5-bundle_2.12;1.0.1 in central found org.apache.hive#hive-storage-api;2.8.1 in local-m2-cache found org.slf4j#slf4j-api;1.7.36 in local-m2-cache :: resolution report :: resolve 229ms :: artifacts dl 6ms :: modules in use: org.apache.hive#hive-storage-api;2.8.1 from local-m2-cache in [default] org.apache.hudi#hudi-spark3.5-bundle_2.12;1.0.1 from central in [default] org.slf4j#slf4j-api;1.7.36 from local-m2-cache in [default] --------------------------------------------------------------------- | | modules || artifacts | | conf | number| search|dwnlded|evicted|| number|dwnlded| --------------------------------------------------------------------- | default | 3 | 0 | 0 | 0 || 3 | 0 | --------------------------------------------------------------------- :: retrieving :: org.apache.spark#spark-submit-parent-437a3ba5-d207-4661-9eaf-271eb586aef7 confs: [default] 0 artifacts copied, 3 already retrieved (0kB/5ms) 25/04/20 23:35:25 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 25/04/20 23:35:26 WARN SchedulerConfGenerator: Job Scheduling Configs will not be in effect as spark.scheduler.mode is not set to FAIR at instantiation time. Continuing without scheduling configs 25/04/20 23:35:26 INFO SparkContext: Running Spark version 3.5.5 25/04/20 23:35:26 INFO SparkContext: OS info Linux, 5.15.167.4-microsoft-standard-WSL2, amd64 25/04/20 23:35:26 INFO SparkContext: Java version 17.0.2 25/04/20 23:35:26 INFO ResourceUtils: ============================================================== 25/04/20 23:35:26 INFO ResourceUtils: No custom resources configured for spark.driver. 25/04/20 23:35:26 INFO ResourceUtils: ============================================================== 25/04/20 23:35:26 INFO SparkContext: Submitted application: streamer-employees 25/04/20 23:35:26 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: , offHeap -> name: offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0) 25/04/20 23:35:26 INFO ResourceProfile: Limiting resource is cpu 25/04/20 23:35:26 INFO ResourceProfileManager: Added ResourceProfile id: 0 25/04/20 23:35:26 INFO SecurityManager: Changing view acls to: root 25/04/20 23:35:26 INFO SecurityManager: Changing modify acls to: root 25/04/20 23:35:26 INFO SecurityManager: Changing view acls groups to: 25/04/20 23:35:26 INFO SecurityManager: Changing modify acls groups to: 25/04/20 23:35:26 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: root; groups with view permissions: EMPTY; users with modify permissions: root; groups with modify permissions: EMPTY 25/04/20 23:35:26 INFO deprecation: mapred.output.compression.codec is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.codec 25/04/20 23:35:26 INFO deprecation: mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress 25/04/20 23:35:26 INFO deprecation: mapred.output.compression.type is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.type 25/04/20 23:35:26 INFO Utils: Successfully started service 'sparkDriver' on port 41493. 25/04/20 23:35:26 INFO SparkEnv: Registering MapOutputTracker 25/04/20 23:35:26 INFO SparkEnv: Registering BlockManagerMaster 25/04/20 23:35:26 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information 25/04/20 23:35:26 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up 25/04/20 23:35:26 INFO SparkEnv: Registering BlockManagerMasterHeartbeat 25/04/20 23:35:26 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-08ba099f-821c-419b-9a20-a568534ef670 25/04/20 23:35:26 INFO MemoryStore: MemoryStore started with capacity 434.4 MiB 25/04/20 23:35:26 INFO SparkEnv: Registering OutputCommitCoordinator 25/04/20 23:35:26 INFO JettyUtils: Start Jetty 0.0.0.0:8090 for SparkUI 25/04/20 23:35:26 WARN Utils: Service 'SparkUI' could not bind on port 8090. Attempting port 8091. 25/04/20 23:35:26 INFO Utils: Successfully started service 'SparkUI' on port 8091. 25/04/20 23:35:26 INFO SparkContext: Added JAR file:///root/.ivy2/jars/org.apache.hudi_hudi-spark3.5-bundle_2.12-1.0.1.jar at spark://10.255.255.254:41493/jars/org.apache.hudi_hudi-spark3.5-bundle_2.12-1.0.1.jar with timestamp 1745181326171 25/04/20 23:35:26 INFO SparkContext: Added JAR file:///root/.ivy2/jars/org.apache.hive_hive-storage-api-2.8.1.jar at spark://10.255.255.254:41493/jars/org.apache.hive_hive-storage-api-2.8.1.jar with timestamp 1745181326171 25/04/20 23:35:26 INFO SparkContext: Added JAR file:///root/.ivy2/jars/org.slf4j_slf4j-api-1.7.36.jar at spark://10.255.255.254:41493/jars/org.slf4j_slf4j-api-1.7.36.jar with timestamp 1745181326171 25/04/20 23:35:26 INFO SparkContext: Added JAR file:/opt/spark/jars/hudi-utilities-slim-bundle_2.12-1.0.1.jar at spark://10.255.255.254:41493/jars/hudi-utilities-slim-bundle_2.12-1.0.1.jar with timestamp 1745181326171 25/04/20 23:35:26 INFO Executor: Starting executor ID driver on host 10.255.255.254 25/04/20 23:35:26 INFO Executor: OS info Linux, 5.15.167.4-microsoft-standard-WSL2, amd64 25/04/20 23:35:26 INFO Executor: Java version 17.0.2 25/04/20 23:35:26 INFO Executor: Starting executor with user classpath (userClassPathFirst = false): '' 25/04/20 23:35:26 INFO Executor: Created or updated repl class loader org.apache.spark.util.MutableURLClassLoader@50b93353 for default. 25/04/20 23:35:26 INFO Executor: Fetching spark://10.255.255.254:41493/jars/hudi-utilities-slim-bundle_2.12-1.0.1.jar with timestamp 1745181326171 25/04/20 23:35:26 INFO TransportClientFactory: Successfully created connection to /10.255.255.254:41493 after 22 ms (0 ms spent in bootstraps) 25/04/20 23:35:27 INFO Utils: Fetching spark://10.255.255.254:41493/jars/hudi-utilities-slim-bundle_2.12-1.0.1.jar to /tmp/spark-e1869b95-56b2-4fc7-af42-b0e6b858c57c/userFiles-b3d945b3-5b33-4d28-8bc6-2454ccc8a404/fetchFileTemp4327358528652470493.tmp 25/04/20 23:35:27 INFO Executor: Adding file:/tmp/spark-e1869b95-56b2-4fc7-af42-b0e6b858c57c/userFiles-b3d945b3-5b33-4d28-8bc6-2454ccc8a404/hudi-utilities-slim-bundle_2.12-1.0.1.jar to class loader default 25/04/20 23:35:27 INFO Executor: Fetching spark://10.255.255.254:41493/jars/org.apache.hudi_hudi-spark3.5-bundle_2.12-1.0.1.jar with timestamp 1745181326171 25/04/20 23:35:27 INFO Utils: Fetching spark://10.255.255.254:41493/jars/org.apache.hudi_hudi-spark3.5-bundle_2.12-1.0.1.jar to /tmp/spark-e1869b95-56b2-4fc7-af42-b0e6b858c57c/userFiles-b3d945b3-5b33-4d28-8bc6-2454ccc8a404/fetchFileTemp3571753006441842551.tmp 25/04/20 23:35:27 INFO Executor: Adding file:/tmp/spark-e1869b95-56b2-4fc7-af42-b0e6b858c57c/userFiles-b3d945b3-5b33-4d28-8bc6-2454ccc8a404/org.apache.hudi_hudi-spark3.5-bundle_2.12-1.0.1.jar to class loader default 25/04/20 23:35:27 INFO Executor: Fetching spark://10.255.255.254:41493/jars/org.apache.hive_hive-storage-api-2.8.1.jar with timestamp 1745181326171 25/04/20 23:35:27 INFO Utils: Fetching spark://10.255.255.254:41493/jars/org.apache.hive_hive-storage-api-2.8.1.jar to /tmp/spark-e1869b95-56b2-4fc7-af42-b0e6b858c57c/userFiles-b3d945b3-5b33-4d28-8bc6-2454ccc8a404/fetchFileTemp8185769743879859779.tmp 25/04/20 23:35:27 INFO Executor: Adding file:/tmp/spark-e1869b95-56b2-4fc7-af42-b0e6b858c57c/userFiles-b3d945b3-5b33-4d28-8bc6-2454ccc8a404/org.apache.hive_hive-storage-api-2.8.1.jar to class loader default 25/04/20 23:35:27 INFO Executor: Fetching spark://10.255.255.254:41493/jars/org.slf4j_slf4j-api-1.7.36.jar with timestamp 1745181326171 25/04/20 23:35:27 INFO Utils: Fetching spark://10.255.255.254:41493/jars/org.slf4j_slf4j-api-1.7.36.jar to /tmp/spark-e1869b95-56b2-4fc7-af42-b0e6b858c57c/userFiles-b3d945b3-5b33-4d28-8bc6-2454ccc8a404/fetchFileTemp5764230525616327387.tmp 25/04/20 23:35:27 INFO Executor: Adding file:/tmp/spark-e1869b95-56b2-4fc7-af42-b0e6b858c57c/userFiles-b3d945b3-5b33-4d28-8bc6-2454ccc8a404/org.slf4j_slf4j-api-1.7.36.jar to class loader default 25/04/20 23:35:27 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 36669. 25/04/20 23:35:27 INFO NettyBlockTransferService: Server created on 10.255.255.254:36669 25/04/20 23:35:27 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy 25/04/20 23:35:27 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.255.255.254, 36669, None) 25/04/20 23:35:27 INFO BlockManagerMasterEndpoint: Registering block manager 10.255.255.254:36669 with 434.4 MiB RAM, BlockManagerId(driver, 10.255.255.254, 36669, None) 25/04/20 23:35:27 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.255.255.254, 36669, None) 25/04/20 23:35:27 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 10.255.255.254, 36669, None) 25/04/20 23:35:28 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir. 25/04/20 23:35:28 INFO SharedState: Warehouse path is 'file:/opt/spark/spark-warehouse'. 25/04/20 23:35:28 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from file:///home/data/hoodie/employe 25/04/20 23:35:28 INFO HoodieTableConfig: Loading table properties from file:/home/data/hoodie/employe/.hoodie/hoodie.properties 25/04/20 23:35:28 WARN ConfigUtils: Could not read properties from file:/home/data/hoodie/employe/.hoodie/hoodie.properties: java.io.FileNotFoundException: File file:/home/data/hoodie/employe/.hoodie/hoodie.properties does not exist 25/04/20 23:35:28 WARN ConfigUtils: Could not read properties from file:/home/data/hoodie/employe/.hoodie/hoodie.properties.backup: java.io.FileNotFoundException: File file:/home/data/hoodie/employe/.hoodie/hoodie.properties.backup does not exist 25/04/20 23:35:29 WARN ConfigUtils: Could not read properties from file:/home/data/hoodie/employe/.hoodie/hoodie.properties: java.io.FileNotFoundException: File file:/home/data/hoodie/employe/.hoodie/hoodie.properties does not exist 25/04/20 23:35:29 WARN ConfigUtils: Could not read properties from file:/home/data/hoodie/employe/.hoodie/hoodie.properties.backup: java.io.FileNotFoundException: File file:/home/data/hoodie/employe/.hoodie/hoodie.properties.backup does not exist 25/04/20 23:35:30 WARN ConfigUtils: Could not read properties from file:/home/data/hoodie/employe/.hoodie/hoodie.properties: java.io.FileNotFoundException: File file:/home/data/hoodie/employe/.hoodie/hoodie.properties does not exist 25/04/20 23:35:30 WARN ConfigUtils: Could not read properties from file:/home/data/hoodie/employe/.hoodie/hoodie.properties.backup: java.io.FileNotFoundException: File file:/home/data/hoodie/employe/.hoodie/hoodie.properties.backup does not exist 25/04/20 23:35:31 WARN ConfigUtils: Could not read properties from file:/home/data/hoodie/employe/.hoodie/hoodie.properties: java.io.FileNotFoundException: File file:/home/data/hoodie/employe/.hoodie/hoodie.properties does not exist 25/04/20 23:35:31 WARN ConfigUtils: Could not read properties from file:/home/data/hoodie/employe/.hoodie/hoodie.properties.backup: java.io.FileNotFoundException: File file:/home/data/hoodie/employe/.hoodie/hoodie.properties.backup does not exist 25/04/20 23:35:32 WARN ConfigUtils: Could not read properties from file:/home/data/hoodie/employe/.hoodie/hoodie.properties: java.io.FileNotFoundException: File file:/home/data/hoodie/employe/.hoodie/hoodie.properties does not exist 25/04/20 23:35:32 WARN ConfigUtils: Could not read properties from file:/home/data/hoodie/employe/.hoodie/hoodie.properties.backup: java.io.FileNotFoundException: File file:/home/data/hoodie/employe/.hoodie/hoodie.properties.backup does not exist 25/04/20 23:35:33 WARN HoodieStreamer: Full exception msg Could not load Hoodie properties from file:/home/data/hoodie/employe/.hoodie/hoodie.properties, msg Could not load Hoodie properties from file:/home/data/hoodie/employe/.hoodie/hoodie.properties 25/04/20 23:35:33 INFO HoodieStreamer: Creating Hudi Streamer with configs: auto.offset.reset: earliest hoodie.auto.adjust.lock.configs: true hoodie.datasource.hive_sync.enable: false hoodie.datasource.hive_sync.use_jdbc: false hoodie.datasource.write.hive_style_partitioning: true hoodie.datasource.write.keygenerator.class: org.apache.hudi.keygen.SimpleKeyGenerator hoodie.datasource.write.operation: upsert hoodie.datasource.write.partitionpath.field: country hoodie.datasource.write.precombine.field: ts hoodie.datasource.write.reconcile.schema: false hoodie.datasource.write.recordkey.field: employee_id hoodie.datasource.write.table.type: COPY_ON_WRITE hoodie.streamer.schemaprovider.registry.schemaconverter: hoodie.streamer.schemaprovider.registry.url: http://localhost:8082/subjects/dbserver1.public.employees-value/versions/latest hoodie.streamer.source.kafka.topic: dbserver1.public.employees hoodie.streamer.source.kafka.value.deserializer.class: org.apache.hudi.utilities.deser.KafkaAvroSchemaDeserializer kafka.bootstrap.servers: localhost:9092 schema.registry.url: http://localhost:8082 25/04/20 23:35:34 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from file:///home/data/hoodie/employe 25/04/20 23:35:34 INFO HoodieTableConfig: Loading table properties from file:/home/data/hoodie/employe/.hoodie/hoodie.properties 25/04/20 23:35:34 WARN ConfigUtils: Could not read properties from file:/home/data/hoodie/employe/.hoodie/hoodie.properties: java.io.FileNotFoundException: File file:/home/data/hoodie/employe/.hoodie/hoodie.properties does not exist 25/04/20 23:35:34 WARN ConfigUtils: Could not read properties from file:/home/data/hoodie/employe/.hoodie/hoodie.properties.backup: java.io.FileNotFoundException: File file:/home/data/hoodie/employe/.hoodie/hoodie.properties.backup does not exist 25/04/20 23:35:35 WARN ConfigUtils: Could not read properties from file:/home/data/hoodie/employe/.hoodie/hoodie.properties: java.io.FileNotFoundException: File file:/home/data/hoodie/employe/.hoodie/hoodie.properties does not exist 25/04/20 23:35:35 WARN ConfigUtils: Could not read properties from file:/home/data/hoodie/employe/.hoodie/hoodie.properties.backup: java.io.FileNotFoundException: File file:/home/data/hoodie/employe/.hoodie/hoodie.properties.backup does not exist 25/04/20 23:35:36 WARN ConfigUtils: Could not read properties from file:/home/data/hoodie/employe/.hoodie/hoodie.properties: java.io.FileNotFoundException: File file:/home/data/hoodie/employe/.hoodie/hoodie.properties does not exist 25/04/20 23:35:36 WARN ConfigUtils: Could not read properties from file:/home/data/hoodie/employe/.hoodie/hoodie.properties.backup: java.io.FileNotFoundException: File file:/home/data/hoodie/employe/.hoodie/hoodie.properties.backup does not exist 25/04/20 23:35:37 WARN ConfigUtils: Could not read properties from file:/home/data/hoodie/employe/.hoodie/hoodie.properties: java.io.FileNotFoundException: File file:/home/data/hoodie/employe/.hoodie/hoodie.properties does not exist 25/04/20 23:35:37 WARN ConfigUtils: Could not read properties from file:/home/data/hoodie/employe/.hoodie/hoodie.properties.backup: java.io.FileNotFoundException: File file:/home/data/hoodie/employe/.hoodie/hoodie.properties.backup does not exist 25/04/20 23:35:38 WARN ConfigUtils: Could not read properties from file:/home/data/hoodie/employe/.hoodie/hoodie.properties: java.io.FileNotFoundException: File file:/home/data/hoodie/employe/.hoodie/hoodie.properties does not exist 25/04/20 23:35:38 WARN ConfigUtils: Could not read properties from file:/home/data/hoodie/employe/.hoodie/hoodie.properties.backup: java.io.FileNotFoundException: File file:/home/data/hoodie/employe/.hoodie/hoodie.properties.backup does not exist 25/04/20 23:35:39 WARN StreamSync: Full exception msg Could not load Hoodie properties from file:/home/data/hoodie/employe/.hoodie/hoodie.properties 25/04/20 23:35:39 WARN StreamSync: Base path exists, but table is not fully initialized. Re-initializing again 25/04/20 23:35:39 INFO HoodieTableMetaClient: Initializing file:/home/data/hoodie/employe as hoodie table 25/04/20 23:35:39 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from file:/home/data/hoodie/employe 25/04/20 23:35:39 INFO HoodieTableConfig: Loading table properties from file:/home/data/hoodie/employe/.hoodie/hoodie.properties 25/04/20 23:35:39 INFO HoodieTableMetaClient: Finished Loading Table of type COPY_ON_WRITE(version=2) from file:/home/data/hoodie/employe 25/04/20 23:35:39 INFO HoodieTableMetaClient: Finished initializing Table of type COPY_ON_WRITE from file:/home/data/hoodie/employe 25/04/20 23:35:39 INFO ActiveTimelineV2: Loaded instants upto : Optional.empty 25/04/20 23:35:39 INFO SparkContext: SparkContext is stopping with exitCode 0. 25/04/20 23:35:39 INFO SparkUI: Stopped Spark web UI at http://10.255.255.254:8091 25/04/20 23:35:39 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 25/04/20 23:35:39 INFO MemoryStore: MemoryStore cleared 25/04/20 23:35:39 INFO BlockManager: BlockManager stopped 25/04/20 23:35:39 INFO BlockManagerMaster: BlockManagerMaster stopped 25/04/20 23:35:39 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 25/04/20 23:35:39 INFO SparkContext: Successfully stopped SparkContext Exception in thread "main" java.io.IOException: Could not load source class org.apache.hudi.utilities.sources.AvroKafkaSource at org.apache.hudi.utilities.UtilHelpers.createSource(UtilHelpers.java:165) at org.apache.hudi.utilities.streamer.StreamSync.<init>(StreamSync.java:321) at org.apache.hudi.utilities.streamer.HoodieStreamer$StreamSyncService.<init>(HoodieStreamer.java:768) at org.apache.hudi.utilities.streamer.HoodieStreamer.<init>(HoodieStreamer.java:176) at org.apache.hudi.utilities.streamer.HoodieStreamer.<init>(HoodieStreamer.java:151) at org.apache.hudi.utilities.streamer.HoodieStreamer.<init>(HoodieStreamer.java:137) at org.apache.hudi.utilities.streamer.HoodieStreamer.main(HoodieStreamer.java:646) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1034) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:199) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:222) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1125) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1134) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: org.apache.hudi.exception.HoodieException: Unable to instantiate class org.apache.hudi.utilities.sources.AvroKafkaSource at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:75) at org.apache.hudi.utilities.UtilHelpers.createSource(UtilHelpers.java:158) ... 18 more Caused by: java.lang.NoSuchMethodException: org.apache.hudi.utilities.sources.AvroKafkaSource.<init>(org.apache.hudi.common.config.TypedProperties,org.apache.spark.api.java.JavaSparkContext,org.apache.spark.sql.SparkSession,org.apache.hudi.utilities.schema.SchemaProvider) at java.base/java.lang.Class.getConstructor0(Class.java:3585) at java.base/java.lang.Class.getConstructor(Class.java:2271) at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:73) ... 19 more 25/04/20 23:35:39 INFO ShutdownHookManager: Shutdown hook called 25/04/20 23:35:39 INFO ShutdownHookManager: Deleting directory /tmp/spark-29e276dc-643e-4c4e-87f2-2d7382bd13fb 25/04/20 23:35:39 INFO ShutdownHookManager: Deleting directory /tmp/spark-e1869b95-56b2-4fc7-af42-b0e6b858c57c -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
