nsivabalan commented on issue #7657: URL: https://github.com/apache/hudi/issues/7657#issuecomment-1396365792
there are some issues wrt some of the metadata configs. Some of the metadata table configs are not meant to be overridden. --hoodie-conf hoodie.metadata.compact.max.delta.commits=10 \ --hoodie-conf hoodie.metadata.cleaner.commits.retained=60 \ --hoodie-conf hoodie.metadata.populate.meta.fields=true \ --hoodie-conf hoodie.metadata.clean.async=true \ --hoodie-conf hoodie.metadata.keep.max.commits=80 \ --hoodie-conf hoodie.metadata.insert.parallelism=20 \ --hoodie-conf hoodie.metadata.keep.min.commits=70 \ Please do not override any of these configs. I have created a follow up ticket to not expose these configs https://issues.apache.org/jira/browse/HUDI-5582 Can you restart metadata from scratch w/ right set of configs and let us know how it goes. But I have some clarification as well. I see you are setting async indexing for metadata table to true. So, are you spinning up a separate spark submit job (other than deltastreamer) to build the metadata indexes. If not, you may have to fix that as well. More details can be found here https://hudi.apache.org/docs/metadata_indexing -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
