deepakpanda93 commented on issue #16412: URL: https://github.com/apache/hudi/issues/16412#issuecomment-4852217662
This is fixed in Hudi 1.1.0 (and backported to 0.15.1). **Root cause:** with hoodie.datasource.write.reconcile.schema=true and hoodie.schema.on.read.enable=true, each commit reconciled the schema and persisted a new InternalSchema version under .hoodie/.schema/ even when the schema hadn't actually changed. The accumulated schema history then grew unbounded and eventually caused the OOM. **Fix:** HUDI-9597 / [#13731](https://github.com/apache/hudi/pull/13731) ("Do not evolve schema if reconciled schema keeps unchanged except version id") — the writer no longer emits a new schema version when the reconciled schema is unchanged, so the schema commits stop accumulating on no-op-schema writes. It's in 1.1.0 and was backported to 0.15.1 ([#17829](https://github.com/apache/hudi/pull/17829)). Please move to 1.1.x (or 0.15.1). Closing as fixed — if you still see the .schemacommit history growing on unchanged-schema commits on those versions, please reopen with the .hoodie/.schema/ listing over several commits. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
