parisni commented on issue #8925: URL: https://github.com/apache/hudi/issues/8925#issuecomment-1585086168
You are right this is likely the default. You can make sure by looking into the marker directory while writing process. .hoodie/.temp/commit When timeline server used then few large files are appended. However one file per new parquet file is creates. In case of very large commit with many written files there is an overhead creating/dropping them. Could you share a screenshot of your spark ui after job conpletion ? On June 9, 2023 8:00:32 PM UTC, Samarth Raval ***@***.***> wrote: >@parisni >Do you mean using this ? >https://hudi.apache.org/docs/next/configurations#hoodiewritemarkerstype > >Default is hoodiewritemarkerstype: TIMELINE_SERVER_BASED > >Or should I specify explicitly ? > >Using hudi 0.12.1 > >-- >Reply to this email directly or view it on GitHub: >https://github.com/apache/hudi/issues/8925#issuecomment-1585067556 >You are receiving this because you were mentioned. > >Message ID: ***@***.***> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
