ccl125 commented on issue #3102: URL: https://github.com/apache/parquet-java/issues/3102#issuecomment-2537600167
I noticed that when I set withDictionaryEncoding(false), the writer switches from using FallbackValuesWriter to PlainValuesWriter. These two have significantly different memory usage. It seems that using PlainValuesWriter might address my issue. Here is the context: • Each file has a fixed 500 rows. • The number of columns varies, ranging from approximately 1 to 30,000. I would like to know: 1. Can I directly solve the problem by setting withDictionaryEncoding(false)? 2. How will this impact file size, write efficiency, and read performance? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@parquet.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@parquet.apache.org For additional commands, e-mail: issues-h...@parquet.apache.org