usbrandon opened a new issue, #2902: URL: https://github.com/apache/hop/issues/2902
### What would you like to happen? I spent considerable time deploying Dremio and working with their professional services group to understand the settings in Hop and their impact on how typical query engines can most efficiently process queries upon files output. Please update the dialog labels to communicate to users the unit of measure (bytes, rows, etc) and suggest values that are optimal out of the gate for most use cases, which should revolve around the target files being used in queries. Concretely, label on the left updated and suggested default on the right Default: File Version: Parquet Version 2.0 Row group size (bytes): 268435456 <- 256 MB (V-core to memory ratio 4 to 8. In md5.4xlarge that is 64GB of memory per core, so each row group has enough room for sorts etc.) Data page size (bytes): 8192 Data Dictionary size (bytes): 10485760 Please updated hop to the latest release of parquet 1.13.0 from 1.12 Apache Parquet recommends not exceeding 8kb for the data page size: https://parquet.apache.org/docs/file-format/configurations/#:~:text=Data%20Page%20Size,-Data%20pages%20should&text=Note%3A%20for%20sequential%20scans%2C%20it,recommend%208KB%20for%20page%20sizes. The 10 MB Data Dictionary size is typically recommended in practice to allow enough bytes to store metadata about the columns.  ### Issue Priority Priority: 3 ### Issue Component Component: Transforms -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
