DanielMorales9 commented on PR #32939: URL: https://github.com/apache/beam/pull/32939#issuecomment-2437438886
I think a more scalable approach here would be to encapsulate the writing logic within a Parquet writer class. This would be similar to how Spark or Flink handle Parquet writes (e.g., i.e. [SparkParquetWriter](https://github.com/apache/iceberg/blob/fdc2c223efa3367f8a10d9c49985326352b687b4/spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/data/SparkParquetWriters.java#L61), [FlinkParquetWriter](https://github.com/apache/iceberg/blob/fdc2c223efa3367f8a10d9c49985326352b687b4/flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/data/FlinkParquetWriters.java#L53)), allowing you to manage the type conversions and partitioning requirements specific to Iceberg in a centralized and reusable way." -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
