Thanks, Steven, I’ll try this out! I’m having a spot of trouble locating FlinkSchemaUtil – which Iceberg module is that from? I thought it belonged to iceberg-flink, but seems 0.13.1 doesn’t contain it.
ah From: Steven Wu <stevenz...@gmail.com> Sent: Thursday, March 31, 2022 10:42 PM To: dev@iceberg.apache.org Subject: Re: Flink GenericRecord Iceberg Usage funny that I was just working on this recently. You can plugin this mapper to the FlinkSink builder https://gist.github.com/stevenzwu/4b824556973b47178824852083ab7a50<https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.github.com_stevenzwu_4b824556973b47178824852083ab7a50&d=DwMFaQ&c=7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=hRr4SA7BtUvKoMBP6VDhfisy2OJ1ZAzai-pcCC6TFXM&m=hg3GJeOj4I2ktkldMegIhMKlQw6J0ZIchFOXRjWJjXw&s=HUI4x5nvdxCJG7gTwvUyn6hwFsNHEbvg6Wr4hAvXcws&e=> RowType rowType = FlinkSchemaUtil.convert(icebergSchema); FlinkSink.builderFor( dataStream, AvroGenericRecordToRowDataMapper.forAvroSchema(avroSchema), FlinkCompatibilityUtil.toTypeInfo(rowType)) .table(table) .tableLoader(tableLoader) .writeParallelism(parallelism) .append(); On Thu, Mar 31, 2022 at 11:32 AM Hailu, Andreas <andreas.ha...@gs.com<mailto:andreas.ha...@gs.com>> wrote: Hello, I’m looking to write a proof of concept incorporating Iceberg into our ETL application. We use Avro as our schema management solution, and the Iceberg Flink documentation examples only show writing DataStream<Row> and DataStream<RowData> types. How can this be applied to for DataStream<GenericRecord>? Is there a translation step/utility? I found a discussion on an issue from 2020 [2] regarding the same thing, but it’s unclear of a solution was reached. [1] https://iceberg.apache.org/docs/latest/flink/<https://urldefense.proofpoint.com/v2/url?u=https-3A__iceberg.apache.org_docs_latest_flink_&d=DwMFaQ&c=7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=hRr4SA7BtUvKoMBP6VDhfisy2OJ1ZAzai-pcCC6TFXM&m=hg3GJeOj4I2ktkldMegIhMKlQw6J0ZIchFOXRjWJjXw&s=bCLuKffWSJGedPeqtEfLZe1UJZWw9BhQITqttcQdhFc&e=> [2] https://github.com/apache/iceberg/issues/1885<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_iceberg_issues_1885&d=DwMFaQ&c=7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=hRr4SA7BtUvKoMBP6VDhfisy2OJ1ZAzai-pcCC6TFXM&m=hg3GJeOj4I2ktkldMegIhMKlQw6J0ZIchFOXRjWJjXw&s=v3mVfawvHkDP645gX2lgLv9JmuNB4fGso-j3Wm91nRA&e=> best, ah ________________________________ Your Personal Data: We may collect and process information about you that may be subject to data protection laws. For more information about how we use and disclose your personal data, how we protect your information, our legal basis to use your information, your rights and who you can contact, please refer to: www.gs.com/privacy-notices<http://www.gs.com/privacy-notices> ________________________________ Your Personal Data: We may collect and process information about you that may be subject to data protection laws. For more information about how we use and disclose your personal data, how we protect your information, our legal basis to use your information, your rights and who you can contact, please refer to: www.gs.com/privacy-notices<http://www.gs.com/privacy-notices>
