Hey Luyao! Thanks for reaching out. Beam's Iceberg connector supports dynamic destinations so long as they have equivalent Schemas. There's no support for dynamic schemas yet, i.e. there's no `getDataSchema(<destination>)` method.
There's a feature request on Github: https://github.com/apache/beam/issues/33724. > I don't think Managed.write support with DynamicDestinations ManagedIO does support dynamic destinations, in the format described here: https://cloud.google.com/dataflow/docs/guides/managed-io#dynamic-destinations. It also does not support dynamic schemas yet. Best, Ahmed On 2025/01/23 04:02:57 Ling Li wrote: > Hi team, > > Happy New Year! I have a question regarding > the "org.apache.beam:beam-sdks-java-io-iceberg:2.61.0". > > Does the org.apache.beam.sdk.schemas.Schema getDataSchema(String > destination) of class DynamicDestinations already exist? > > Our team tried to use DynamicDestinations to decide which iceberg table to > write in the runtime but failed. > > > Attached is my code class DynamicIcebergDestinations. Our problem is that > we don't have a universal schema that can match all events to be > implemented for getDataSchema(). We need to use a parameter to get the > correct schema for each event, so we want to use: getDataSchema(String > destination). But seems getDataSchema(String destination) is not > implemented for org.apache.beam.sdk.io.iceberg.DynamicDestinations. Since > getDataSchema() will be automatically called and it cannot return a > universal schema, there is no way for us to > use IcebergIO.writeRows(icebergCatalogConfig).to(dynamicDestinations). I > also tried to use managed I/O connector, but I don't think > Managed.write support > with DynamicDestinations. Could you give some urgent guidance? > > Thank you so much. > > Sincerely, > Luyao >
