+1 Agree with you
TaoZex <[email protected]> 于2023年7月22日周六 13:37写道： > Agree with you. We should also consider what metadata information is > necessary to count in the catalog. > > > Best Regards > TaoZex > ------------------ 原始邮件 ------------------ > 发件人: > "dev" > < > [email protected]>; > 发送时间: 2023年7月22日(星期六) 上午10:17 > 收件人: "dev"<[email protected]>; > > 主题: [Proposal] Spark&Flink compatible catalog > > > > *background* > > Due to the existence of commercial versions and open source web, it is > necessary to clearly know the data source table and the table structure > processed by each node when defining tasks. So the entire architecture > layer needs to use a catalog approach, and the previous description of > SeaTunnelRowType cannot meet the synchronization requirements. > At present, the Zeta engine has supported the creation of connectors using > the catalog method, but spark/link still uses the SeaTunnelRowType > interface, which requires the connector to implement both the Catalog and > SeaTunnelRowType interfaces in order to execute on different engines. This > will require a lot of work and is meaningless. > To solve this problem, we need spark/link to first support the Catalog > method, so that the new connector can only implement the Catalog interface, > while the old connector can gradually remove the SeaTunnelRowType > interface. > > -- > > Best Regards > > ------------ > > EricJoy2048 > [email protected]
