Hi all,

I'm attempting to create a POC in flink to create a pipeline to stream parquet 
to a data warehouse in iceberg format.

Ideally - I'd like to watch a directory in s3 (minio locally) and stream those 
to iceberg, doing the appropriate schema mapping/translation.

I guess first; does this sound like a crazy idea?
Assuming not is anyone able to share examples that might get me going. I've 
found lots of iceberg and flink sql examples but I think I'll need something in 
java to do the schema mapping. Also some examples reading parquet for s3 seem a 
little hard to come by.

I'm aware I'll need a catalog, I can use nessie for the prototype. I'm also 
trying to use minio to get this all working locally but this might just be 
adding complexity at the moment.

T. Rowe Price International Ltd (registered number 3957748) is registered in 
England and Wales with its registered office at Warwick Court, 5 Paternoster 
Square, London EC4M 7DX. T. Rowe Price International Ltd is authorised and 
regulated by the Financial Conduct Authority. The company has a branch in Dubai 
International Financial Centre (regulated by the DFSA as a Representative 

T. Rowe Price (including T. Rowe Price International Ltd and its affiliates) 
and its associates do not provide legal or tax advice. Any tax-related 
discussion contained in this e-mail, including any attachments, is not intended 
or written to be used, and cannot be used, for the purpose of (i) avoiding any 
tax penalties or (ii) promoting, marketing, or recommending to any other party 
any transaction or matter addressed herein. Please consult your independent 
legal counsel and/or professional tax advisor regarding any legal or tax issues 
raised in this e-mail.

The contents of this e-mail and any attachments are intended solely for the use 
of the named addressee(s) and may contain confidential and/or privileged 
information. Any unauthorized use, copying, disclosure, or distribution of the 
contents of this e-mail is strictly prohibited by the sender and may be 
unlawful. If you are not the intended recipient, please notify the sender 
immediately and delete this e-mail.

Reply via email to