Hi, Harshvardhan I think CaiZhi is right. I only have a small addition. Because I see that you want to convert Table to DataStream, you can look at FileSink (ParquetWriterFactory)[1].
[1] https://ci.apache.org/projects/flink/flink-docs-master/docs/connectors/datastream/file_sink/#bulk-encoded-formats Best, Guowei On Sun, Sep 26, 2021 at 10:31 AM Caizhi Weng <tsreape...@gmail.com> wrote: > Hi! > > Try the PARTITIONED BY clause. See > https://ci.apache.org/projects/flink/flink-docs-master/docs/connectors/table/formats/parquet/ > > Harshvardhan Shinde <harshvardhan.shi...@oyorooms.com> 于2021年9月24日周五 > 下午5:52写道： > >> Hi, >> I wanted to know if we can write streaming data to S3 in parquet format >> with partitioning. >> Here's what I want to achieve: >> I have a kafka table which gets updated with the data from kafka topic >> and I'm using select statement to get the data into a Table and converting >> into a stream as: >> >> StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env); >> Table table = tableEnv.sqlQuery("Select * from test"); >> DataStream<Row> stream = tableEnv.toDataStream(table); >> >> Now I want to write this stream to S3 in parquet files with hourly >> partitions. >> >> Here are my questions: >> 1. Is this possible? >> 2. If yes, how it can be achieved or link to appropriate documentation. >> >> Thanks and Regards, >> Harshvardhan >> >>
