yarenty commented on issue #1544: URL: https://github.com/apache/arrow-datafusion/issues/1544#issuecomment-1165574649
I just trying to integrate datafusion with kafka, final goal is to have end-to-end streaming. But I started from a "different side" => step 1 is to publish output to kafka, so I copied code/ created kafka publisher: https://github.com/yarenty/arrow-datafusion/tree/master/datafusion/core/src/physical_plan/kafka Test case is here: https://github.com/yarenty/arrow-datafusion/blob/master/datafusion/core/tests/ordered_sql_to_kafka.rs All finished with something like this: ```rust #[tokio::main] async fn main() -> Result<()> { let ctx = SessionContext::new(); ctx.register_csv("example", "tests/example.csv", CsvReadOptions::new()).await?; let df = ctx .sql("SELECT a, MIN(b) as bmin FROM example GROUP BY a ORDER BY a LIMIT 100") .await?; // kafka context let stream_ctx = KafkaContext::with_config( KafkaConfig::new("test_topic") .set("bootstrap.servers", "127.0.0.1:9092") .set("compression.codec", "snappy"), ); df.publish_to_kafka( stream_ctx).await?; Ok(()) } ``` Still not sure if this is the correct way to do it and if I put code in the proper places ... still ... you are learning something new every day. Is there any other place where you can share code / check ideas? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
