spetz commented on code in PR #1826: URL: https://github.com/apache/iggy/pull/1826#discussion_r2118056766
########## core/connectors/README.md: ########## @@ -0,0 +1,93 @@ +# Apache Iggy Connectors + +The highly performant and modular runtime for statically typed, yet dynamically loaded connectors. Ingest the data from the external sources and push it further to the Iggy streams, or fetch the data from the Iggy streams and push it further to the external sources. Create your own Rust plugins by simply implementing either the `Source` or `Sink` trait and build custom pipelines for the data processing. + +**This is still WiP, and the runtime can be started only after compilation from the source code (no installable package yet).** + +## Features +- **High Performance**: Utilizes Rust's performance characteristics to ensure fast data ingestion and egress. +- **Low memory footprint**: Designed with memory efficiency in mind, minimizing the memory footprint of the connectors. +- **Modular Design**: Designed with modularity in mind, allowing for easy extension and customization. +- **Dynamic Loading**: Supports dynamic loading of plugins, enabling seamless integration with various data sources and sinks. +- **Statically Typed**: Ensures type safety and compile-time checks, reducing runtime errors. +- **Easy Customization**: Provides a simple interface for implementing custom connectors, making it easy to create new plugins. +- **Data transformation**: Supports data transformation with the help of existing functions. +- **Powerful configuration**: Define your sinks, sources, and transformations in the configuration file. + +## Quick Start + +1. Uncomment the `crate-type = ["cdylib"]` in `Cargo.toml` for the connectors under `core/connectors/sources` and `core/connectors/sinks` to compile them as dynamic libraries (`.so`, `.dylib`, `.dll`). Review Comment: Just in case, someone would like to use Windows. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
