Hi Gerard, Good to know :) FYI, Linkedin created https://github.com/apache/incubator-gobblin as the data ingestion framework which could potentially solve your use case. But I am not sure it provides support for sqlserver as sink.
Best, -Tao On Fri, May 31, 2019 at 6:58 AM Gerard Toonstra <[email protected]> wrote: > Hi all, > > We're running airflow and write data to sqlserver. Python and sqlserver > isn't a marriage made in heaven and we've literally spent hours fixing > driver issues, installations and working around things that should be easy, > time that's better spent writing app code. > > I looked at Sqoop, but that runs on Hadoop only. I didn't find anything > that was opensource / standalone to use as a "generic data pump", so > started a new project for that (let me know if something does exist). The > overall idea is that an airflow operator generates a command file for this > process and then spawns a java process to run the pipeline. > > It's available here: > > https://github.com/gtoonstra/genevere > > If you have similar issues with your environment, maybe this helps you out. > And if you want to contribute, just reach out! > > Rgds, > > Gerard >
