I personally have no problem with accepting it. I don't see big issue with adding one more sql operator - we already have quite "specialized" operators there - and the whole purpose of providers is to allow the users who are not very proficient in writing Python code to create their DAGs from "predefined building blocks" - that one seems to be nicely fitting that category. It might be a bit tricky to describe it well (especially what exactly is the format of the data in XCom?) and some error scenarios on parsing those records, but if you have it generic enough and tried/tested, I see no big problem with accepting it.
On Sun, Feb 9, 2025 at 5:43 AM Amogh Desai <amoghdesai....@gmail.com> wrote: > Thanks for the proposal and apologies for the delayed response. > > I think the use case you are offering is a very niche one and I am not sure > as to how many users would have > similar requirements here. > > Providers are meant to provide means to interact with external systems and > not any specific *personalised* customisation > on that basic interaction. > > As far as I understand the use case you have here, I'd really like to hear > from the community on whether there are > similar use cases and then consider it. > > Also, looking at it from a maintenance perspective, would you / your > company be owning the maintenance of this new operator? > > Thanks & Regards, > Amogh Desai > > > On Thu, Jan 16, 2025 at 12:30 AM Blain David <david.bl...@infrabel.be> > wrote: > > > Hello, > > > > > > > > Here I’m again with another proposal, which shouldn’t be that hard to > > evaluate and is also related to the work I did regarding the dialects and > > performance enhancements in the common sql provider but also the PR > > regarding the deferred pagination in the GenericTransfer operator which > I’m > > finishing as we speak and I also mentioned in the medium > > < > https://medium.com/apache-airflow/transfering-data-from-sap-hana-to-mssql-using-the-airflow-generictransfer-d29f147a9f1f > > > > article I wrote about it. > > > > At our company we are using a custom SQLInsertRowsOperator, which allows > > us to persist XCom’s directly without the need to write a custom Python > > code, so it’s again like some kind of facilitator on top of the > DbApiHook. > > > > Hence why the work with the dialects and other related PR’s were so > > important to be able to implement it in a correct way, meaning as less as > > possible logic within the operator so that all logic can be handled > within > > the hook and both options can be used the same way. > > > > > > > > So my question is if that operator would be accepted? The code would be > > minimal, it could be added beside the other SQL operators within the > common > > sql provider. > > > > It’s similar to the GenericTransfer operator, except it doesn’t read data > > from another database, it uses an XCom as input for the rows to be > > persisted by the insert_rows method of the DbAPiHook. > > > > It also offers some handy callbacks parameters to process the rows which > > has to be persisted. > > > > > > > > As I already explained before, at our company we try to strive to have a > > less as possible custom python code within our DAG’s, and use as much as > > possible existing Airflow operators, which make maintenance of DAG’s > easier. > > > > This one would allow Airflow users to easily persist XCom’s without the > > need to write Python code. > > > > > > > > Below an example on how it could be used: > > > > persist_records_task = SQLInsertRowsOperator( > > task_id="persist_records", > > conn_id=conn_id”, > > schema="schema", > > table_name=”table_name”, > > insert_args={ > > "commit_every": 5000, > > "replace": True, > > "executemany": True, > > "fast_executemany": True, > > }, > > rows=csv_to_records_tasks.output, > > > > ) > > > > > > > > What do you think about this proposal? > > > > *David Blain* > > > > Data Engineer *at* ICT-514 - BI End User Reporting > > > > > > > > > > >
