Hi David, Thanks for starting this, +1 from me. Having fixed the folder marker issue in several transfer operators across the AWS, GCP and Azure providers, I'd be glad to see it solved once.
That said, one thing I'm a little worried about is that a regression in the generic logic would reach every pair at once, while in those fixes each change only affected the two services of its pair. Happy to help where I can. Best regards, Yuseok On Mon, Nov 9, 2026 at 6:49 PM Blain David <[email protected]> wrote: > Hi all, > > While reviewing #76408 [1] I noticed it was the fourth fix of the same bug. > > Object stores create a zero-byte "folder marker" for every folder. A > transfer operator that lists keys and copies them one by one either fails > on the marker or copies it as an empty object. The same rule (skip a marker > that has keys under it) was applied to S3ToGCSOperator [2], GCSToS3Operator > [3], AzureBlobStorageToS3Operator [4] and S3ToAzureBlobStorageOperator [1]. > That is four PRs in three providers over six months, and main now carries > three copies of the same helper, because provider packages cannot share > code without a version floor. Nobody did anything wrong there. The > structure leaves no other option. > > The structure is one operator per source/destination pair. On main today I > count 86 transfer modules, about 17,500 lines, in 13 providers (google 37, > amazon 21, microsoft.azure 6, apache.hive 6, and nine smaller ones). Sorted > by what they do, roughly: > > * > 28 copy files from one store to another (S3, GCS, Azure Blob, ADLS, SFTP, > FTP, local, Drive, Samba) > * > 16 move rows from one database to another > * > 11 load files into a database > * > 26 read records from a database or an API and write them to files > (SqlToS3, HttpToS3, HttpToGCS, MongoToS3, SalesforceToGCS, and > BaseSQLToGCSOperator with six dialect subclasses) > * > 5 send records elsewhere (Slack, Sheets) > > Every fix and every feature (pagination, a file format, async, lineage) > lands in one of them and is missing from the others. HttpOperator can > paginate, for example; HttpToS3Operator and HttpToGCSOperator cannot. > We have been here before, and we solved it. Until 2022 every database > provider shipped its own operator. common.sql 1.3 replaced them with > SQLExecuteQueryOperator on top of the DbApiHook contract, the dialect moved > into the hooks, and the specific operators became deprecated shells. AIP-14 > "Create composable operators using common interfaces" [5] asked for the > same thing for transfers in 2019: hooks behind a common interface and one > copy operator that takes a source hook and a destination hook. It was only > partly implemented. Its file half landed as AIP-58 [6], which gave us > ObjectStoragePath, and FileTransferOperator in common.io, since Airflow > 2.8. The AIP-14 page is marked "Superseded & Completed" by AIP-58 since > 2024, but the goal it was written for, no more X-to-Y operators, was never > reached. > > So AIP-14 is incomplete in two ways. > > 1. > The migration never happened for files. FileTransferOperator copies > between any two registered filesystems, but the 28 pair operators were > never rebased on it or deprecated. They are still what users find first, > and what we keep fixing one by one. > 2. > There is no contract for sources that produce records instead of files: > paginated HTTP/REST and SaaS APIs, and SQL when the destination is a file. > AIP-14 mentions databases in one line and AIP-58 did not cover them. This > is where new pair operators keep coming from. In my own team the latest > request was an "MSGraphToS3Operator". > > The direction I would like to discuss, not a design yet: > > * > Reading in batches, pagination included, is a capability of the hook, as > get_df_by_chunks is for SQL. #75649 [7] proposes the same for MS Graph. > * > One generic operator connects a source hook to a destination hook. > GenericTransfer looks like the natural starting point: it already takes > source_hook_params and destination_hook_params for the two hooks and > insert_args for the destination call. With the same on the source side, the > arguments of each side are forwarded to that side's hook and checked when > the task runs. The operator knows neither vocabulary. > * > File destinations go through ObjectStoragePath, so every registered > filesystem works. > * > The existing pair operators become thin shells and are deprecated on the > usual schedule, as the SQL operators were. > * > Out of scope here: Aldo is preparing a dedicated proposal on optimising > GenericTransfer between ADBC-enabled hooks, moving Arrow record batches end > to end instead of Python rows. That proposal stands on its own and I do not > want to mix the two threads. It fits this direction well, since it also > puts the capability in the hooks and keeps the operator generic. > > My questions to the list: > > 1. > Do we agree that the per-pair operators are a maintenance problem worth > solving, and should we stop accepting new ones in the meantime? > 2. > Can step one happen without any new design: rebase the file-to-file > operators on ObjectStoragePath and deprecate them in favour of > FileTransferOperator? > 3. > For record sources, where should the contract and the operator live: > common.sql next to GenericTransfer, common.io next to > FileTransferOperator, or somewhere new? > 4. > Does this need a new AIP? I think it does. AIP-14 is seven years old and > closed, so a new one would reference it and cover the part that was never > implemented. > > I also remember TP presenting something similar at Airflow Summit last > year, I believe in "Assets: Past, Present, Future" [8], where Connection, > Object Storage and Asset were tied closer together. TP, please correct me > if I remember it wrong; I am especially interested in how this would fit > with what you have in mind. > > I am willing to drive this: write the AIP if the list agrees one is > needed, and implement it. It touches many providers though, so everyone who > wants to contribute or help is very welcome. The author of the four > folder-marker fixes has already offered to join [1]. > > Best regards, > David > > [1] https://github.com/apache/airflow/pull/76408 > [2] https://github.com/apache/airflow/pull/65724 > [3] https://github.com/apache/airflow/pull/72497 > [4] https://github.com/apache/airflow/pull/72892 > [5] > https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/103091837/AIP-14+Create+composable+operators+using+common+interfaces > [6] > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263430565 > [7] https://github.com/apache/airflow/pull/75649 > [8] https://airflowsummit.org/sessions/2025/assets-past-present-future/ > > General (Internal Property) >
