Hi David, all, +1 on the direction, and I'd like to help build it.
Some context on why I'm interested. I use transfer operators day to day as a data engineer, and in some cases I've replaced Airbyte ingestions with them, writing Iceberg files from APIs or databases into object storage instead of flattened or Parquet files. I also ran into the drift you describe while working on #76533 [1], which adds pagination helpers and max_pages to HttpOperator. HttpOperator can paginate, but HttpToS3Operator and HttpToGCSOperator can't, so every pair would need the same fix again. Moving pagination into the hook fixes that once. To get evidence before arguing for a design, I put a small prototype in [2]. The earlier PRs there are the prototype itself: a hook-contract Protocol, a generic transfer operator, adapters and parity tests for the S3/GCS/Azure pairs, local emulators, and a scale bench. Two follow-ups are the evidence for this thread: - PR #8 [3] audits the 84 concrete transfer operators and runs the generic operator across all nine S3/GCS/Azure pairs on local emulators. 15 operators fit a generic shape as is, 43 fit once a few shared capabilities exist (compression, replace, move, ACL/extra args, encryption, wildcards), and 26 are genuinely pair-specific. The split matters more than the average. File and record pairs are mostly coverable (about 91%), while SQL-side pairs are mostly special cases (about 30%). On the nine pairs, 34 of 35 parity checks matched the existing operators, but only 4 of 249 upstream tests passed through a thin wrapper, mostly because they check exact hook calls or pair-specific options. - PR #9 [4] reproduces the one parity miss, which is a real bug: S3ToAzureBlobStorageOperator(replace=False) doesn't skip existing blobs under blob_prefix. I filed that as #76880 [5]. My takeaway on your questions: 1. Yes, I'd stop accepting new pair operators where a capability can live in the hook instead. 2. For file-to-file, I'd build on ObjectStoragePath/FileTransferOperator with shared helpers (replace, markers, move), and keep the pair operators as thin wrappers verified by a shared conformance suite, rather than deprecating them right away. 3. For record sources, I'd start with a small structural Protocol (a batch reader and a batch writer) in a neutral place, so hooks don't need to import a base class or raise their minimum versions. 4. I think an AIP makes sense once the file contract and conformance suite are agreed. Happy to start with the conformance suite for the object-store pairs if that sounds useful. Thanks, Aldo Escobar (aoelvp94) [1] https://github.com/apache/airflow/pull/76533 [2] https://github.com/aoelvp94/airflow-generic-transfer-contract-demo [3] https://github.com/aoelvp94/airflow-generic-transfer-contract-demo/pull/8 [4] https://github.com/aoelvp94/airflow-generic-transfer-contract-demo/pull/9 [5] https://github.com/apache/airflow/issues/76880 --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
