mattcasters opened a new issue, #8719: URL: https://github.com/apache/hop/issues/8719
### What needs to happen? #8671 introduces shared pipelines under `integration-tests/vfs` and runs them from `integration-tests/smb` against a Samba container. Those pipelines are the generic file operations a named VFS connection is expected to support: - create a folder - write a text file and read it back - append a line and read both lines back - list the folder - copy a file - move a file - delete a file - delete a folder, then create it again - a rooted connection, where the metadata has a base folder `hop-tech-smb` is the first project on that suite. This task is to run the same pipelines from the other VFS driver projects. Each operation stays its own `main-*.hwf`, so a failure names the operation. The driver project keeps its own metadata, environment variables, and compose service. Protocol-specific workflows stay in the driver project. Jenkins already runs `integration-tests/scripts/run-tests-docker.sh` with no project filter and publishes `integration-tests/**/surefire-reports/*.xml`. A container-backed driver needs `docker/integration-tests/integration-tests-<project>.yaml`, with a healthcheck on the service Hop waits for. Projects that need real credentials stay behind `disabled.txt`. ### Candidate plugins These already have an integration project. Point the generic file steps at `integration-tests/vfs`: - **hop-tech-minio** (`integration-tests/minio`, `main-0001-minio-file-handling`). SeaweedFS, named scheme `minio:///`. - **hop-tech-sftp** (`integration-tests/sftp`, `main-003-sftp-connection-in-regular-transforms`). Keep `main-001` (last-modified), `main-002` (SFTP Put), and `main-004` (SFTP actions). The SFTP service needs a healthcheck so the nightly job cannot hang on it. - **hop-tech-ftp** (`integration-tests/ftp`, `main-003-ftp-connection-in-regular-transforms`). Keep `main-001` (FTP Put), `main-002` (FTP connection), and `main-004` (FTP action round trip). FTPS is the same plugin (`FtpsVfsPlugin`) and has no project yet. - **hop-tech-hadoop** (`integration-tests/hadoop`, WebHDFS text handling in `main-0001-hdfs-file-handling`). Keep the Parquet, gzip, copies, and HA coverage. - **hop-tech-azure** (`integration-tests/azure`, behind `disabled.txt`). `main-0003` move, `main-0004` copy, `main-0007` file manipulation, and `main-0008` create/fill/delete folder, for `azure:///` and `azfs:///`. Keep the Parquet and multi-account workflows. Runs when Azure credentials are present. - **hop-tech-google**, Google Storage (`integration-tests/gcp`, behind `disabled.txt`). The project is Beam and BigQuery, plus `main-0008-create-delete-file` on `gs://`. Add the shared file operations when a service account is present, and keep Beam, BigQuery, and the format-specific output. Google Drive (`GoogleDriveVfsPlugin`, scheme `googledrive`) is the same module and has no integration project. These plugins have no generic file-operation project yet: - **hop-tech-aws**, S3 (`S3VfsPlugin`) - **hop-tech-dropbox** - **hop-tech-webdav**, WebDAV (`webdav4`) and WebDAV over HTTPS (`webdav4s`) - **hop-tech-databricks**, Databricks Volumes and workspace files The Git VFS connection from #8663 should use the same pipelines once that plugin is on main. ### Issue Priority Priority: 3 ### Issue Component Component: Integration Testing, Component: VFS -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
