AngryHelper opened a new pull request #16792: URL: https://github.com/apache/airflow/pull/16792
closes: AIRFLOW-5703 Added next features: transfer list of files, transfer directory, transfer files on regex pattern in directories.  <details> <summary>I writen next dag for tests</summary> ``` from airflow.utils.edgemodifier import Label from airflow.operators.bash import BashOperator from airflow import DAG from airflow.providers.sftp.operators.sftp import SFTPOperator, SFTPOperation from airflow.providers.ssh.operators.ssh import SSHOperator from airflow.utils.dates import days_ago with DAG( dag_id="test_stfp", schedule_interval="@once", start_date=days_ago(1), max_active_runs=1 ) as dag: create_test_files = BashOperator( task_id="create_test_files", bash_command=""" touch /tmp/file.txt & echo "transfer data from local" >> /tmp/file.txt & touch /tmp/file1.txt & echo "transfer data from local" >> /tmp/file1.txt & touch /tmp/file2.txt & echo "transfer data from local" >> /tmp/file2.txt & touch /tmp/file3.txt & echo "transfer data from local" >> /tmp/file3.txt & mkdir /tmp/dir_for_remote_transfer & touch /tmp/dir_for_remote_transfer/file2.txt & touch /tmp/dir_for_remote_transfer/file3.csv & ls /tmp """ ) put_file = SFTPOperator( task_id="put_file", ssh_conn_id="ssh_default", local_filepath="/tmp/file.txt", remote_filepath="/tmp/tmp1/tmp2/file.txt", operation=SFTPOperation.PUT, create_intermediate_dirs=True ) put_file_without_full_remote_path = SFTPOperator( task_id="put_file_without_full_remote_path", ssh_conn_id="ssh_default", local_filepath="/tmp/file3.txt", remote_filepath="/tmp/tmp1/tmp2/", operation=SFTPOperation.PUT, create_intermediate_dirs=True ) put_dir_txt_files = SFTPOperator( task_id="put_dir_txt_files", ssh_conn_id="ssh_default", local_folder="/tmp/dir_for_remote_transfer/", remote_folder="/tmp/dir_for_remote_transfer/txt", regexp_mask=".*.txt", operation=SFTPOperation.PUT, create_intermediate_dirs=True ) put_dir_csv_files = SFTPOperator( task_id="put_dir_csv_files", ssh_conn_id="ssh_default", local_folder="/tmp/dir_for_remote_transfer/", remote_folder="/tmp/dir_for_remote_transfer/csv/", regexp_mask=".*.csv", operation=SFTPOperation.PUT, create_intermediate_dirs=True ) put_files = SFTPOperator( task_id="put_files", ssh_conn_id="ssh_default", local_filepath=["/tmp/file1.txt", "/tmp/file2.txt"], remote_filepath="/tmp/tmp1/tmp2/", operation=SFTPOperation.PUT, create_intermediate_dirs=True ) drop_local_test_files = BashOperator( task_id="drop_local_test_files", bash_command="""rm -rf /tmp/file* & rm -rf /tmp/dir_for_remote_transfer/ & ls /tmp/""" ) get_file = SFTPOperator( task_id="get_file", ssh_conn_id="ssh_default", local_filepath="/tmp/file.txt", remote_filepath="/tmp/tmp1/tmp2/file.txt", operation=SFTPOperation.GET, create_intermediate_dirs=True ) get_files = SFTPOperator( task_id="get_files", ssh_conn_id="ssh_default", local_filepath="/tmp/", remote_filepath=["/tmp/tmp1/tmp2/file1.txt", "/tmp/tmp1/tmp2/file2.txt"], operation=SFTPOperation.GET, create_intermediate_dirs=True ) get_dir = SFTPOperator( task_id="get_dir", ssh_conn_id="ssh_default", local_folder="/tmp/dir_for_remote_transfer/csv", remote_folder="/tmp/dir_for_remote_transfer/csv", operation=SFTPOperation.GET, create_intermediate_dirs=True ) drop_remote_files = SSHOperator( task_id="drop_remote_files", ssh_conn_id="ssh_default", command="rm -rf /tmp/tmp1 & ls /tmp/" ) create_test_files >> Label("transfer data to remote server") >> \ [put_file, put_files, put_file_without_full_remote_path, put_dir_txt_files, put_dir_csv_files] >> \ drop_local_test_files >> Label("transfer data to local server") >> \ [get_file, get_files, get_dir] >> drop_remote_files ``` </details> <details> <summary>Pre-commit Tests</summary> ``` ubuntu@ubuntu-advanced-8-16-10gb:~/airflow$ pre-commit run --files airflow/providers/sftp/operators/sftp.py identity.................................................................................Passed - hook id: identity - duration: 0.04s airflow/providers/sftp/operators/sftp.py Check hooks apply to the repository..................................(no files to check)Skipped No-tabs checker..........................................................................Passed Add license for all SQL files........................................(no files to check)Skipped Add license for all other files......................................(no files to check)Skipped Add license for all rst files........................................(no files to check)Skipped Add license for all JS/TS/TSX/CSS/PUML files.........................(no files to check)Skipped Add license for all JINJA template files.............................(no files to check)Skipped Add license for all shell files......................................(no files to check)Skipped Add license for all Python files.........................................................Passed Add license for all XML files........................................(no files to check)Skipped Add license for all YAML files.......................................(no files to check)Skipped Add license for all md files.........................................(no files to check)Skipped Add license for all mermaid files....................................(no files to check)Skipped Add TOC for md files.................................................(no files to check)Skipped black....................................................................................Passed blacken-docs.............................................................................Passed Check for merge conflicts................................................................Passed Debug Statements (Python)................................................................Passed Check builtin type constructor use.......................................................Passed Detect Private Key.......................................................................Passed Fix End of Files.........................................................................Passed Mixed line ending........................................................................Passed Check that executables have shebangs.................................(no files to check)Skipped Check Xml............................................................(no files to check)Skipped Trim Trailing Whitespace.................................................................Passed Fix python encoding pragma...............................................................Passed Pretty format JSON...................................................(no files to check)Skipped pyupgrade................................................................................Passed rst ``code`` is two backticks........................................(no files to check)Skipped use logger.warning(......................................................................Passed Check YAML files with yamllint.......................................(no files to check)Skipped Run isort to sort imports in Python files................................................Passed Run pydocstyle...........................................................................Passed Strip unnecessary `# noqa`s..............................................................Passed Lint OpenAPI using spectral..........................................(no files to check)Skipped Lint OpenAPI using openapi-spec-validator............................(no files to check)Skipped Lint dockerfile......................................................(no files to check)Skipped Check order of dependencies in setup.cfg and setup.py................(no files to check)Skipped Checks setup extra packages..........................................(no files to check)Skipped Update output of breeze command in BREEZE.rst........................(no files to check)Skipped Update mounts in the local yml file..................................(no files to check)Skipped Update setup.cfg file with all licenses..............................(no files to check)Skipped Build cross-dependencies for providers packages..........................................Passed Update extras in documentation.......................................(no files to check)Skipped Check for pydevd debug statements accidentally left......................................Passed Don't use safe in templates..........................................(no files to check)Skipped No providers imports in core example DAGs............................(no files to check)Skipped No relative imports......................................................................Passed Check for language that we do not accept as community....................................Passed Check BaseOperator[Link] core imports................................(no files to check)Skipped Check BaseOperator[Link] other imports...................................................Passed Check provide_session and create_session imports.........................................Passed Make sure LoggingMixin is not used alone.................................................Passed Make sure days_ago is imported from airflow.utils.dates..................................Passed 'start_date' should not be defined in default_args in example_dags...(no files to check)Skipped Check if integration list is aligned.................................(no files to check)Skipped Check if licenses are OK for Apache..................................(no files to check)Skipped Checks for consistency between config.yml and default_config.cfg.....(no files to check)Skipped Verify that DB Migrations have been documented.......................(no files to check)Skipped Checks for Boring Cyborg configuration consistency.......................................Passed Sort INTHEWILD.md alphabetically.....................................(no files to check)Skipped Sort alphabetically and uniquify spelling_wordlist.txt...............(no files to check)Skipped Lint Helm Chart......................................................(no files to check)Skipped Check Shell scripts syntax correctness...............................(no files to check)Skipped Run BATS bash tests for changed Breeze bash files....................(no files to check)Skipped Run BATS bash tests for changed bash files...........................(no files to check)Skipped stylelint............................................................(no files to check)Skipped Provider init file.......................................................................Passed Validate providers.yaml files........................................(no files to check)Skipped Generate mermaid images..............................................(no files to check)Skipped Check if pre-commits are described...................................(no files to check)Skipped Ensure hook ids are not overly long..................................(no files to check)Skipped Checks providers available when declared by extras in setup.py...........................Passed Run markdownlint.....................................................(no files to check)Skipped Lint JSON Schema files with JSON Schema..............................(no files to check)Skipped Lint NodePort Service with JSON Schema...............................(no files to check)Skipped Lint Docker compose files with JSON Schema...........................(no files to check)Skipped Lint chart/values.schema.json file with JSON Schema..................(no files to check)Skipped Lint chart/values.yaml file with JSON Schema.........................(no files to check)Skipped Lint airflow/config_templates/config.yml file with JSON Schema.......(no files to check)Skipped Convert to f-strings with flynt..........................................................Passed ESLint against airflow/ui............................................(no files to check)Skipped ESLint against current UI js files...................................(no files to check)Skipped Check if image build is needed...........................................................Passed Run in container bats tests..........................................(no files to check)Skipped Run mypy.................................................................................Passed Run mypy for helm chart tests........................................(no files to check)Skipped Run mypy for /docs/ folder...........................................(no files to check)Skipped Run flake8...............................................................................Passed ``` </details> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
