DanielCarter-stack commented on issue #10504: URL: https://github.com/apache/seatunnel/issues/10504#issuecomment-3940686033
<!-- code-pr-reviewer --> Thanks for proposing this enhancement. This is indeed a valuable feature that would improve flexibility across all file-based connectors. **Current state**: `AbstractReadStrategy.collectFileNamesByPath()` (lines 172-177) unconditionally forces recursive directory scanning with no way to disable it. **Proposed path forward**: 1. Add `recursive_file_scan` to `FileBaseSourceOptions.java` (default `true` for backward compatibility) 2. Update `AbstractReadStrategy.collectFileNamesByPath()` to respect this flag 3. All ~11 file connectors (LocalFile, S3, HDFS, FTP, SFTP, etc.) will inherit this behavior automatically **Example behavior**: ``` /test/file.txt /test/subdir/file2.txt ``` With `path = "/test"` and `recursive_file_scan = false`, only `file.txt` should be processed. Do you also need a `max_recursion_depth` parameter, or is the boolean flag sufficient? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
