goutamadwant commented on issue #11306: URL: https://github.com/apache/seatunnel/issues/11306#issuecomment-4912206889
@davidzollo @DanielLeens Opened #11377 for this. I Kept the first pass scoped to the SftpFile submit-time path. For SftpFile binary sources in once discovery mode, the config creation no longer does the full file listing. The file discovery now runs when the split enumerator opens, after the job has been created. I left the other file formats on the existing eager path since some of them still need discovered files for schema inference. Also added timing log around file discovery so the production-shaped config/logs can verify where time is spent. Tested with the focused split enumerator test, connector-file-base package, and connector-file-sftp package with `-am`. Let me know if any suggestions! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
