[
https://issues.apache.org/jira/browse/CAMEL-26635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Federico Mariani resolved CAMEL-26635.
--------------------------------------
Resolution: Fixed
> camel-mina-sftp, camel-sftp - stop listing the directory once
> maxMessagesPerPoll files are accepted
> ---------------------------------------------------------------------------------------------------
>
> Key: CAMEL-26635
> URL: https://issues.apache.org/jira/browse/CAMEL-26635
> Project: Camel
> Issue Type: Improvement
> Components: camel-mina-sftp, camel-sftp
> Reporter: Federico Mariani
> Assignee: Federico Mariani
> Priority: Major
>
> With {{maxMessagesPerPoll}} set (and {{eagerMaxMessagesPerPoll=true}}, the
> default), the SFTP consumers still list the whole directory on every poll,
> and only then stop picking files:
> * {{AbstractSftpConsumer.doPollDirectory}} gets the full array from
> {{operations.listFiles(dir)}}; {{canPollMoreFiles}} only stops the loop over
> it.
> * {{MinaSftpOperations.listFiles}} drains {{sftpClient.readDir(path)}};
> {{SftpOperations.listFiles}} calls {{channel.ls(path)}}.
> The option description says "If eager then the limit is during the scanning
> of files", so users expect the listing itself to stop at the limit.
> h3. Impact
> Measured with the embedded MINA SFTP server, 30,000 files,
> {{maxMessagesPerPoll=1000}}, {{stepwise=false}}:
> ||Listing||Entries read in one poll||READDIR requests||
> |today (whole directory)|30,002|145|
> |stop once 1,000 files are accepted|about 1,040|5|
> 30,002 is the files plus {{.}} and {{..}}; about 1,040 because the server
> sends entries in batches, so the last batch goes past 1,000.
> With {{delete=true}} and {{greedy=true}}, every poll re-reads what is left:
> draining 25,000 files reads about 325,000 entries (25k + 24k + ... + 1k)
> instead of about 25,000.
> A user hit this with camel-mina-sftp and camel-sftp on Azure Blob Storage
> SFTP (a folder of 25k+ files, {{maxMessagesPerPoll=1000}},
> {{stepwise=false}}): the listing gets cut, and Azure closes connections that
> are idle for two minutes. It is not known yet whether a shorter listing
> avoids the cut there (it depends on where Azure spends the time; keep-alive
> with {{serverAliveInterval}} is being tried first), but reading only what a
> poll needs is worth it regardless.
> h3. Proposal
> Stop reading the directory once {{maxMessagesPerPoll}} files are *accepted*
> (as counted by {{canPollMoreFiles}}, after include/exclude, filters,
> idempotent and in-progress checks), rather than capping the raw listing at
> {{maxMessagesPerPoll}} entries.
> A raw cap (for example {{listFiles(path, maxMessages)}}) starves when
> rejected entries stay in the directory, because they keep filling the cap:
> files not matching {{include}}, files already in the idempotent repository,
> {{noop=true}}. Reproduced: with {{noop=true}}, 300 files and
> {{maxMessagesPerPoll=100}}, a raw cap consumes 98 files and then nothing more
> ({{.}} and {{..}} count against the cap too); stopping on accepted files
> consumes all 300.
> No new option: use it whenever the outcome is the same as today, and keep the
> full listing otherwise:
> * {{maxMessagesPerPoll > 0}} and {{eagerMaxMessagesPerPoll=true}}
> * no {{preSort}} (it sorts the whole listing, CAMEL-23931)
> * no {{doneFileName}} ({{isMatched}} looks the done file up in the listing;
> see also CAMEL-25728 and CAMEL-25772)
> * both {{stepwise=true}} (the default) and {{stepwise=false}}
> Nothing is kept between polls: each poll lists from the start, as today. With
> {{delete}} or {{move}} a poll reads about {{maxMessagesPerPoll}} entries;
> with {{noop}} or idempotent it reads further on each poll, up to the whole
> directory.
> h3. Implementation notes
> * camel-mina-sftp: use {{sftpClient.openDir(path)}} and {{listDir(handle)}}
> in try-with-resources. {{readDir(path)}} closes the handle only at the end of
> the directory, so leaving a loop over it early leaves the handle open.
> * camel-sftp (JSch): {{ChannelSftp.ls(path, LsEntrySelector)}} can stop with
> {{BREAK}}, but an SFTP request sent from inside the selector corrupts the
> channel (reproduced: a {{stat}} in the selector makes {{ls}} fail with
> {{ArrayIndexOutOfBoundsException}}). The consumer sends requests while it
> walks the listing (resolving symbolic links, recursion), so camel-sftp needs
> either a two-step approach or only the cases that send none. Suggest doing
> camel-mina-sftp first.
> * Test: count the READDIR requests on the embedded server (with an
> {{SftpEventListener}}) instead of measuring time.
> h3. Same pattern elsewhere (not in scope)
> * camel-smb: {{SmbOperations.listFiles}} returns {{share.list(path,
> searchPattern)}}, the whole directory; smbj 0.15.0 also offers a lazy
> {{Directory.iterator(...)}}.
> * camel-file: {{File.listFiles()}} reads the whole directory, while
> {{Files.newDirectoryStream}} is lazy. Mostly matters on network file systems.
> * camel-ftp: {{LIST}} is a single data transfer, so there is little to gain.
> _Claude Code on behalf of Federico Mariani_
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
