[
https://issues.apache.org/jira/browse/WAGON-537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16685265#comment-16685265
]
Olaf Otto commented on WAGON-537:
---------------------------------
Hi [~michael-o]
I agree that it would be great if the progress monitor only made an update if a
decimal changes. The only potential issue I see is that if a transfer is slow,
users may notice no activity on the console and may believe maven is "hanging".
>From an architectural perspective, I would personally not want to create any
>dependency between TransferListeners and the transfer loop. The loop should
>simply make sure not to invoke the listeners overly frequently as this slows
>down transfers (the main objective of this change), and the TransferListeners
>should do whatever they want with the data.
Perhaps one could discuss whether TransferListeners must be invoked
synchronously during data transfer. For me, the term "Listener" implies
asynchronism. Are there cases where a listener actively aborts transfer by
throwing an exception, perhaps something related to checksums? If not, one
could e.g. put progress notifications on a queue and asynchronously notify
listeners.
> Maven transfer speed of large artifacts is slow due to unsuitable buffer
> strategy
> ---------------------------------------------------------------------------------
>
> Key: WAGON-537
> URL: https://issues.apache.org/jira/browse/WAGON-537
> Project: Maven Wagon
> Issue Type: Improvement
> Components: wagon-http, wagon-provider-api
> Affects Versions: 3.2.0
> Environment: Windows 10, JDK 1.8, Nexus Artifact store > 100MB/s
> network connection.
> Reporter: Olaf Otto
> Assignee: Michael Osipov
> Priority: Major
> Labels: perfomance
> Fix For: 3.2.1
>
> Attachments: wagon-issue.png
>
>
> We are using maven for build process automation with docker. This sometimes
> involves uploading and downloading artifacts with a few gigabytes in size.
> Here, maven's transfer speed is consistently and reproducibly slow. For
> instance, an artifact with 7,5 GB in size took almost two hours to transfer
> in spite of a 100 MB/s connection with respective reproducible download speed
> from the remote nexus artifact repository when using a browser to download.
> The same is true when uploding such an artifact.
> I have investigated the issue using JProfiler. The result shows an issue in
> AbstractWagon's transfer( Resource resource, InputStream input, OutputStream
> output, int requestType, long maxSize ) method used for remote artifacts and
> the same issue in AbstractHttpClientWagon#writeTo(OutputStream).
> Here, the input stream is read in a loop using a 4 Kb buffer. Whenever data
> is received, the received data is pushed to downstream listeners via
> fireTransferProgress. These listeners (or rather consumers) perform expensive
> tasks.
> Now, the underlying InputStream implementation used in transfer will return
> calls to read(buffer, offset, length) as soon as *some* data is available.
> That is, fireTransferProgress may well be invoked with an average number of
> bytes less than half the buffer capacity (this varies with the underlying
> network and hardware architecture). Consequently, fireTransferProgress is
> invoked *millions of times* for large files. As this is a blocking operation,
> the time spent in fireTransferProgress dominates and drastically slows down
> the transfers by at least one order of magnitude.
> !wagon-issue.png!
> In our case, we found download speed reduced from a theoretical optimum of
> ~80 seconds to to more than 3200 seconds.
> From an architectural perspective, I would not want to make the consumers /
> listeners invoked via fireTransferProgress aware of their potential impact on
> download speed, but rather refactor the transfer method such that it uses a
> buffer strategy reducing the the number of fireTransferProgress invocations.
> This should be done with regard to the expected file size of the transfer,
> such that fireTransferProgress is invoked often enough but not to frequent.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
