Hi Jan,
On 18.04.23 07:48, Jan Lahoda wrote:
I apologize for being contrarian, but since the index download started for
me (again) while on a bus with very poor internet connection, I guess I
should tell you my view.
Unless I am mistaken, the index gz has currently roughly 1.9GB, and it
tooks several minutes to actually create the Lucene index from it,
consuming some more space and CPU.
correct. However, this is the initial download and initial indexing.
Weekly index updates are usually ~8MB.
To be honest, it never seemed very polite to me to download and process so
much without asking.
I guess alternatives that I would see would include (combination of options
possible):
- explicitly ask before downloading (possibly allowing the user to select
auto-download)
+1 something we should do. First we had to add the actual setting
(#5646), NB 18 will be able to turn off remote indexing separately from
local .m2 scanning.
- have the features that use the index do some query on a server, if there
isn't a downloaded index (or if it is stale/obsolete)
see other mail. But yes I experimented with that (#4971).
- given that https://github.com/apache/netbeans/pull/4999 produces a
smaller index, we could have a download location (server) at least for
maven central that would serve this optimized index. If I understand it
properly, the smallest index under that PR is 0.8GB, and if it would
compress reasonably well, it might be (say) 0.5GB compressed - much better
than 1.9GB, and no significant CPU usage after the index is downloaded.
(Even if it was 0.8GB, it is still much better than 1.9GB+CPU churn.)
I personally don't care about the weekly index update at all - I barely
notice it when it updates once per week. However I am always open to
optimize it!
But keep in mind that remote index downloads work for more repos than
just central, e.g apache has one too.
If you suggest that we should index repos server-side and provide a
compressed index as a service for NB I wouldn't have anything against it
in principle, but I also don't want to run this service in my spare time.
We could in theory build it in CI and bundle the index with NB. However,
remember that this might annoy users which don't use java/maven. NB will
soon support rust and is popular for PHP etc.
regarding optimizations:
one step which runs unreasonably slow ATM is the index update. The delta
download is always below 10MB for central, while the index merge still
takes minutes. This is something I want to check once #4999 is merged.
best regards,
michael
There was also an argument on conserving the ASF resources in another
discussion recently. If I consider there would be (only) 10 000
installations of NetBeans, with the default setting to download the index
once a week, it is almost 20TB of data every week if I count correctly.
+the CPU cycles to convert the index on user's machines. It seems there may
be a way to conserve the ASF resources and provide better experience to the
users at the same time.
Jan
On Thu, Apr 13, 2023 at 7:38 PM Michael Bien <mbie...@gmail.com> wrote:
thanks Matthias,
ok lets keep it on :)
I never really had any issues with it either.
the thought was to switch it off in NB 18 and switch it on again in NB
19 once we are able to bump the dependencies - not as permanent new
default.
best regards,
michael
On 13.04.23 19:18, Matthias Bläsing wrote:
Hi,
I don't think this is a good idea. If your disc is to small to hold the
maven index there is a pretty good chance, that your setup is often
broken.
Quite frankly at work I never noticed NetBeans downloading the index,
it is lost in the noice of normal operation.
We will basicly loose all advantages of maven indexing. My local
repository does not hold the latested artifact (rendering the
corresponding hint useless) and completion is interesting because I did
not yet use the artifact. Else I most probably have the right setup for
the setup already at hand.
The option to disable is now there.
Given that part of the discussion around JDK 11 baseline was to make a
faster maven indexer available gives this another bad taste.
So from my POV this should not be done.
Greetings
Matthias
Am Donnerstag, dem 13.04.2023 um 18:47 +0200 schrieb Michael Bien:
Hello devs,
NB 18 will split the indexer setting into two check boxes, one for local
indexing, one for remote index extraction,
see screenshot: https://github.com/apache/netbeans/pull/5646
I propose to disable remote index extraction by default, until we solved
some more of the issues at least.
main complains are:
- failing extraction when temp folder is too small e.g #5815 (NB 18
will deactivate indexing on extraction failure)
- extraction takes too long e.g #5809
Local index scanning scans your .m2 repo, which is fast and might be
sufficient for many use cases and therefore a better default for now.
thoughts?
-mbien
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@netbeans.apache.org
For additional commands, e-mail: dev-h...@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@netbeans.apache.org
For additional commands, e-mail: dev-h...@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@netbeans.apache.org
For additional commands, e-mail: dev-h...@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@netbeans.apache.org
For additional commands, e-mail: dev-h...@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
