On Thursday 22 January 2009 03:58, Dennis Nezic wrote:
> > > My main point in my last post was a suggestion to have the error
> > > message more informative. As another example, have it output it's
> > > memory/cpu usage before it shuts itself down, in the case of the
> > > deadlock I mentioned.
> > 
> > How do we get CPU usage from java? We can say how much memory is in
> > use, how many threads are running, get a thread dump...
> 
> Well--if the node knows enough to say that it is in a deadlock, and if
> it still has enough control over itself to be able to shut itself down
> cleanly, surely there is something it can do to investigate itself
> before doing so? Currently the messages do not appear to be helping us
> at all. Before shutting itself down in such deadlocks/freezes, it should
> at least output a thread-dump, and it's memory stats, if not a
> deeper/clearer analysis of what in particular, within MessageCore or
> PacketSender, is causing the problem.

I agree we could do a stack dump, I thought we did.
> 
> > > Also, why is there such a high requirement?? Why on earth is 100MB
> > > memory not enough? If it can't allocate any more memory, it should
> > > wait or throttle itself. Restricting freenet to the latest
> > > unecessary super-computers is dumb. (It really should be developed
> > > on a 486.)
> > 
> > Because it has a lot of stuff to track.
> 
> :|. A megabyte is pretty big. A hundred megabytes is huge. How much
> data (CHKs) does it really need to hold in memory. Though we're
> digressing a little since I don't believe memory is a problem with me
> in this particular case. (I have it set to 200MB now.)

We do not hold CHKs in memory except when we are transferring data. However, 
we do need to track a vast number of them (even with a block size of 32K).
> 
> > People propose rewriting Freenet in kernel-mode C with 1KB blocks
> > every so often, as an example. That means 32X more disk seeks, 32X
> > bigger bloom filter, and so on; it's not feasible.
> 
> Why 1K blocks? (A c implementation would be great though! :)

IMHO it would not achieve the order of magnitude improvement that everyone 
assumes it would. To take an analogy, a number field sieve is vastly faster 
than trying to factor every even number, even if the former is coded in java 
and the latter in assembler. "It's the algorithms, stupid!". Also java *does* 
have some significant performance advantages in some areas (memory 
management, function calls), as well as disadvantages in other areas.
> 
> > Memory requirements depend on two things:
> > - The datastore. The bdbje datastore uses a significant amount of
> > memory, with significant churn, inside the JVM's allocated space; the
> > salted hash datastore uses very little memory inside the memory
> > limits but uses 1/2048th of the store outside of the limits as a
> > memory mapped bloom filter to limit I/O.
> > - Client layer activity. Lots of large downloads use lots of memory,
> > uploads use even more (because of inefficient architecture).
> 
> Why are large file transfers worse than small file transfers? (And why
> are uploads any different than downloads, aside from the fact that we
> have to initially package the file for distribution.) Aren't transfers
> done in segments? For example, why can't it work on transferring 10MB
> segments at a time--that is, restricting it's memory usage per file
> transfer to 10MB (or whatever similarly effective yet small size)--and
> thus treating 2000MB files the same as 20MB files... both sailing along
> 10MB at a time?

Because that's not the most effective way to fetch blocks from the network! To 
some degree we do fetch segment-wise in the db4o branch as an optimisation, 
but we still need to track every key. Even on the db4o branch, we need to be 
able to quickly determine "Am I interested in key K?", if it is offered to 
us, if another node fetches it through us, etc. On the db4o branch we use 
bloom filters, approx 3.7MB per 10GB queued data; we keep a bloom filter in 
RAM for every download, and use that to quickly determine whether a block is 
of interest. This greatly reduces the number of database queries (= many disk 
seeks) we have to do, since the actual details of the downloads are not kept 
in RAM on the db4o branch.

With regards to uploads, the current code (including on the db4o branch) for 
uploads is somewhat less efficient than the code for downloads. It's 
something that I may or may not resolve before db4o is merged.
> 
> > > When is the db4o stuff expected to be released?
> > 
> > When I get around to it. :| The immediate todo:
> > - Release 1203
> > - Implement basic progress screen.
> > - Get db4o sorted and merged.
> > - Sort out plugins (IMHO important for 0.8).
> > - Auto-update (and update.* update) the seednodes file.
> > - Maybe new metadata (IMHO the assumptions underlying this item may
> > no longer be valid...)
> > > 
> > > > Seriously, EVERY time I have investigated these sorts of issues
> > > > the answer has been either that it is showing constant Full GC's
> > > > because it has slightly too little memory, or that there is
> > > > external CPU load. Are you absolutely completely totally
> > > > 100000000000000000000000000% sure that that is not the problem?
> > > > AFAICS there are two posters here, and just because one of them is
> > > > sure that the problem isn't memory doesn't necessarily mean that
> > > > the other one's problems are not due to memory??
> > > 
> > > There are reports on FMS of people with gigs of ram, and powerful
> > > machines, with crashing nodes. 
> > 
> > Just because they have lots of RAM doesn't mean they've set the
> > wrapper memory limit that high.
> 
> They did. Out of 2G, they set it to 500M. And it still kept crashing.
> Though, this was a few weeks ago. (Early January.)

With big download queues? Big datastores?
> 
> > > Though, I'm not sure if it's the same 
> > > problem, as my node hasn't really crashed this time--it just shut
> > > itself down. (Before I would get JVM hung errors, without any clean
> > > shut downs.)
> > 
> > This is closely related IMHO.
> 
> It feels that way to me too.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 827 bytes
Desc: not available
URL: 
<https://emu.freenetproject.org/pipermail/support/attachments/20090122/5c6ece78/attachment.pgp>

Reply via email to