"You could calculate locations rather easily .. chunk = n / chunksize then offset = a modulus n % chunksize. This would allow you to continue adding items without ever having full copy operations. I think someone has already mentioned a similar strategy but this is the way to go. If you ran out of initial chunks, you could easily resize this array (copying 1000-2000 items as opposed to 10000000) "
"As list grows, the inner list that contains the byte arrays will get copied, but the underlying bytes will not. Arrays are objects, not structs, so a copy is always a copy of the pointer. So even as the list grows, the copy operation will still be very quick, as the arrays are copied, not the bytes in the array." Thanks for the clarification but I believe we just said the exact same thing. Once the point of chunking is taken I believe the rest of this discussion is of no consequence with the possible exception of how to manage the last chunk which may not be filled. You will receive almost identical performance characteristics using a List<byte[]>, an ArrayList, or growing a byte [] on your own. My personal opinion is the List<> becuase it is the strongest typed and handles growth operations for me but performance wise they are all just about identical (within a few %) Cheers, Greg On 7/6/06, Erick Thompson <[EMAIL PROTECTED]> wrote:
Right, I understand, but List<x> copies the inner list on a grow, but does not do a deep copy. List<byte[]> list = new List<byte[]>(); As list grows, the inner list that contains the byte arrays will get copied, but the underlying bytes will not. Arrays are objects, not structs, so a copy is always a copy of the pointer. So even as the list grows, the copy operation will still be very quick, as the arrays are copied, not the bytes in the array. Erick > -----Original Message----- > From: Discussion of advanced .NET topics. > [mailto:ADVANCED-DOTNET@DISCUSS.DEVELOP.COM ] On Behalf Of Stoyan Damov > Sent: Thursday, July 06, 2006 5:12 PM > To: ADVANCED-DOTNET@DISCUSS.DEVELOP.COM > Subject: Re: [ADVANCED-DOTNET] trim byte array > > Gregory was talking about the implementation details of > List<> which copies the existing items when it grows. On a > side note, List<*byte[]> is an invalid construct and won't > even compile. > > On 7/7/06, Erick Thompson <[EMAIL PROTECTED]> wrote: > > If you have an List<byte[]> and the list needs to expand, > it should only copy the pointer to the byte[] object, which > is cheap. I see no reason why it would do a deep copy. If it > does do a deep copy, make the pointer explicit, e.g., > List<*byte[]>. However, I don't believe you need to do this. > > > > Erick > > > > > -----Original Message----- > > > From: Discussion of advanced .NET topics. > > > [mailto:ADVANCED-DOTNET@DISCUSS.DEVELOP.COM ] On Behalf Of gregory > > > young > > > Sent: Thursday, July 06, 2006 2:50 PM > > > To: ADVANCED-DOTNET@DISCUSS.DEVELOP.COM > > > Subject: Re: [ADVANCED-DOTNET] trim byte array > > > > > > An arraylist containing bytes would have far worse > performance (it > > > also does doubling internally except now you would be > dealing with > > > references instead of of single bytes). > > > A List<byte> also will do doubling internally (if you > don't set it > > > large enough initially) but will do it dealing with a byte [] > > > internally (a memorystream will work in a similar > fashion). If you > > > require fast random access this would probably be the fastest. > > > Another problem will exist in that you would be dealing > with large > > > objects (they get garbage colelcted differently) and this can > > > actually open a DOS attack. > > > > > > The best long term solution is probably to make smaller > byte arrays > > > and link them together. This would be extremely good if > you are not > > > changing data but simply appending to it. So instead of > managing an > > > array that is 100 mb you hand 2000 smaller arrays. > > > > > > Think for a minute about having an array chunks .. each > chunk is n > > > bytes so ... > > > > > > byte [][] chunks > > > > > > You could calculate locations rather easily .. chunk = n > / chunksize > > > then offset = a modulus n % chunksize. This would allow you to > > > continue adding items without ever having full copy operations. I > > > think someone has already mentioned a similar strategy > but this is > > > the way to go. If you ran out of initial chunks, you could easily > > > resize this array (copying 1000-2000 items as opposed to 10000000) > > > > > > Cheers, > > > > > > Greg > > > > > > > > > On 7/6/06, dave wanta <[EMAIL PROTECTED]> wrote: > > > > > > > > Hi Erick, > > > > hmm... this is interesting. > > > > > > > > I wouldn't have to worry about constantly expanding the byte > > > > array, and in this case I don't need to worry about > accessing any > > > > internal bytes, just the end of the overall byte array to check > > > > for > > > terminating > > > > characters. > > > > > > > > (thinking out loud) > > > > I wonder if even something quick and dirty of an > ArrayList of byte > > > > arrays would work? Simply as a quick prototype? > > > > > > > > Cheers! > > > > Dave > > > > > > > > > > > > ----- Original Message ----- > > > > From: "Erick Thompson" < [EMAIL PROTECTED]> > > > > To: <ADVANCED-DOTNET@DISCUSS.DEVELOP.COM> > > > > Sent: Thursday, July 06, 2006 3:26 PM > > > > Subject: Re: [ADVANCED-DOTNET] trim byte array > > > > > > > > > > > > > It's the copy that gets pricey, which is going to be hard > > > to avoid > > > > > in > > > > this > > > > situation. A list just won't cut it with this size. I would > > > suggest a > > > > new class that uses a list of byte[], and then uses the > > > length of the > > > > internal byte array to determine the right array to > touch in the > > > > indexer. I have to head out of the office, but I could > > > throw together > > > > some code later if you need me to. It makes random access > > > slower, but > > > > then you don't need to ever copy the data until the final step > > > > when you spit out the array, which should speed it up a lot. > > > > > > > > > > Erick > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > From: Discussion of advanced .NET topics. > > > > > > [mailto:[EMAIL PROTECTED] On Behalf > > > Of gregory > > > > > > young > > > > > > Sent: Thursday, July 06, 2006 1:00 PM > > > > > > To: ADVANCED-DOTNET@DISCUSS.DEVELOP.COM > > > > > > Subject: Re: [ADVANCED-DOTNET] trim byte array > > > > > > > > > > > > Dave this is how list<byte> would work as well ... you > > > can however > > > > > > set an initial size if you know it to avoid these doublings. > > > > > > > > > > > > Cheers, > > > > > > > > > > > > Greg > > > > > > > > > > > > > > > > > > On 7/6/06, dave wanta < [EMAIL PROTECTED]> wrote: > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > btw, just fyi...if you Reflector a MemoryStream, it > > > doubles it's > > > > > > > internal buffer every time it exceeds it's length. > > > > > > > > > > > > > > The MemoryStream is really clean from a programmatic > > > > > > standpoint, it's > > > > > > > just not the performance I really want. > > > > > > > > > > > > > > Thanks for all the suggestions guys! I really > appreciate it. > > > > > > > > > > > > > > Cheers! > > > > > > > Dave > > > > > > > > > > > > > > =================================== > > > > > > > This list is hosted by DevelopMentor > http://www.develop.com > > > > > > > > > > > > > > View archives and manage your subscription(s) at > > > > > > > http://discuss.develop.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > If knowledge can create problems, it is not through > > > ignorance that > > > > > > we can solve them. > > > > > > > > > > > > Isaac Asimov > > > > > > > > > > > > > > > > > > > =================================== > > > > This list is hosted by DevelopMentor http://www.develop.com > > > > > > > > View archives and manage your subscription(s) at > > > > http://discuss.develop.com > > > > > > > > > > > > > > > > -- > > > If knowledge can create problems, it is not through > ignorance that > > > we can solve them. > > > > > > Isaac Asimov > > > > > > > > -- > > Cheers, > Stoyan > > =================================== > This list is hosted by DevelopMentor(r) http://www.develop.com > > View archives and manage your subscription(s) at > http://discuss.develop.com > =================================== This list is hosted by DevelopMentor(r) http://www.develop.com View archives and manage your subscription(s) at http://discuss.develop.com
-- If knowledge can create problems, it is not through ignorance that we can solve them. Isaac Asimov =================================== This list is hosted by DevelopMentor® http://www.develop.com View archives and manage your subscription(s) at http://discuss.develop.com
