> > And in the case of hundreds of megs of data > > ... and I would be surprised at someone that would embed hundreds of > megs of data into an object such that it had to be serialized... seems > like the proper design is to point at the data, or a subset of it, in a > big buffer. Then data transfers would just transfer the offset/length > and the reference to the buffer. > > > and/or thousands of data structure instances, > > ... and this is another surprise! You have thousands of objects (data > structure instances) to move from one thread to another? >
I think we miscommunicated there--I'm actually agreeing with you. I was trying to make the same point you were: that intricate and/or large structures are meant to be passed around by a top-level pointer, not using and serialization/messaging. This is what I've been trying to explain to others here; that IPC and shared memory unfortunately aren't viable options, leaving app threads (rather than child processes) as the solution. > Of course, I know that data get large, but typical multimedia streams > are large, binary blobs. I was under the impression that processing > them usually proceeds along the lines of keeping offsets into the blobs, > and interpreting, etc. Editing is usually done by making a copy of a > blob, transforming it or a subset in some manner during the copy > process, resulting in a new, possibly different-sized blob. Your instincts are right. I'd only add on that when you're talking about data structures associated with an intricate video format, the complexity and depth of the data structures is insane -- the LAST thing you want to burn cycles on is serializing and unserializing that stuff (so IPC is out)--again, we're already on the same page here. I think at one point you made the comment that shared memory is a solution to handle large data sets between a child process and the parent. Although this is certainty true in principle, it doesn't hold up in practice since complex data structures often contain 3rd party and OS API objects that have their own allocators. For example, in video encoding, there's TONS of objects that comprise memory-resident video from all kinds of APIs, so the idea of having them allocated from shared/mapped memory block isn't even possible. Again, I only raise this to offer evidence that doing real-world work in a child process is a deal breaker--a shared address space is just way too much to give up. Andy -- http://mail.python.org/mailman/listinfo/python-list
