On 01/11/2017 01:55 AM, Tom Pantelis wrote:
> 
> 
> On Tue, Jan 10, 2017 at 7:08 PM, Robert Varga <n...@hq.sk
> <mailto:n...@hq.sk>> wrote:
> 
>     On 01/10/2017 09:20 PM, Robert Varga wrote:
>     > On 01/10/2017 08:16 PM, Tom Pantelis wrote:
>     >> Since the length field of an array is an int, it's constrained to
>     >> Integer.MAX_VALUE (~2G). To handle snapshots larger than 2G we'll have
>     >> to chuck it into multiple byte[].
>     > That could get funky on the reassembly side of things. If we go that
>     > route, we may as well switch to something more friendly, like
>     > java.nio.ByteBuffer(), too.
> 
>     Thinking about it a bit more, we really need some way of streaming the
>     snapshot.
> 
>     Tom, is there something in akka persistence which would allow us to feed
>     the data directly to an OutputStream?
> 
>     If not, then would it be okay to use something else for snapshots?
> 
>     We could get rid of the intermediate buffer on both snapshot and
>     recovery.
> 
> 
> yeah I was thinking about that too. Instead of serializing and storing
> the byte[] in Snapshot we should be able to store the root
> NormalizedNode and stream it directly to the ObjectOutputStream when
> akka serializes to persistence. I hope the ObjectOutputStream is backed
> by a FileOutputStream and not a ByteArrayOutputStream :)

That depends on snapshot store architecture -- I sure hope it is backed
by a file and does not go 'this will fit into a memory buffer, wheee!' :)

> However we will run into the same issue when installing a snapshot on a
> follower. We do chunk that over the wire (actually via an
> InstallSnapshot message), as per raft, but  we use the fully serialized
> byte[] to determine the # of chunks and drive it. We'd have to think of
> some other way. Perhaps we can use a temporary RandomAccessFile to stage it.

Any InputStream will do as we traverse it only once -- hence we can
instantiate the stream, start deserializing from it and feed chunks as
they arrive -- assuming deserialization does not run on the same thread
as chunk reception does -- which I am afraid does not hold due to
raftactor/shard being the same actor...

We could also go for channels, but that will complicate deserialization,
as the logic would have to deal with 'not yet available' state at every
byte.

Bye,
Robert

Attachment: signature.asc
Description: OpenPGP digital signature

_______________________________________________
controller-dev mailing list
controller-dev@lists.opendaylight.org
https://lists.opendaylight.org/mailman/listinfo/controller-dev

Reply via email to