chaokunyang commented on issue #1528: URL: https://github.com/apache/incubator-fury/issues/1528#issuecomment-2060836842
> We have quite large files on disk and can not use protobuf because of the 2GB limitation. Thats why we where looking for alternatives: fast serialization with cross-language support. I feel in the future if we store machine learning embeddings in a column based style to a file, the 2GB limit will be a problem quite a lot. This is interesting, if embeddings are storaged, we may need larger limitation. Could we split a big object into some small objects for serialization. I mean, you can serialize like this: ```java Fury fury = xxx; OutputStream stream = xxx; fury.serialize(stream, o1); fury.serialize(stream, o2); fury.serialize(stream, o3); ``` Then for deserializaion, you can: ```java Fury fury = xxx; FuryInputStream stream = xxx; Object o1 = fury.deserialize(stream); Object o2 = fury.deserialize(stream); Object o3 = fury.deserialize(stream); ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
