[
https://issues.apache.org/jira/browse/IGNITE-12543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17058505#comment-17058505
]
Aleksey Plekhanov commented on IGNITE-12543:
--------------------------------------------
[~Pavlukhin], I think the best way to solve this issue is to fix IGNITE-12625
I've checked object size produced by the following code:
{code:java}
public void testSize() throws Exception {
try (Ignite ignite = Ignition.start(Config.getServerConfiguration());
IgniteClient client = Ignition.startClient(getClientConfiguration())
) {
List<TestClass> obj = new ArrayList<>();
for (int i = 0; i < 100; i++)
obj.add(new TestClass());
client.getOrCreateCache(Config.DEFAULT_CACHE_NAME).put(0, obj);
}
}
private static class TestClass {
byte[] arr = new byte[100];
}
{code}
After object {{obj}} is marshalled by thin client the whole request size is
13426 bytes (20 bytes - request header + key size, 6 bytes value collection
header, 134 bytes * 100 items - collection items), but when this object is
unmarshalled (on server side) and marshalled second time (before put it into
cache) size of corresponding {{CacheObject}} instance grows to 1343506 bytes (6
bytes collection header, 13435 bytes * 100 items - collection items). Each
collection item occupying 13435 bytes now - its original request size (whole
byte array is copied each time) + binary object header.
> When put List<List<SomeObject>>, the data was increased much larger.
> --------------------------------------------------------------------
>
> Key: IGNITE-12543
> URL: https://issues.apache.org/jira/browse/IGNITE-12543
> Project: Ignite
> Issue Type: Bug
> Components: thin client
> Affects Versions: 2.6
> Reporter: LEE PYUNG BEOM
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> I use Ignite 2.6 version of Java Thin Client.
>
> When I put data in the form List<List<SomeObject>>,
> The size of the original 200KB data was increased to 50MB when inquired by
> Ignite servers.
> On the Heap Dump, the list element was repeatedly accumulated, increasing the
> data size.
>
> When I checked org.apacheignite.internal.binary.BinaryWriterExImpl.java
> doWriteBinaryObject() method,
> {code:java}
> // org.apacheignite.internal.binary.BinaryWriterExImpl.java
> public void doWriteBinaryObject(@Nullable BinaryObjectImpl po) {
> if (po == null)
> out.writeByte(GridBinaryMarshaller.NULL);
> else {
> byte[] poArr = po.array();
> out.unsafeEnsure(1 + 4 + poArr.length +4);
> out.unsafeWriteByte(GridBinaryMarshaller.BINARY_OBJ);
> out.unsafeWriteInt(poArr.length);
> out.writeByteArray(poArr);
> out.unsafeWriteInt(po.start());
> }
> }
> {code}
>
> The current Ignite implementation for storing data in the form
> List<List<Some_Objectject>> is:
> In the Marshalling stage, for example, data the size of List(5
> members)<List(10 members)<Some_Object(size:200 KB)> is:
> As many as 10*5 of the list's elements are duplicated.
> If the above data contains five objects of 200KB size, ten by one,
> 50 iterations are stored and 200K*10**5 = 100MB of data is used for cache and
> transfer.
> As a result of this increase in data size, it is confirmed that the failure
> of OOM, GC, etc. is caused by occupying Heap memory.
> Unnecessarily redundant data is used for cache storage and network transport.
> When looking up cache data, only some of the data at the top is read based on
> file location information from the entire data, so that normal data is
> retrieved.
> The way we're implemented today is safe from basic behavior, but we're
> wasting memory and network unnecessarily using inefficient algorithms
> This can have very serious consequences. Please check.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
