Can you please share your heapdump? --Yakov
2016-02-18 13:29 GMT+03:00 Vladimir Ozerov <[email protected]>: > Hi Vidhu, > > That is right - you cannot use indexes with raw binary serialization > because in raw mode we do not write object's metadata (this is exactly why > resulting object is more compact). As a partial solution you can try > writing only fields participating in queries using normal binary > serialization (i.e. with field names) and write the rest fields in binary > mode. > > Also, as Val mentioned, indexes also consume some space. > > Vladimir. > > On Wed, Feb 17, 2016 at 9:36 PM, vidhu <[email protected]> wrote: > >> Hi Vladimir, >> >> We are using 2 server nodes of 5 GB each. Also keeping backup count 1. >> >> We used binarylizable and raw mode which shrinked size but SQL queries >> are not running on raw mode. Is there any way we can run SQL queries with >> raw mode or we could use any external serializer with which SQL queries >> could run? >> On 17 Feb 2016 23:54, "Vladimir Ozerov [via Apache Ignite Users]" <[hidden >> email] <http:///user/SendEmail.jtp?type=node&node=3057&i=0>> wrote: >> >>> Hi vidhu, >>> >>> If your object have 101 int field, each instance serialized with >>> BinaryMarshaller (the default one) will be an array of 731 byte. This means >>> 1.4GB of heap for 2M objects. How many nodes are there in your application? >>> Do you have backups? It possible pleas share your configuration and source >>> code. >>> >>> Using Binarylizable interface and raw mode could shrink size of >>> serialized object to somewhat about 450 bytes per instance. It looks >>> strange to me that you cannot get objects serialized in this way form >>> cache. Again, you source code will help us to understand what the problem >>> is. >>> >>> Vladimir. >>> >>> On Wed, Feb 17, 2016 at 8:18 AM, vidhu <[hidden email] >>> <http:///user/SendEmail.jtp?type=node&node=3056&i=0>> wrote: >>> >>>> Thanks for quick reply. >>>> >>>> I mistakenly write 20 million to earlier but objects I am putting in >>>> cache are 2 million which means .9 GB approx. >>>> >>>> Sorry for putting wrong number of objects earlier. >>>> >>>> I am starting 2 server nodes of 5 GB each and a client node of 5 gb to >>>> load data into cache. >>>> >>>> Apart from this cache I am having 2 more caches whose size combined is >>>> less than 400 MB. >>>> On 17 Feb 2016 10:44, "vkulichenko [via Apache Ignite Users]" <[hidden >>>> email] <http:///user/SendEmail.jtp?type=node&node=3051&i=0>> wrote: >>>> >>>>> Hi vidhu, >>>>> >>>>> First of all, can you please subscribe to the mailing list so that >>>>> community receives email notifications? Please follow this simple >>>>> instruction: >>>>> http://apache-ignite-users.70518.x6.nabble.com/mailing_list/MailingListOptions.jtp?forum=1 >>>>> >>>>> vidhu wrote >>>>> I am facing a problem where my heap reaches to about 2.8 gb when I am >>>>> storing 20 million objects. Size of each object is approx. 462 bytes. >>>>> Structure of my object is a simple pojo having 101 fields each of int >>>>> type. >>>>> >>>>> I tried using following 3 marshalling techniques: >>>>> >>>>> 1.) Default marshaller of Ignite. (Size of heap ~ 2.8 gb, looking with >>>>> Visual VM find that most of objects on heap are byte array.) >>>>> >>>>> 2.) Optimized marshaller. (Again size was ~ 2.8 gb, looking with >>>>> Visual VM find that this time size of objects on heap of byte array is >>>>> half >>>>> + there are lot of objects of class that I put in cache. ) >>>>> >>>>> 3.) Tried implementing Binarylizable in my class that I am storing in >>>>> cache. I used rawWriter. This time size reduced considerably but I was not >>>>> able to read values out of cache, (getting null on read). >>>>> >>>>> I am implementing this in java version 1.8. >>>>> >>>>> 20,000,000 * 462 bytes is already around 8.6GB of memory, so 2.8 is >>>>> not enough even without Ignite's overhead. Did you try to allocate more >>>>> heap memory and/or start more nodes? With your sizings I would start with >>>>> two nodes, 8GB of memory per node. >>>>> >>>>> As for options you tried, I would recommend to use default binary >>>>> marshaller because it's much more compact. Also you should take a look at >>>>> offheap memory [1]. It can be useful if you're going to have a lot of >>>>> data. >>>>> >>>>> [1] https://apacheignite.readme.io/docs/off-heap-memory >>>>> >>>>> -Val >>>>> >>>>> ------------------------------ >>>>> If you reply to this email, your message will be added to the >>>>> discussion below: >>>>> >>>>> http://apache-ignite-users.70518.x6.nabble.com/Huge-heap-size-for-ignite-cache-when-storing-20-million-objects-of-small-size-tp3049p3050.html >>>>> To unsubscribe from Huge heap size for ignite cache when storing 20 >>>>> million objects of small size, click here. >>>>> NAML >>>>> <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> >>>>> >>>> >>>> ------------------------------ >>>> View this message in context: Re: Huge heap size for ignite cache when >>>> storing 20 million objects of small size >>>> <http://apache-ignite-users.70518.x6.nabble.com/Huge-heap-size-for-ignite-cache-when-storing-20-million-objects-of-small-size-tp3049p3051.html> >>>> >>>> Sent from the Apache Ignite Users mailing list archive >>>> <http://apache-ignite-users.70518.x6.nabble.com/> at Nabble.com. >>>> >>> >>> >>> >>> ------------------------------ >>> If you reply to this email, your message will be added to the discussion >>> below: >>> >>> http://apache-ignite-users.70518.x6.nabble.com/Huge-heap-size-for-ignite-cache-when-storing-20-million-objects-of-small-size-tp3049p3056.html >>> To unsubscribe from Huge heap size for ignite cache when storing 20 >>> million objects of small size, click here. >>> NAML >>> <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> >>> >> >> ------------------------------ >> View this message in context: Re: Huge heap size for ignite cache when >> storing 20 million objects of small size >> <http://apache-ignite-users.70518.x6.nabble.com/Huge-heap-size-for-ignite-cache-when-storing-20-million-objects-of-small-size-tp3049p3057.html> >> Sent from the Apache Ignite Users mailing list archive >> <http://apache-ignite-users.70518.x6.nabble.com/> at Nabble.com. >> > >
