Re: getSizeInBytes() return type

2017-07-10 Thread Michael Stolz 
I don't think it is appropriate to store objects of 2GB size in Geode.
I know we have hit networking issues with such large objects in the past.

--
Mike Stolz
Principal Engineer, GemFire Product Manager
Mobile: +1-631-835-4771

On Sun, Jul 9, 2017 at 2:13 AM, Daniel Farcovich <
daniel.farcov...@amdocs.com> wrote:

> Raising this question again.
>
> Daniel
>
> From: Daniel Farcovich
> Sent: Wednesday, July 05, 2017 1:26 PM
> To: 'dev@geode.apache.org' <dev@geode.apache.org>
> Subject: getSizeInBytes() return type
>
>
> Hi,
> We implement getSizeInBytes() in from Sizeable interface.
> We have objects with size bigger than MAXINT, bigger than 2GB.
> What is the impact of refactoring the code to return long instead of int?
> I mean except the technical aspect of changing the return types of the
> calling functions etc.
> Which mechanisms / functionalities will be affected from this change, I
> know rebalancing will be affected for example.
>
> Thanks,
> Daniel Farcovich
>
>
> This message and the information contained herein is proprietary and
> confidential and subject to the Amdocs policy statement,
>
> you may review at https://www.amdocs.com/about/email-disclaimer <
> https://www.amdocs.com/about/email-disclaimer>
>

Re: getSizeInBytes() return type

2017-07-10 Thread Darrel Schneider 
I agree with Jake that supporting values whose serialized size is larger
than 2G would be a big change.
Geode has a number of places in the implementation that use a byte array to
store the serialized object form and the maximum array length in Java is 2G.

On Mon, Jul 10, 2017 at 8:43 AM, Jacob Barrett <jbarr...@pivotal.io> wrote:

> Daniel,
>
> A change like this would be pretty deep and impact things like off heap and
> the network protocol.
>
> You really need to consider the problems with storing such large objects
> and ask yourself if you can do something different. Transferring a large
> object requires that it be serialized into memory first so a 2GB object
> requires 2GB of contiguous memory space for the byte[]. Allocating a 2GB
> array will always be done immediately in the old generation and very high
> probability result in a stop the world (STW) GC compaction event to
> defragment the memory enough to allocate it. Every JVM that touches this
> object (clients, primary, secondaries) will be STW GC'ing on nearly every
> transmission of this object. If there are many of these objects on the
> server then STW compaction events may take longer than the server timeout
> and result in servers being ejected from the cluster. If auto
> recover/rebalancing is enabled this could cause other servers to be force
> to take on these large objects are result in them also entering a STW
> compaction event and they too could be ejected.
>
> I mention this because I have seen this happen with an application where
> they were trying to store 1GB objects. It is highly recommend that you
> restructure your objects into regions of smaller objects. Since it is
> likely your 2GB object isn't a single object but a relationship of many
> much smaller objects then this should be very easy to do.
>
> -Jake
>
>
> On Sat, Jul 8, 2017 at 11:13 PM Daniel Farcovich <
> daniel.farcov...@amdocs.com> wrote:
>
> > Raising this question again.
> >
> > Daniel
> >
> > From: Daniel Farcovich
> > Sent: Wednesday, July 05, 2017 1:26 PM
> > To: 'dev@geode.apache.org' <dev@geode.apache.org>
> > Subject: getSizeInBytes() return type
> >
> >
> > Hi,
> > We implement getSizeInBytes() in from Sizeable interface.
> > We have objects with size bigger than MAXINT, bigger than 2GB.
> > What is the impact of refactoring the code to return long instead of int?
> > I mean except the technical aspect of changing the return types of the
> > calling functions etc.
> > Which mechanisms / functionalities will be affected from this change, I
> > know rebalancing will be affected for example.
> >
> > Thanks,
> > Daniel Farcovich
> >
> >
> > This message and the information contained herein is proprietary and
> > confidential and subject to the Amdocs policy statement,
> >
> > you may review at https://www.amdocs.com/about/email-disclaimer <
> > https://www.amdocs.com/about/email-disclaimer>
> >
>

Re: getSizeInBytes() return type

2017-07-10 Thread Jacob Barrett 
Daniel,

A change like this would be pretty deep and impact things like off heap and
the network protocol.

You really need to consider the problems with storing such large objects
and ask yourself if you can do something different. Transferring a large
object requires that it be serialized into memory first so a 2GB object
requires 2GB of contiguous memory space for the byte[]. Allocating a 2GB
array will always be done immediately in the old generation and very high
probability result in a stop the world (STW) GC compaction event to
defragment the memory enough to allocate it. Every JVM that touches this
object (clients, primary, secondaries) will be STW GC'ing on nearly every
transmission of this object. If there are many of these objects on the
server then STW compaction events may take longer than the server timeout
and result in servers being ejected from the cluster. If auto
recover/rebalancing is enabled this could cause other servers to be force
to take on these large objects are result in them also entering a STW
compaction event and they too could be ejected.

I mention this because I have seen this happen with an application where
they were trying to store 1GB objects. It is highly recommend that you
restructure your objects into regions of smaller objects. Since it is
likely your 2GB object isn't a single object but a relationship of many
much smaller objects then this should be very easy to do.

-Jake


On Sat, Jul 8, 2017 at 11:13 PM Daniel Farcovich <
daniel.farcov...@amdocs.com> wrote:

> Raising this question again.
>
> Daniel
>
> From: Daniel Farcovich
> Sent: Wednesday, July 05, 2017 1:26 PM
> To: 'dev@geode.apache.org' <dev@geode.apache.org>
> Subject: getSizeInBytes() return type
>
>
> Hi,
> We implement getSizeInBytes() in from Sizeable interface.
> We have objects with size bigger than MAXINT, bigger than 2GB.
> What is the impact of refactoring the code to return long instead of int?
> I mean except the technical aspect of changing the return types of the
> calling functions etc.
> Which mechanisms / functionalities will be affected from this change, I
> know rebalancing will be affected for example.
>
> Thanks,
> Daniel Farcovich
>
>
> This message and the information contained herein is proprietary and
> confidential and subject to the Amdocs policy statement,
>
> you may review at https://www.amdocs.com/about/email-disclaimer <
> https://www.amdocs.com/about/email-disclaimer>
>

RE: getSizeInBytes() return type

2017-07-09 Thread Daniel Farcovich 
Raising this question again.

Daniel

From: Daniel Farcovich
Sent: Wednesday, July 05, 2017 1:26 PM
To: 'dev@geode.apache.org' <dev@geode.apache.org>
Subject: getSizeInBytes() return type


Hi,
We implement getSizeInBytes() in from Sizeable interface.
We have objects with size bigger than MAXINT, bigger than 2GB.
What is the impact of refactoring the code to return long instead of int? I 
mean except the technical aspect of changing the return types of the calling 
functions etc.
Which mechanisms / functionalities will be affected from this change, I know 
rebalancing will be affected for example.

Thanks,
Daniel Farcovich


This message and the information contained herein is proprietary and 
confidential and subject to the Amdocs policy statement,

you may review at https://www.amdocs.com/about/email-disclaimer 
<https://www.amdocs.com/about/email-disclaimer>

getSizeInBytes() return type

2017-07-05 Thread Daniel Farcovich 

Hi,
We implement getSizeInBytes() in from Sizeable interface.
We have objects with size bigger than MAXINT, bigger than 2GB.
What is the impact of refactoring the code to return long instead of int? I 
mean except the technical aspect of changing the return types of the calling 
functions etc.
Which mechanisms / functionalities will be affected from this change, I know 
rebalancing will be affected for example.

Thanks,
Daniel Farcovich


This message and the information contained herein is proprietary and 
confidential and subject to the Amdocs policy statement,

you may review at https://www.amdocs.com/about/email-disclaimer