Re: Recreate Cache -- is it possible?

[Anilkumar Gingade]

A server/peer can have its own cache configuration addition to
cluster-configuration; if we still support that; does the new
"reloadNewClusterConfiguration()" takes care of it?

-Anil.


On Wed, Mar 21, 2018 at 1:06 PM, Jinmei Liao  wrote:

> Sounds like this is a slippery slope. I reworked the strategy: instead of
> calling cache.close, I only issue a call to the locator to get the cluster
> configuration again and do a reload of the properties and cacheXml. Here is
> the PR for this approach:
> https://github.com/apache/geode/pull/1656
>
> Basically this is what the reloadClusterConfiguration does:
> https://github.com/apache/geode/pull/1656/files#diff-
> 14ace6c5abf2f68c480b55a7c882e18c
>
> If you see anything obviously wrong, or even vaguely wrong, please comment
> on the PR, we will try to test it out.
>
> Thanks!
>
> On Wed, Mar 21, 2018 at 12:42 PM, Kirk Lund  wrote:
>
> > The non-daemon thread in a process launched with ServerLauncher is
> looping
> > in waitOnServer. When you close the Cache, that loop exits and the
> > ServerLauncher process exits.
> >
> > As Bruce pointed you, JUnit and the DUnit VMs have other non-daemon
> > threads.
> >
> > You might need to alter ServerLauncher.waitOnServer() and
> > LocatorLauncher.waitOnLocator() for what you're doing.
> >
> > On Wed, Mar 21, 2018 at 10:28 AM, Jinmei Liao  wrote:
> >
> > > Bruce: this sounds like the root cause of the differences between the
> > dunit
> > > test and reall app test.
> > >
> > > On Wed, Mar 21, 2018 at 10:22 AM, Bruce Schuchardt <
> > bschucha...@pivotal.io
> > > >
> > > wrote:
> > >
> > > > It's likely that the JVM is exiting because the AcceptorImpl thread
> is
> > > the
> > > > only non-daemon thread and it is stopped when the cache is closed.
> > DUnit
> > > > JVMs have a non-daemon main() thread that keeps them alive.
> > > >
> > > >
> > > >
> > > > On 3/21/18 9:48 AM, Jinmei Liao wrote:
> > > >
> > > >> We would like to allow users to import a new set of cluster
> > > configuration
> > > >> with running servers as long as we make sure these servers are
> vanilla
> > > >> servers (servers that are just started with nothing in it). Now
> since
> > > the
> > > >> servers are already up, caches are already created, we will need to
> > > >> re-create the cache with the new xml received from the locator.
> > > Originally
> > > >> our implementation on the servers boils down to:
> > > >>
> > > >> cache.close("Re-create Cache", true, true);
> > > >>
> > > >> GemFireCacheImpl.create(oldDs, cacheConfig);
> > > >>
> > > >>
> > > >> but the cache.close call eventually leads to a VM exit (somehow in
> the
> > > >> DUunit VM, it doesn not), so this does not work with real
> application
> > > >> environment. Now we are wondering is there a safe to recreate the
> > cache
> > > >> instance with a new set of properties/cacheXml without triggering
> the
> > > >> entire shutdown sequence?
> > > >>
> > > >>
> > > >>
> > > >
> > >
> > >
> > > --
> > > Cheers
> > >
> > > Jinmei
> > >
> >
>
>
>
> --
> Cheers
>
> Jinmei
>

Re: Recreate Cache -- is it possible?

[Jinmei Liao]

Sounds like this is a slippery slope. I reworked the strategy: instead of
calling cache.close, I only issue a call to the locator to get the cluster
configuration again and do a reload of the properties and cacheXml. Here is
the PR for this approach:
https://github.com/apache/geode/pull/1656

Basically this is what the reloadClusterConfiguration does:
https://github.com/apache/geode/pull/1656/files#diff-14ace6c5abf2f68c480b55a7c882e18c

If you see anything obviously wrong, or even vaguely wrong, please comment
on the PR, we will try to test it out.

Thanks!

On Wed, Mar 21, 2018 at 12:42 PM, Kirk Lund  wrote:

> The non-daemon thread in a process launched with ServerLauncher is looping
> in waitOnServer. When you close the Cache, that loop exits and the
> ServerLauncher process exits.
>
> As Bruce pointed you, JUnit and the DUnit VMs have other non-daemon
> threads.
>
> You might need to alter ServerLauncher.waitOnServer() and
> LocatorLauncher.waitOnLocator() for what you're doing.
>
> On Wed, Mar 21, 2018 at 10:28 AM, Jinmei Liao  wrote:
>
> > Bruce: this sounds like the root cause of the differences between the
> dunit
> > test and reall app test.
> >
> > On Wed, Mar 21, 2018 at 10:22 AM, Bruce Schuchardt <
> bschucha...@pivotal.io
> > >
> > wrote:
> >
> > > It's likely that the JVM is exiting because the AcceptorImpl thread is
> > the
> > > only non-daemon thread and it is stopped when the cache is closed.
> DUnit
> > > JVMs have a non-daemon main() thread that keeps them alive.
> > >
> > >
> > >
> > > On 3/21/18 9:48 AM, Jinmei Liao wrote:
> > >
> > >> We would like to allow users to import a new set of cluster
> > configuration
> > >> with running servers as long as we make sure these servers are vanilla
> > >> servers (servers that are just started with nothing in it). Now since
> > the
> > >> servers are already up, caches are already created, we will need to
> > >> re-create the cache with the new xml received from the locator.
> > Originally
> > >> our implementation on the servers boils down to:
> > >>
> > >> cache.close("Re-create Cache", true, true);
> > >>
> > >> GemFireCacheImpl.create(oldDs, cacheConfig);
> > >>
> > >>
> > >> but the cache.close call eventually leads to a VM exit (somehow in the
> > >> DUunit VM, it doesn not), so this does not work with real application
> > >> environment. Now we are wondering is there a safe to recreate the
> cache
> > >> instance with a new set of properties/cacheXml without triggering the
> > >> entire shutdown sequence?
> > >>
> > >>
> > >>
> > >
> >
> >
> > --
> > Cheers
> >
> > Jinmei
> >
>



-- 
Cheers

Jinmei

Re: Recreate Cache -- is it possible?

[Kirk Lund]

The non-daemon thread in a process launched with ServerLauncher is looping
in waitOnServer. When you close the Cache, that loop exits and the
ServerLauncher process exits.

As Bruce pointed you, JUnit and the DUnit VMs have other non-daemon threads.

You might need to alter ServerLauncher.waitOnServer() and
LocatorLauncher.waitOnLocator() for what you're doing.

On Wed, Mar 21, 2018 at 10:28 AM, Jinmei Liao  wrote:

> Bruce: this sounds like the root cause of the differences between the dunit
> test and reall app test.
>
> On Wed, Mar 21, 2018 at 10:22 AM, Bruce Schuchardt  >
> wrote:
>
> > It's likely that the JVM is exiting because the AcceptorImpl thread is
> the
> > only non-daemon thread and it is stopped when the cache is closed.  DUnit
> > JVMs have a non-daemon main() thread that keeps them alive.
> >
> >
> >
> > On 3/21/18 9:48 AM, Jinmei Liao wrote:
> >
> >> We would like to allow users to import a new set of cluster
> configuration
> >> with running servers as long as we make sure these servers are vanilla
> >> servers (servers that are just started with nothing in it). Now since
> the
> >> servers are already up, caches are already created, we will need to
> >> re-create the cache with the new xml received from the locator.
> Originally
> >> our implementation on the servers boils down to:
> >>
> >> cache.close("Re-create Cache", true, true);
> >>
> >> GemFireCacheImpl.create(oldDs, cacheConfig);
> >>
> >>
> >> but the cache.close call eventually leads to a VM exit (somehow in the
> >> DUunit VM, it doesn not), so this does not work with real application
> >> environment. Now we are wondering is there a safe to recreate the cache
> >> instance with a new set of properties/cacheXml without triggering the
> >> entire shutdown sequence?
> >>
> >>
> >>
> >
>
>
> --
> Cheers
>
> Jinmei
>

Re: Recreate Cache -- is it possible?

[Swapnil Bawaskar]

Does this mean auto-reconnect is broken as well?

On Wed, Mar 21, 2018 at 10:28 AM Jinmei Liao  wrote:

> Bruce: this sounds like the root cause of the differences between the dunit
> test and reall app test.
>
> On Wed, Mar 21, 2018 at 10:22 AM, Bruce Schuchardt  >
> wrote:
>
> > It's likely that the JVM is exiting because the AcceptorImpl thread is
> the
> > only non-daemon thread and it is stopped when the cache is closed.  DUnit
> > JVMs have a non-daemon main() thread that keeps them alive.
> >
> >
> >
> > On 3/21/18 9:48 AM, Jinmei Liao wrote:
> >
> >> We would like to allow users to import a new set of cluster
> configuration
> >> with running servers as long as we make sure these servers are vanilla
> >> servers (servers that are just started with nothing in it). Now since
> the
> >> servers are already up, caches are already created, we will need to
> >> re-create the cache with the new xml received from the locator.
> Originally
> >> our implementation on the servers boils down to:
> >>
> >> cache.close("Re-create Cache", true, true);
> >>
> >> GemFireCacheImpl.create(oldDs, cacheConfig);
> >>
> >>
> >> but the cache.close call eventually leads to a VM exit (somehow in the
> >> DUunit VM, it doesn not), so this does not work with real application
> >> environment. Now we are wondering is there a safe to recreate the cache
> >> instance with a new set of properties/cacheXml without triggering the
> >> entire shutdown sequence?
> >>
> >>
> >>
> >
>
>
> --
> Cheers
>
> Jinmei
>

Re: Recreate Cache -- is it possible?

[Jinmei Liao]

Bruce: this sounds like the root cause of the differences between the dunit
test and reall app test.

On Wed, Mar 21, 2018 at 10:22 AM, Bruce Schuchardt 
wrote:

> It's likely that the JVM is exiting because the AcceptorImpl thread is the
> only non-daemon thread and it is stopped when the cache is closed.  DUnit
> JVMs have a non-daemon main() thread that keeps them alive.
>
>
>
> On 3/21/18 9:48 AM, Jinmei Liao wrote:
>
>> We would like to allow users to import a new set of cluster configuration
>> with running servers as long as we make sure these servers are vanilla
>> servers (servers that are just started with nothing in it). Now since the
>> servers are already up, caches are already created, we will need to
>> re-create the cache with the new xml received from the locator. Originally
>> our implementation on the servers boils down to:
>>
>> cache.close("Re-create Cache", true, true);
>>
>> GemFireCacheImpl.create(oldDs, cacheConfig);
>>
>>
>> but the cache.close call eventually leads to a VM exit (somehow in the
>> DUunit VM, it doesn not), so this does not work with real application
>> environment. Now we are wondering is there a safe to recreate the cache
>> instance with a new set of properties/cacheXml without triggering the
>> entire shutdown sequence?
>>
>>
>>
>


-- 
Cheers

Jinmei

Re: Recreate Cache -- is it possible?

[Jinmei Liao]

Kirk: good call. I will see if I can add an interface method to the
InternalCache.

On Wed, Mar 21, 2018 at 10:13 AM, Kirk Lund  wrote:

> I'm actively working to remove direct usage of GemFireCacheImpl from all of
> our code. Is there a reason you have to directly use GemFireCacheImpl? This
> is going to add one more thing that I have to change.
>
> Maybe the VM exit is happening because you've forked a new VM with
> ServerLauncher or LocatorLauncher? I think those launchers create a loop in
> the main thread which keeps the process alive -- if something causes that
> main thread to terminate then the VM will go away.
>
> On Wed, Mar 21, 2018 at 10:08 AM, Anilkumar Gingade 
> wrote:
>
> > Instead of adding new feature; can we address the issue with VM exit. The
> > auto-reconnect feature does cache close without exiting the VM.
> >
> > -Anil.
> >
> >
> >
> >
> > On Wed, Mar 21, 2018 at 9:48 AM, Jinmei Liao  wrote:
> >
> > > We would like to allow users to import a new set of cluster
> configuration
> > > with running servers as long as we make sure these servers are vanilla
> > > servers (servers that are just started with nothing in it). Now since
> the
> > > servers are already up, caches are already created, we will need to
> > > re-create the cache with the new xml received from the locator.
> > Originally
> > > our implementation on the servers boils down to:
> > >
> > > cache.close("Re-create Cache", true, true);
> > >
> > > GemFireCacheImpl.create(oldDs, cacheConfig);
> > >
> > >
> > > but the cache.close call eventually leads to a VM exit (somehow in the
> > > DUunit VM, it doesn not), so this does not work with real application
> > > environment. Now we are wondering is there a safe to recreate the cache
> > > instance with a new set of properties/cacheXml without triggering the
> > > entire shutdown sequence?
> > >
> > >
> > > --
> > > Cheers
> > >
> > > Jinmei
> > >
> >
>



-- 
Cheers

Jinmei

Re: Recreate Cache -- is it possible?

[Bruce Schuchardt]

It's likely that the JVM is exiting because the AcceptorImpl thread is 
the only non-daemon thread and it is stopped when the cache is closed.  
DUnit JVMs have a non-daemon main() thread that keeps them alive.



On 3/21/18 9:48 AM, Jinmei Liao wrote:

We would like to allow users to import a new set of cluster configuration
with running servers as long as we make sure these servers are vanilla
servers (servers that are just started with nothing in it). Now since the
servers are already up, caches are already created, we will need to
re-create the cache with the new xml received from the locator. Originally
our implementation on the servers boils down to:

cache.close("Re-create Cache", true, true);

GemFireCacheImpl.create(oldDs, cacheConfig);


but the cache.close call eventually leads to a VM exit (somehow in the
DUunit VM, it doesn not), so this does not work with real application
environment. Now we are wondering is there a safe to recreate the cache
instance with a new set of properties/cacheXml without triggering the
entire shutdown sequence?

Re: Recreate Cache -- is it possible?

[Kirk Lund]

I'm actively working to remove direct usage of GemFireCacheImpl from all of
our code. Is there a reason you have to directly use GemFireCacheImpl? This
is going to add one more thing that I have to change.

Maybe the VM exit is happening because you've forked a new VM with
ServerLauncher or LocatorLauncher? I think those launchers create a loop in
the main thread which keeps the process alive -- if something causes that
main thread to terminate then the VM will go away.

On Wed, Mar 21, 2018 at 10:08 AM, Anilkumar Gingade 
wrote:

> Instead of adding new feature; can we address the issue with VM exit. The
> auto-reconnect feature does cache close without exiting the VM.
>
> -Anil.
>
>
>
>
> On Wed, Mar 21, 2018 at 9:48 AM, Jinmei Liao  wrote:
>
> > We would like to allow users to import a new set of cluster configuration
> > with running servers as long as we make sure these servers are vanilla
> > servers (servers that are just started with nothing in it). Now since the
> > servers are already up, caches are already created, we will need to
> > re-create the cache with the new xml received from the locator.
> Originally
> > our implementation on the servers boils down to:
> >
> > cache.close("Re-create Cache", true, true);
> >
> > GemFireCacheImpl.create(oldDs, cacheConfig);
> >
> >
> > but the cache.close call eventually leads to a VM exit (somehow in the
> > DUunit VM, it doesn not), so this does not work with real application
> > environment. Now we are wondering is there a safe to recreate the cache
> > instance with a new set of properties/cacheXml without triggering the
> > entire shutdown sequence?
> >
> >
> > --
> > Cheers
> >
> > Jinmei
> >
>

Re: Recreate Cache -- is it possible?

[Jinmei Liao]

Great, how do we trigger cache.close indicating auto-reconnect?

On Wed, Mar 21, 2018 at 10:08 AM, Anilkumar Gingade 
wrote:

> Instead of adding new feature; can we address the issue with VM exit. The
> auto-reconnect feature does cache close without exiting the VM.
>
> -Anil.
>
>
>
>
> On Wed, Mar 21, 2018 at 9:48 AM, Jinmei Liao  wrote:
>
> > We would like to allow users to import a new set of cluster configuration
> > with running servers as long as we make sure these servers are vanilla
> > servers (servers that are just started with nothing in it). Now since the
> > servers are already up, caches are already created, we will need to
> > re-create the cache with the new xml received from the locator.
> Originally
> > our implementation on the servers boils down to:
> >
> > cache.close("Re-create Cache", true, true);
> >
> > GemFireCacheImpl.create(oldDs, cacheConfig);
> >
> >
> > but the cache.close call eventually leads to a VM exit (somehow in the
> > DUunit VM, it doesn not), so this does not work with real application
> > environment. Now we are wondering is there a safe to recreate the cache
> > instance with a new set of properties/cacheXml without triggering the
> > entire shutdown sequence?
> >
> >
> > --
> > Cheers
> >
> > Jinmei
> >
>



-- 
Cheers

Jinmei

Re: Recreate Cache -- is it possible?

[Anilkumar Gingade]

Instead of adding new feature; can we address the issue with VM exit. The
auto-reconnect feature does cache close without exiting the VM.

-Anil.




On Wed, Mar 21, 2018 at 9:48 AM, Jinmei Liao  wrote:

> We would like to allow users to import a new set of cluster configuration
> with running servers as long as we make sure these servers are vanilla
> servers (servers that are just started with nothing in it). Now since the
> servers are already up, caches are already created, we will need to
> re-create the cache with the new xml received from the locator. Originally
> our implementation on the servers boils down to:
>
> cache.close("Re-create Cache", true, true);
>
> GemFireCacheImpl.create(oldDs, cacheConfig);
>
>
> but the cache.close call eventually leads to a VM exit (somehow in the
> DUunit VM, it doesn not), so this does not work with real application
> environment. Now we are wondering is there a safe to recreate the cache
> instance with a new set of properties/cacheXml without triggering the
> entire shutdown sequence?
>
>
> --
> Cheers
>
> Jinmei
>

Recreate Cache -- is it possible?

[Jinmei Liao]

We would like to allow users to import a new set of cluster configuration
with running servers as long as we make sure these servers are vanilla
servers (servers that are just started with nothing in it). Now since the
servers are already up, caches are already created, we will need to
re-create the cache with the new xml received from the locator. Originally
our implementation on the servers boils down to:

cache.close("Re-create Cache", true, true);

GemFireCacheImpl.create(oldDs, cacheConfig);


but the cache.close call eventually leads to a VM exit (somehow in the
DUunit VM, it doesn not), so this does not work with real application
environment. Now we are wondering is there a safe to recreate the cache
instance with a new set of properties/cacheXml without triggering the
entire shutdown sequence?


-- 
Cheers

Jinmei