Re: Addition of rmi-io library

[Jinmei Liao]

As far as I can see, we are utilizing the streaming capability provided by
the rmi-io, the content is going over the wire on whatever port that was
port before. When streaming content from the gfsh to the jmxManager, it's
using the jmx port; when getting jars between locator/servers, it's using
the FunctionService, so it's whatever communication channel that
FunctionService is using.

All the FileContent are saved in temp folder, and get cleaned up after each
deployment.

On Tue, Jan 23, 2018 at 3:17 PM, Dan Smith  wrote:

> I don't have an issue with the dependency. But if we are opening up new
> ports for RMI connections, that seems like a potential security risk. If
> someone has enabled cluster SSL we shouldn't be opening up an insecure port
> for RMI connections.
>
> We should also make sure this is not leaking open sockets/file decriptors.
> How does this SimpleRemoteInputStream we are creating get shutdown and
> cleaned up?
>
> -Dan
>
> On Tue, Jan 23, 2018 at 2:36 PM, Jens Deppe  wrote:
>
> > Apologies that this was not raised earlier in discussion but I'm happy to
> > describe it now.
> >
> > *Background:*
> >
> > When deploying jars into Geode they are moved through the system as
> simple
> > byte[] blobs. This obviously consumes memory. The various affected areas
> > are:
> >
> > - gfsh reads the jars into memory
> > - the jars are pushed to the locator (via a jmx call) - again creating a
> > byte[] blob on the locator
> > - from the locator, the jars are pushed to all servers via a function
> call
> > (also sending the jars as byte[] blobs).
> >
> > Obviously if the jar is small this would not be a problem, however in
> > memory constrained systems or with large jars this is obviously going to
> > put pressure on memory and possibly result in OOM situations. In fact,
> the
> > reason this came up was that some folks were unable to deploy a 40Mb jar
> to
> > a 512Mb (heap) locator.
> >
> > *rmi-io:*
> >
> > After doing some research, it seemed that the ideal solution would be
> > something that allows for serializing Input/OutputStreams. Java doesn't
> > provide anything natively.
> >
> > One library that stood out as being robust and feature complete was
> rmi-io
> > [1]. This allows for serializing a remote Input/OutputStream object which
> > then lets us completely avoid having to pull deploying jars into memory
> > everywhere. Under the covers it uses RMI and allows for either 'pulling'
> or
> > 'pushing' data. The reference page [2] has nice sequence diagrams.
> >
> > If anyone sees any issues with this, please do raise them. The current
> > usage of this has not changed any user-facing interaction so ultimately
> > changing the actual implemented fix for this problem (if we needed to)
> > would not have any external effect.
> >
> > Thanks
> > --Jens
> >
> > [1] http://openhms.sourceforge.net/rmiio
> > [2] http://openhms.sourceforge.net/rmiio/class_reference.html
> >
>



-- 
Cheers

Jinmei

Re: [VOTE] Apache Geode release - 1.4.0 RC1

[Dan Smith]

This release candidate does have the rmi-io changes. I think we need to
sort out if we are creating insecure sockets due to those changes - if so
we need to fix that before releasing.

-Dan

On Tue, Jan 23, 2018 at 2:32 PM, Swapnil Bawaskar 
wrote:

> This is the first release candidate for Apache Geode, version 1.4.0.
> Thanks to all the community members for their contributions to this
> release!
>
> *** Please download, test and vote by Friday, January 26, 1200 hrs
> US Pacific. ***
>
> It fixes 277 issues. release notes can be found at:
> https://issues.apache.org/jira/projects/GEODE/versions/12341842
>
> Note that we are voting upon the source tags: rel/v1.4.0.RC1
> https://github.com/apache/geode/tree/rel/v1.4.0.RC1
> https://github.com/apache/geode-examples/tree/rel/v1.4.0.RC1
>
> Commit ID:
> 7dbb280fbff0320efd427c5553a6e666328bc50c (geode)
> ced35f88173b04ac8e104b9cae10cc38289675fa (geode-examples)
>
> Source and binary files:
> https://dist.apache.org/repos/dist/dev/geode/1.4.0.RC1
>
> Maven staging repo:
> https://repository.apache.org/content/repositories/orgapachegeode-1036
>
>
> Geode's KEYS file containing PGP keys we use to sign the release:
> https://github.com/apache/geode/blob/develop/KEYS
>
> Release Signed with Key: pub 4096R/18F902DB 2016-04-07
> Fingerprint: E1B1 ABE3 4753 E7BA 8097 4285 8F8F 2BCC 18F9 02DB
>

Geode unit tests completed in 'develop/AcceptanceTest' with non-zero exit code

[apachegeodeci]

Pipeline results can be found at:

Concourse: 
https://concourse.apachegeode-ci.info/teams/main/pipelines/develop/jobs/AcceptanceTest/builds/177

Re: Geode enterprise support

[Akihiro Kitada]

Hello Vladi,

There are no enterprise support services provided by Pivotal Customer
Support at least.

https://discuss.pivotal.io/hc/en-us/articles/206097347-How-can-I-get-support-for-Geode-

Currently, Apache Geode and Pivotal GemFire are same things. If you need
enterprise support services, why don't you choose Pivotal GemFire rather
than Apache Geode?



-- 
Akihiro Kitada  |  Staff Customer Engineer |  +81 80 3716 3736
Support.Pivotal.io   |  Mon-Fri  9:00am to
5:30pm JST  |  1-877-477-2269
[image: support]  [image: twitter]
 [image: linkedin]
 [image: facebook]
 [image: google plus]
 [image: youtube]



2018-01-23 17:14 GMT+09:00 Vladi Polonsky :

> Hi
>
> Is there a company giving enterprise support of Apache Geode? (not Gemfire)
>
> Regards
> Vladi
> This message and the information contained herein is proprietary and
> confidential and subject to the Amdocs policy statement,
>
> you may review at https://www.amdocs.com/about/email-disclaimer <
> https://www.amdocs.com/about/email-disclaimer>
>

Re: Addition of rmi-io library

[Dan Smith]

I don't have an issue with the dependency. But if we are opening up new
ports for RMI connections, that seems like a potential security risk. If
someone has enabled cluster SSL we shouldn't be opening up an insecure port
for RMI connections.

We should also make sure this is not leaking open sockets/file decriptors.
How does this SimpleRemoteInputStream we are creating get shutdown and
cleaned up?

-Dan

On Tue, Jan 23, 2018 at 2:36 PM, Jens Deppe  wrote:

> Apologies that this was not raised earlier in discussion but I'm happy to
> describe it now.
>
> *Background:*
>
> When deploying jars into Geode they are moved through the system as simple
> byte[] blobs. This obviously consumes memory. The various affected areas
> are:
>
> - gfsh reads the jars into memory
> - the jars are pushed to the locator (via a jmx call) - again creating a
> byte[] blob on the locator
> - from the locator, the jars are pushed to all servers via a function call
> (also sending the jars as byte[] blobs).
>
> Obviously if the jar is small this would not be a problem, however in
> memory constrained systems or with large jars this is obviously going to
> put pressure on memory and possibly result in OOM situations. In fact, the
> reason this came up was that some folks were unable to deploy a 40Mb jar to
> a 512Mb (heap) locator.
>
> *rmi-io:*
>
> After doing some research, it seemed that the ideal solution would be
> something that allows for serializing Input/OutputStreams. Java doesn't
> provide anything natively.
>
> One library that stood out as being robust and feature complete was rmi-io
> [1]. This allows for serializing a remote Input/OutputStream object which
> then lets us completely avoid having to pull deploying jars into memory
> everywhere. Under the covers it uses RMI and allows for either 'pulling' or
> 'pushing' data. The reference page [2] has nice sequence diagrams.
>
> If anyone sees any issues with this, please do raise them. The current
> usage of this has not changed any user-facing interaction so ultimately
> changing the actual implemented fix for this problem (if we needed to)
> would not have any external effect.
>
> Thanks
> --Jens
>
> [1] http://openhms.sourceforge.net/rmiio
> [2] http://openhms.sourceforge.net/rmiio/class_reference.html
>

Proposal: GEODE-4367 - Return PDXInstance when Domain Object can't be found

[Addison Huddy]

Hi Geode Devs,

I'm proposing the following change to how we handle deserialization when
Domain Objects can't be found and pdx-serialize=false.

https://issues.apache.org/jira/browse/GEODE-4367

Looking forward to the discussion.

\ah

[Spring CI] Spring Data GemFire > Nightly-ApacheGeode > #806 was SUCCESSFUL (with 2324 tests)

[Spring CI]

---
Spring Data GemFire > Nightly-ApacheGeode > #806 was successful.
---
Scheduled
2326 tests in total.

https://build.spring.io/browse/SGF-NAG-806/





--
This message is automatically generated by Atlassian Bamboo

Addition of rmi-io library

[Jens Deppe]

Apologies that this was not raised earlier in discussion but I'm happy to
describe it now.

*Background:*

When deploying jars into Geode they are moved through the system as simple
byte[] blobs. This obviously consumes memory. The various affected areas
are:

- gfsh reads the jars into memory
- the jars are pushed to the locator (via a jmx call) - again creating a
byte[] blob on the locator
- from the locator, the jars are pushed to all servers via a function call
(also sending the jars as byte[] blobs).

Obviously if the jar is small this would not be a problem, however in
memory constrained systems or with large jars this is obviously going to
put pressure on memory and possibly result in OOM situations. In fact, the
reason this came up was that some folks were unable to deploy a 40Mb jar to
a 512Mb (heap) locator.

*rmi-io:*

After doing some research, it seemed that the ideal solution would be
something that allows for serializing Input/OutputStreams. Java doesn't
provide anything natively.

One library that stood out as being robust and feature complete was rmi-io
[1]. This allows for serializing a remote Input/OutputStream object which
then lets us completely avoid having to pull deploying jars into memory
everywhere. Under the covers it uses RMI and allows for either 'pulling' or
'pushing' data. The reference page [2] has nice sequence diagrams.

If anyone sees any issues with this, please do raise them. The current
usage of this has not changed any user-facing interaction so ultimately
changing the actual implemented fix for this problem (if we needed to)
would not have any external effect.

Thanks
--Jens

[1] http://openhms.sourceforge.net/rmiio
[2] http://openhms.sourceforge.net/rmiio/class_reference.html

[VOTE] Apache Geode release - 1.4.0 RC1

[Swapnil Bawaskar]

This is the first release candidate for Apache Geode, version 1.4.0.
Thanks to all the community members for their contributions to this
release!

*** Please download, test and vote by Friday, January 26, 1200 hrs
US Pacific. ***

It fixes 277 issues. release notes can be found at:
https://issues.apache.org/jira/projects/GEODE/versions/12341842

Note that we are voting upon the source tags: rel/v1.4.0.RC1
https://github.com/apache/geode/tree/rel/v1.4.0.RC1
https://github.com/apache/geode-examples/tree/rel/v1.4.0.RC1

Commit ID:
7dbb280fbff0320efd427c5553a6e666328bc50c (geode)
ced35f88173b04ac8e104b9cae10cc38289675fa (geode-examples)

Source and binary files:
https://dist.apache.org/repos/dist/dev/geode/1.4.0.RC1

Maven staging repo:
https://repository.apache.org/content/repositories/orgapachegeode-1036


Geode's KEYS file containing PGP keys we use to sign the release:
https://github.com/apache/geode/blob/develop/KEYS

Release Signed with Key: pub 4096R/18F902DB 2016-04-07
Fingerprint: E1B1 ABE3 4753 E7BA 8097 4285 8F8F 2BCC 18F9 02DB

Re: build is broken

[Mark Bretl]

I agree with Kirk. A simple notice to the dev list to say what dependency
and why it is being added, this would also be an opportunity to make sure
the dependency is not using a Cat-X license [1].

--Mark

[1] https://www.apache.org/legal/resolved.html#category-x

On Tue, Jan 23, 2018 at 2:09 PM, Kirk Lund  wrote:

> Discussing the addition of new dependencies is something the community
> should be doing on the dev-list regardless of how many git repos we have.
> It's part of the Apache Way that we are beholden to as an Apache project.
>
> On Tue, Jan 23, 2018 at 11:25 AM, Udo Kohlmeyer 
> wrote:
>
> > This really just another reason that we should be pushing for the
> > separation of Geode into different (independent) modules with potentially
> > separate repos.
> > This way modules can advance and iterate faster without having to be
> > influenced by external factors causing unrelated failures.
> >
> > I would push for this, rather than having to slow the whole development
> > process down, whilst waiting for consensus on libraries used for external
> > tooling.
> >
> > --Udo
> >
> > On Tue, Jan 23, 2018 at 11:15 AM, Kirk Lund  wrote:
> >
> > > I have it working now. My IntelliJ project was using JDK1.8.0_66 for
> > Gradle
> > > which prevented updating of dependencies which then caused compilation
> in
> > > IntelliJ to fail on that new dependency.
> > >
> > > This doesn't resolve my main issue here which is this: *all new
> > > dependencies should be discussed on the dev list before introducing
> them
> > in
> > > a commit*. Can you please point me at an email that proposes adding
> this
> > > dependency?
> > >
> > > On Tue, Jan 23, 2018 at 10:10 AM, Anthony Baker 
> > wrote:
> > >
> > > > I just ran `gradle clean build` from HEAD and did not get any errors.
> > > >
> > > > > On Jan 23, 2018, at 10:08 AM, Jens Deppe 
> wrote:
> > > > >
> > > > > The dependency is the rmi-io library. It provides functionality for
> > > > > streaming data over RMI.
> > > > >
> > > > > It was added a few weeks ago, so I'm not sure why your build is
> > > breaking
> > > > > now.
> > > > >
> > > > > --Jens
> > > > >
> > > > > On Tue, Jan 23, 2018 at 9:41 AM, Kirk Lund 
> wrote:
> > > > >
> > > > >> Looks like this dependency is Jackcess (used for reading and
> writing
> > > MS
> > > > >> Access databases). I cannot find any discussion about adding this
> > > > >> dependency on the dev-list. Why does Geode need to talk to MS
> Access
> > > > >> databases for Cluster Configuration?
> > > > >>
> > > > >> It does have an Apache 2.0 license, but I think there are two
> > problems
> > > > with
> > > > >> adding Jackcess: 1) all new dependencies should be discussed on
> > > dev-list
> > > > >> BEFORE adding them to Geode, 2) we shouldn't need to talk to MS
> > Access
> > > > for
> > > > >> Cluster Configuration.
> > > > >>
> > > > >> On Tue, Jan 23, 2018 at 9:30 AM, Kirk Lund 
> > wrote:
> > > > >>
> > > > >>> I just tried to compile Geode develop.
> ClusterConfigurationService
> > > has
> > > > >> two
> > > > >>> broken imports that don't exist:
> > > > >>>
> > > > >>> import com.healthmarketscience.rmiio.RemoteInputStream;
> > > > >>> import com.healthmarketscience.rmiio.RemoteInputStreamClient;
> > > > >>>
> > > > >>> Was there a new dependency added to Geode?
> > > > >>>
> > > > >>
> > > >
> > > >
> > >
> >
>

Failed: jinmeiliao/geode#189 (expiry - 0b1f73d)

[Travis CI]

Build Update for jinmeiliao/geode
-

Build: #189
Status: Failed

Duration: 15 minutes and 43 seconds
Commit: 0b1f73d (expiry)
Author: Jinmei Liao
Message: GEODE-3915: use ClassName type for cache-loader, writer and listeners

Signed-off-by: Patrick Rhomberg 

View the changeset: 
https://github.com/jinmeiliao/geode/compare/75f2401527b9^...0b1f73df87d1

View the full build log and details: 
https://travis-ci.org/jinmeiliao/geode/builds/332117201?utm_source=email_medium=notification

--

You can configure recipients for build notifications in your .travis.yml file. 
See https://docs.travis-ci.com/user/notifications

Geode enterprise support

[Vladi Polonsky]

Hi

Is there a company giving enterprise support of Apache Geode? (not Gemfire)

Regards
Vladi
This message and the information contained herein is proprietary and 
confidential and subject to the Amdocs policy statement,

you may review at https://www.amdocs.com/about/email-disclaimer 

Re: build is broken

[Udo Kohlmeyer]

This really just another reason that we should be pushing for the
separation of Geode into different (independent) modules with potentially
separate repos.
This way modules can advance and iterate faster without having to be
influenced by external factors causing unrelated failures.

I would push for this, rather than having to slow the whole development
process down, whilst waiting for consensus on libraries used for external
tooling.

--Udo

On Tue, Jan 23, 2018 at 11:15 AM, Kirk Lund  wrote:

> I have it working now. My IntelliJ project was using JDK1.8.0_66 for Gradle
> which prevented updating of dependencies which then caused compilation in
> IntelliJ to fail on that new dependency.
>
> This doesn't resolve my main issue here which is this: *all new
> dependencies should be discussed on the dev list before introducing them in
> a commit*. Can you please point me at an email that proposes adding this
> dependency?
>
> On Tue, Jan 23, 2018 at 10:10 AM, Anthony Baker  wrote:
>
> > I just ran `gradle clean build` from HEAD and did not get any errors.
> >
> > > On Jan 23, 2018, at 10:08 AM, Jens Deppe  wrote:
> > >
> > > The dependency is the rmi-io library. It provides functionality for
> > > streaming data over RMI.
> > >
> > > It was added a few weeks ago, so I'm not sure why your build is
> breaking
> > > now.
> > >
> > > --Jens
> > >
> > > On Tue, Jan 23, 2018 at 9:41 AM, Kirk Lund  wrote:
> > >
> > >> Looks like this dependency is Jackcess (used for reading and writing
> MS
> > >> Access databases). I cannot find any discussion about adding this
> > >> dependency on the dev-list. Why does Geode need to talk to MS Access
> > >> databases for Cluster Configuration?
> > >>
> > >> It does have an Apache 2.0 license, but I think there are two problems
> > with
> > >> adding Jackcess: 1) all new dependencies should be discussed on
> dev-list
> > >> BEFORE adding them to Geode, 2) we shouldn't need to talk to MS Access
> > for
> > >> Cluster Configuration.
> > >>
> > >> On Tue, Jan 23, 2018 at 9:30 AM, Kirk Lund  wrote:
> > >>
> > >>> I just tried to compile Geode develop. ClusterConfigurationService
> has
> > >> two
> > >>> broken imports that don't exist:
> > >>>
> > >>> import com.healthmarketscience.rmiio.RemoteInputStream;
> > >>> import com.healthmarketscience.rmiio.RemoteInputStreamClient;
> > >>>
> > >>> Was there a new dependency added to Geode?
> > >>>
> > >>
> >
> >
>

Re: build is broken

[Kirk Lund]

I have it working now. My IntelliJ project was using JDK1.8.0_66 for Gradle
which prevented updating of dependencies which then caused compilation in
IntelliJ to fail on that new dependency.

This doesn't resolve my main issue here which is this: *all new
dependencies should be discussed on the dev list before introducing them in
a commit*. Can you please point me at an email that proposes adding this
dependency?

On Tue, Jan 23, 2018 at 10:10 AM, Anthony Baker  wrote:

> I just ran `gradle clean build` from HEAD and did not get any errors.
>
> > On Jan 23, 2018, at 10:08 AM, Jens Deppe  wrote:
> >
> > The dependency is the rmi-io library. It provides functionality for
> > streaming data over RMI.
> >
> > It was added a few weeks ago, so I'm not sure why your build is breaking
> > now.
> >
> > --Jens
> >
> > On Tue, Jan 23, 2018 at 9:41 AM, Kirk Lund  wrote:
> >
> >> Looks like this dependency is Jackcess (used for reading and writing MS
> >> Access databases). I cannot find any discussion about adding this
> >> dependency on the dev-list. Why does Geode need to talk to MS Access
> >> databases for Cluster Configuration?
> >>
> >> It does have an Apache 2.0 license, but I think there are two problems
> with
> >> adding Jackcess: 1) all new dependencies should be discussed on dev-list
> >> BEFORE adding them to Geode, 2) we shouldn't need to talk to MS Access
> for
> >> Cluster Configuration.
> >>
> >> On Tue, Jan 23, 2018 at 9:30 AM, Kirk Lund  wrote:
> >>
> >>> I just tried to compile Geode develop. ClusterConfigurationService has
> >> two
> >>> broken imports that don't exist:
> >>>
> >>> import com.healthmarketscience.rmiio.RemoteInputStream;
> >>> import com.healthmarketscience.rmiio.RemoteInputStreamClient;
> >>>
> >>> Was there a new dependency added to Geode?
> >>>
> >>
>
>

Geode unit tests completed in 'develop/FlakyTest' with non-zero exit code

[apachegeodeci]

Pipeline results can be found at:

Concourse: 
https://concourse.apachegeode-ci.info/teams/main/pipelines/develop/jobs/FlakyTest/builds/142

Re: build is broken

[Anthony Baker]

I just ran `gradle clean build` from HEAD and did not get any errors.

> On Jan 23, 2018, at 10:08 AM, Jens Deppe  wrote:
> 
> The dependency is the rmi-io library. It provides functionality for
> streaming data over RMI.
> 
> It was added a few weeks ago, so I'm not sure why your build is breaking
> now.
> 
> --Jens
> 
> On Tue, Jan 23, 2018 at 9:41 AM, Kirk Lund  wrote:
> 
>> Looks like this dependency is Jackcess (used for reading and writing MS
>> Access databases). I cannot find any discussion about adding this
>> dependency on the dev-list. Why does Geode need to talk to MS Access
>> databases for Cluster Configuration?
>> 
>> It does have an Apache 2.0 license, but I think there are two problems with
>> adding Jackcess: 1) all new dependencies should be discussed on dev-list
>> BEFORE adding them to Geode, 2) we shouldn't need to talk to MS Access for
>> Cluster Configuration.
>> 
>> On Tue, Jan 23, 2018 at 9:30 AM, Kirk Lund  wrote:
>> 
>>> I just tried to compile Geode develop. ClusterConfigurationService has
>> two
>>> broken imports that don't exist:
>>> 
>>> import com.healthmarketscience.rmiio.RemoteInputStream;
>>> import com.healthmarketscience.rmiio.RemoteInputStreamClient;
>>> 
>>> Was there a new dependency added to Geode?
>>> 
>> 

Re: build is broken

[Jens Deppe]

The dependency is the rmi-io library. It provides functionality for
streaming data over RMI.

It was added a few weeks ago, so I'm not sure why your build is breaking
now.

--Jens

On Tue, Jan 23, 2018 at 9:41 AM, Kirk Lund  wrote:

> Looks like this dependency is Jackcess (used for reading and writing MS
> Access databases). I cannot find any discussion about adding this
> dependency on the dev-list. Why does Geode need to talk to MS Access
> databases for Cluster Configuration?
>
> It does have an Apache 2.0 license, but I think there are two problems with
> adding Jackcess: 1) all new dependencies should be discussed on dev-list
> BEFORE adding them to Geode, 2) we shouldn't need to talk to MS Access for
> Cluster Configuration.
>
> On Tue, Jan 23, 2018 at 9:30 AM, Kirk Lund  wrote:
>
> > I just tried to compile Geode develop. ClusterConfigurationService has
> two
> > broken imports that don't exist:
> >
> > import com.healthmarketscience.rmiio.RemoteInputStream;
> > import com.healthmarketscience.rmiio.RemoteInputStreamClient;
> >
> > Was there a new dependency added to Geode?
> >
>

Re: build is broken

[Kirk Lund]

Looks like this dependency is Jackcess (used for reading and writing MS
Access databases). I cannot find any discussion about adding this
dependency on the dev-list. Why does Geode need to talk to MS Access
databases for Cluster Configuration?

It does have an Apache 2.0 license, but I think there are two problems with
adding Jackcess: 1) all new dependencies should be discussed on dev-list
BEFORE adding them to Geode, 2) we shouldn't need to talk to MS Access for
Cluster Configuration.

On Tue, Jan 23, 2018 at 9:30 AM, Kirk Lund  wrote:

> I just tried to compile Geode develop. ClusterConfigurationService has two
> broken imports that don't exist:
>
> import com.healthmarketscience.rmiio.RemoteInputStream;
> import com.healthmarketscience.rmiio.RemoteInputStreamClient;
>
> Was there a new dependency added to Geode?
>

build is broken

[Kirk Lund]

I just tried to compile Geode develop. ClusterConfigurationService has two
broken imports that don't exist:

import com.healthmarketscience.rmiio.RemoteInputStream;
import com.healthmarketscience.rmiio.RemoteInputStreamClient;

Was there a new dependency added to Geode?