Jenkins build is back to normal : Geode-nightly-flaky #184

[Apache Jenkins Server]

See 

Build failed in Jenkins: Geode-nightly #1030

[Apache Jenkins Server]

See 


Changes:

[github] GEODE-3539: enhance rule to start locator joining other locators 
(#1104)

[metatype] GEODE-4023: Add precheckin tests to pipeline.

[github] GEODE-1683: fix ClientAuthorizationDUnit test failures (#1106)

[dbarnes] GEODE-1897 Docs for gfsh eviction, minor correction.

[bschuchardt] GEODE-3923 Provide whitelist/blacklist capability for java 
serialization

[bschuchardt] GEODE-4000: The serializerClass is actually introduced in 1.4 not 
1.3

[bschuchardt] GEODE-3923 Provide whitelist/blacklist capability for java 
serialization

--
[...truncated 567.68 KB...]
at org.jgroups.protocols.UDP.start(UDP.java:266)
at org.jgroups.stack.ProtocolStack.startStack(ProtocolStack.java:966)
at org.jgroups.JChannel.startStack(JChannel.java:889)
at org.jgroups.JChannel._preConnect(JChannel.java:553)
at org.jgroups.JChannel.connect(JChannel.java:288)
at org.jgroups.JChannel.connect(JChannel.java:279)
at 
org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.start(JGroupsMessenger.java:345)
... 63 more

org.apache.geode.distributed.DistributedSystemDUnitTest > testUDPPortRange 
FAILED
org.apache.geode.SystemConnectException: unable to create jgroups channel
at 
org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.start(JGroupsMessenger.java:349)
at 
org.apache.geode.distributed.internal.membership.gms.Services.start(Services.java:157)
at 
org.apache.geode.distributed.internal.membership.gms.GMSMemberFactory.newMembershipManager(GMSMemberFactory.java:107)
at 
org.apache.geode.distributed.internal.membership.MemberFactory.newMembershipManager(MemberFactory.java:91)
at 
org.apache.geode.distributed.internal.DistributionManager.(DistributionManager.java:1158)
at 
org.apache.geode.distributed.internal.DistributionManager.(DistributionManager.java:1207)
at 
org.apache.geode.distributed.internal.DistributionManager.create(DistributionManager.java:576)
at 
org.apache.geode.distributed.internal.InternalDistributedSystem.initialize(InternalDistributedSystem.java:741)
at 
org.apache.geode.distributed.internal.InternalDistributedSystem.newInstance(InternalDistributedSystem.java:334)
at 
org.apache.geode.distributed.internal.InternalDistributedSystem.newInstance(InternalDistributedSystem.java:320)
at 
org.apache.geode.distributed.internal.InternalDistributedSystem.newInstance(InternalDistributedSystem.java:314)
at 
org.apache.geode.distributed.DistributedSystem.connect(DistributedSystem.java:205)
at 
org.apache.geode.test.dunit.internal.JUnit4DistributedTestCase.getSystem(JUnit4DistributedTestCase.java:185)
at 
org.apache.geode.distributed.DistributedSystemDUnitTest.testUDPPortRange(DistributedSystemDUnitTest.java:256)

Caused by:
java.lang.Exception: failed to open a port in range 32769-32771
at 
org.jgroups.protocols.UDP.createMulticastSocketWithBindPort(UDP.java:503)
at org.jgroups.protocols.UDP.createSockets(UDP.java:348)
at org.jgroups.protocols.UDP.start(UDP.java:266)
at 
org.jgroups.stack.ProtocolStack.startStack(ProtocolStack.java:966)
at org.jgroups.JChannel.startStack(JChannel.java:889)
at org.jgroups.JChannel._preConnect(JChannel.java:553)
at org.jgroups.JChannel.connect(JChannel.java:288)
at org.jgroups.JChannel.connect(JChannel.java:279)
at 
org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.start(JGroupsMessenger.java:345)
... 13 more

java.lang.AssertionError: Suspicious strings were written to the log during 
this run.
Fix the strings or use IgnoredException.addIgnoredException to ignore.
---
Found suspect string in log4j at line 220

[fatal 2017/12/01 22:03:23.983 UTC  tid=0x13] Unexpected 
exception while booting membership services
org.apache.geode.SystemConnectException: unable to create jgroups channel
at 
org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.start(JGroupsMessenger.java:349)
at 
org.apache.geode.distributed.internal.membership.gms.Services.start(Services.java:157)
at 
org.apache.geode.distributed.internal.membership.gms.GMSMemberFactory.newMembershipManager(GMSMemberFactory.java:107)
at 
org.apache.geode.distributed.internal.membership.MemberFactory.newMembershipManager(MemberFactory.java:91)
at 
org.apache.geode.distributed.internal.DistributionManager.(DistributionManager.java:1158)
at 
org.apache.geode.distributed.internal.DistributionManager.(DistributionManager.java:1207)
at 
org.apache.geode

Re: DISCUSS: Deprecating and replacing current serializable string encodings

[Jacob Barrett]

On Fri, Dec 1, 2017 at 4:59 PM Dan Smith  wrote:

> I think I'm kinda with Mike on this one. The existing string format does
> seem pretty gnarly. But the complexity of implementing and testing all of
> the backwards compatibility transcoding that would be required in order to
> move to the new proposed format seems to be way more work with much more
> possibility for errors. Do we really expect people to be writing new
> clients that use DataSerializable? It hasn't happened yet, and we're
> working on a new protocol that uses protobuf right now.
>

Consider that any new clients written would have to implement all these
encodings. This is going to make writing new clients using the upcoming new
protocol laborious. The new protocol does not define object encoding, it
strictly defines message encoding. Objects sent over the protocol will have
to be serialized in some format, like PDX or data serializer. We could
alway develop a better serialization format than what we have now. If we
don't develop something new then we have to use the old. Wouldn't it be
nice if the new clients didn't have to deal with legacy encodings?

If the issue is really the complexity of serialization from the C++ client,
> maybe the C++ client could always write UTF-16 strings?
>

You can't assume that a client in one language will only be serializing
strings for it's own consumption. We have many people using strings in PDX
to transform between C++, .NET and Java.

The risk is high not to remove this debt. If I am developing a new Ruby
client I am forced to deal with all 4 of these encodings. Am I really going
to want to build a Ruby client for Geode, am I going to get these encodings
correct? I can tell you that getting them correct may be a challenge if the
current C++ client is any indication, it has a few incorrect assumptions in
its encoding of ASCII and modified UTF-8.

I am fine with a compromise that deprecates but doesn't remove the old
encodings for a few releases. This would give time for users to update. New
clients written would not be be able to read this old data but could read
and write new data.



-Jake

Re: DISCUSS: Deprecating and replacing current serializable string encodings

[Michael Stolz]

It could always WRITE UTF-16 strings, but it will still need to be able to
read the others.

--
Mike Stolz
Principal Engineer, GemFire Product Lead
Mobile: +1-631-835-4771

On Fri, Dec 1, 2017 at 7:58 PM, Dan Smith  wrote:

> I think I'm kinda with Mike on this one. The existing string format does
> seem pretty gnarly. But the complexity of implementing and testing all of
> the backwards compatibility transcoding that would be required in order to
> move to the new proposed format seems to be way more work with much more
> possibility for errors. Do we really expect people to be writing new
> clients that use DataSerializable? It hasn't happened yet, and we're
> working on a new protocol that uses protobuf right now.
>
> If the issue is really the complexity of serialization from the C++ client,
> maybe the C++ client could always write UTF-16 strings?
>
> -Dan
>
> On Fri, Dec 1, 2017 at 4:17 PM, Michael Stolz  wrote:
>
> > My opinion is that risk/reward on this on is not worth it
> >
> > --
> > Mike Stolz
> > Principal Engineer - Gemfire Product Manager
> > Mobile: 631-835-4771
> >
> > On Dec 1, 2017 5:19 PM, "Jacob Barrett"  wrote:
> >
> > > On Fri, Dec 1, 2017 at 1:48 PM Michael Stolz 
> wrote:
> > >
> > > > There also would likely be Disk Stores that would need to be
> converted.
> > > > That would be real ugly too.
> > >
> > >
> > > Disk store could transcode each entry on demand or all at once on first
> > > load. Not saying it will be easy but progress rarely is.
> > >
> > > -Jake
> > >
> >
>

Re: DISCUSS: Deprecating and replacing current serializable string encodings

[Dan Smith]

I think I'm kinda with Mike on this one. The existing string format does
seem pretty gnarly. But the complexity of implementing and testing all of
the backwards compatibility transcoding that would be required in order to
move to the new proposed format seems to be way more work with much more
possibility for errors. Do we really expect people to be writing new
clients that use DataSerializable? It hasn't happened yet, and we're
working on a new protocol that uses protobuf right now.

If the issue is really the complexity of serialization from the C++ client,
maybe the C++ client could always write UTF-16 strings?

-Dan

On Fri, Dec 1, 2017 at 4:17 PM, Michael Stolz  wrote:

> My opinion is that risk/reward on this on is not worth it
>
> --
> Mike Stolz
> Principal Engineer - Gemfire Product Manager
> Mobile: 631-835-4771
>
> On Dec 1, 2017 5:19 PM, "Jacob Barrett"  wrote:
>
> > On Fri, Dec 1, 2017 at 1:48 PM Michael Stolz  wrote:
> >
> > > There also would likely be Disk Stores that would need to be converted.
> > > That would be real ugly too.
> >
> >
> > Disk store could transcode each entry on demand or all at once on first
> > load. Not saying it will be easy but progress rarely is.
> >
> > -Jake
> >
>

Re: Concourse pipeline

[Sean Goller]

Mail should be sent out only on failures, so I would hope they'd be less
frequent than Travis. I think Travis also processes PRs which contributes
to the noise.

-S.

On Fri, Dec 1, 2017 at 4:26 PM, Greg Chase  wrote:

> I wonder - do we want a separate email alias for these messages if they are
> going to be even more frequent than Travis?
>
> On Fri, Dec 1, 2017 at 3:24 PM, Sean Goller  wrote:
>
> > Hi everyone,
> >   If you haven't already heard, we have a new concourse pipeline
> > infrastructure up and running at https://concourse.apachegeode-ci.info/
> .
> > When these jobs fail, email will be sent to the dev list, so be aware! :)
> >
> > -Sean.
> >
>

Re: Concourse pipeline

[Greg Chase]

I wonder - do we want a separate email alias for these messages if they are
going to be even more frequent than Travis?

On Fri, Dec 1, 2017 at 3:24 PM, Sean Goller  wrote:

> Hi everyone,
>   If you haven't already heard, we have a new concourse pipeline
> infrastructure up and running at https://concourse.apachegeode-ci.info/ .
> When these jobs fail, email will be sent to the dev list, so be aware! :)
>
> -Sean.
>

Re: DISCUSS: Deprecating and replacing current serializable string encodings

[Michael Stolz]

My opinion is that risk/reward on this on is not worth it

--
Mike Stolz
Principal Engineer - Gemfire Product Manager
Mobile: 631-835-4771

On Dec 1, 2017 5:19 PM, "Jacob Barrett"  wrote:

> On Fri, Dec 1, 2017 at 1:48 PM Michael Stolz  wrote:
>
> > There also would likely be Disk Stores that would need to be converted.
> > That would be real ugly too.
>
>
> Disk store could transcode each entry on demand or all at once on first
> load. Not saying it will be easy but progress rarely is.
>
> -Jake
>

Concourse pipeline

[Sean Goller]

Hi everyone,
  If you haven't already heard, we have a new concourse pipeline
infrastructure up and running at https://concourse.apachegeode-ci.info/ .
When these jobs fail, email will be sent to the dev list, so be aware! :)

-Sean.

[Spring CI] Spring Data GemFire > Nightly-ApacheGeode > #753 was SUCCESSFUL (with 2264 tests)

[Spring CI]

---
Spring Data GemFire > Nightly-ApacheGeode > #753 was successful.
---
Scheduled
2266 tests in total.

https://build.spring.io/browse/SGF-NAG-753/





--
This message is automatically generated by Atlassian Bamboo

Re: DISCUSS: Deprecating and replacing current serializable string encodings

[Jacob Barrett]

On Fri, Dec 1, 2017 at 1:48 PM Michael Stolz  wrote:

> There also would likely be Disk Stores that would need to be converted.
> That would be real ugly too.


Disk store could transcode each entry on demand or all at once on first
load. Not saying it will be easy but progress rarely is.

-Jake

Re: DISCUSS: Deprecating and replacing current serializable string encodings

[Jacob Barrett]

On Fri, Dec 1, 2017 at 1:25 PM Bruce Schuchardt 
wrote:

> I'm wondering how we would maintain backward compatibility with this
> change?  Geode accepts serialized data from a client and keeps it in
> serialized form and might transmit this serialized data to an older
> version peer or client.  An older peer or client wouldn't be able to
> handle a new string encoding if it tried to access and deserialize the
> data.
>

While not efficient for long term it would be sufficient for rolling
upgrades to have the client/server layer transcode the serialized form for
older clients.

Geode PR cleanup

[Jens Deppe]

Hey all,

Could we do some github pull request housekeeping please? Specifically:

   - If you have an open PR that you know is not going to be merged, please
   close it.
   - If you have reviewed a PR and requested changes which have not been
   made, please bump the PR and see if the original submitter is still working
   it. If there is no response to a bump after a week, I think we should close
   the PR.

Thanks
--Jens

Re: DISCUSS: Deprecating and replacing current serializable string encodings

[Michael Stolz]

There also would likely be Disk Stores that would need to be converted.
That would be real ugly too.

--
Mike Stolz
Principal Engineer - Gemfire Product Manager
Mobile: 631-835-4771

On Dec 1, 2017 4:25 PM, "Bruce Schuchardt"  wrote:

> I'm wondering how we would maintain backward compatibility with this
> change?  Geode accepts serialized data from a client and keeps it in
> serialized form and might transmit this serialized data to an older version
> peer or client.  An older peer or client wouldn't be able to handle a new
> string encoding if it tried to access and deserialize the data.
>
> On 12/1/17 1:13 PM, Jacob Barrett wrote:
>
>> So what I would like to propose is that we deprecate all these methods and
>> replace them with standard UTF-8 prefixed with uint64 length. It is
>> preferable that the length be run length encoded to reduce the overhead of
>> encoding small strings. Why such a large length, well consider that
>> different languages have different limits as well as Java stores strings
>> internally as UTF-16.
>>
>
>

Re: DISCUSS: Deprecating and replacing current serializable string encodings

[Bruce Schuchardt]

I'm wondering how we would maintain backward compatibility with this 
change?  Geode accepts serialized data from a client and keeps it in 
serialized form and might transmit this serialized data to an older 
version peer or client.  An older peer or client wouldn't be able to 
handle a new string encoding if it tried to access and deserialize the data.


On 12/1/17 1:13 PM, Jacob Barrett wrote:

So what I would like to propose is that we deprecate all these methods and
replace them with standard UTF-8 prefixed with uint64 length. It is
preferable that the length be run length encoded to reduce the overhead of
encoding small strings. Why such a large length, well consider that
different languages have different limits as well as Java stores strings
internally as UTF-16.

DISCUSS: Deprecating and replacing current serializable string encodings

[Jacob Barrett]

The current data serializer support 4 different encodings (in addition to
the 3 or 4 other encodings I have found elsewhere in Geode). It supports
encoding ASCII prefixed with an uint16 length (64K), ASCII prefixed with an
uint32 length, Java Modified UTF-8 with unit16 length (64k) and finally
UTF-16 with uint32 length (count of UTF-16 code units, not bytes). When
serializing a string it is first scanned for any non-ascii characters and
the number of bytes a modified UTF-8 encoding would require is calculated.
If the encoded length is equal to the original length then it is ASCII and
if the length is less than 2^16 then it uses the 16-bit length version
otherwise the 32-bit length version. If the encoded length is greater than
original but less than 2^16 then modified UTF-8 is used with 16-bit length
otherwise UTF16 and 32-bit length is used.

When working with non-Java clients dealing with modified UTF-8 requires
conversion between it and standard UTF-8 as modified UTF-8 encodes the NULL
character in two bytes. Only Java's DataInput/Output and JNI use this
encoding and ti was intended for internal and Java serialization only. The
StreamReader/Writer use standard UTF-8. Since our serialization expects
modified UTF-8 strings to be prefixed with at 16-bit length care has to be
taken to calculate the modified length up front (or seek the buffer and
rewrite the length if nulls are encountered). Since the modified length may
vary from the standard length care must then be take to make sure the if
the string must be truncated to fit in the 16bit limit that character is
not truncated in a multibyte sequence.

Encoding in UTF-16 isn't all that bad except that it is mostly wasted space
when strings are ASCII. There are no real encoding issues between languages
since most support it or use it their internal representation, like Java
does. But we are talking about serialization here and typically space is
the constraint. Most latin characters are low enough in the basic plain to
be encoded in 2 bytes of UTF-8 and take up no more space than the UTF-16
encoded version. Other characters will take up more space.

Since we took the care to optimize ASCII one can assume we figure out that
ASCII was our most common character sets. Regardless of the correctness of
this assertion it makes no sense to treat ASCII and UTF-8 streams
differently as ASCII can be fully encoded byte for byte in UTF-8 without
any overhead.

So what I would like to propose is that we deprecate all these methods and
replace them with standard UTF-8 prefixed with uint64 length. It is
preferable that the length be run length encoded to reduce the overhead of
encoding small strings. Why such a large length, well consider that
different languages have different limits as well as Java stores strings
internally as UTF-16.

A java UTF-16 string has max length of 2^31-1, encoded in UTF-8 it would
have a maximum, though highly improbably, length of 2^33-1. Serializing as
UTF-8 with a uint32 length limits the max Java string length to 2^29−1
or 536870911 UTF-16 code points. This is probably a reasonable limitation
but we have the technology to do better. ;) Since the server is Java it is
reasonable to limit the max string length we serialize consistent therefore
we need at least 33 bits of length.

For reference a C++11 std::basic_string has the max capacity that is
platform dependent but on 64bit linux and GCC it is 2^63-1. The
basic_string can be UTF-8, UTF-16 or UTF-32

The important part of this proposal is to convert everything to using
standard UTF-8 and deprecate all the other methods. I would ask that we
drop the other methods completely at the next major release. Not having to
implement 4 encodings in each of our clients will help development of new
clients. Not having to translate between standards and non standards string
types will help performance and reduce coding errors. All the other string
encodings I have found should be handled in the new protocol we are working
on, which is now using standard UTF-8, and are therefore outside the scope
of this proposal and discussion.

-Jake

Re: Apache geode nightly failures

[Dick Cavender]

I just blacklisted H34 but some 12 days back the nightly ran fine on H34
after blacklisting H12 and H14 as not being reliable. Now H34 has become
the same.

This is the complete list of specific machines and the same probably
happened when we added H2.
&&!H2&&!H12&&!H14&&!H34

Smells like there are project/workspaces out there tainting these
machine(s) when they run or complete. Let's see what machine the next
nightly build lands while pushing towards getting some reliable history on
concourse and transition off Jenkins.

Reminder that the geode concourse CI can be viewed here:
https://concourse.apachegeode-ci.info/teams/main/pipelines/develop

-Dick


On Fri, Dec 1, 2017 at 9:29 AM, Jason Huynh  wrote:

> Has anyone been getting emails for the nightly builds and failures (My last
> build email was 1027 and I think the latest is 1030)?  It looks like the
> last few have been a mess with Out of Memory exceptions and I think they
> were all ran on H34.  Should that machine be black listed?  If so, would
> someone with permissions be able to do so?
>

Apache geode nightly failures

[Jason Huynh]

Has anyone been getting emails for the nightly builds and failures (My last
build email was 1027 and I think the latest is 1030)?  It looks like the
last few have been a mess with Out of Memory exceptions and I think they
were all ran on H34.  Should that machine be black listed?  If so, would
someone with permissions be able to do so?

Re: [DISCUSS] changes to registerInterest API

[Anthony Baker]

I think Dan’s suggestion clarifies the intent.

Anthony


> On Nov 30, 2017, at 3:54 PM, Dan Smith  wrote:
> 
> I think it should be registerInterestAll(Iterablekeys) to mirror
> Collection.addAll.
> 
> I could easily see a user creating their own Tuple class or using one that
> happens to also implement Iterable as their key.
> 
> We're not doing a very good job of discouraging people to use iterable
> objects as their key if they work everywhere else and then break with this
> one API.
> 
> -Dan
> 
> On Thu, Nov 30, 2017 at 2:45 PM, John Blum  wrote:
> 
>> No, no... good question.
>> 
>> I just think it would be wiser if users created a single, CompositeKey
>> class type, with properly implements equals and hashCode methods, as I
>> pointed out.
>> 
>> I don't see any advantage in using a java.util.Collection as a key over
>> implementing a CompositeKey type.
>> 
>> As such, anything we can do to discourage users from using Collection types
>> as a key, I think is a good thing.
>> 
>> 
>> On Thu, Nov 30, 2017 at 2:35 PM, Jason Huynh  wrote:
>> 
>>> Yeah I am not sure if anyone does it,and I don't think it would be a good
>>> idea to use a collection as a key but just thought I'd ask the
>> question...
>>> 
>>> On Thu, Nov 30, 2017 at 2:33 PM John Blum  wrote:
>>> 
 For instance, this came up recently...
 
 
 https://stackoverflow.com/questions/46551278/gemfire-
>>> composite-key-pojo-as-gemfire-key
 
 I have seen other similar posts too!
 
 
 
 On Thu, Nov 30, 2017 at 2:30 PM, John Blum  wrote:
 
> Does anyone actually do this in practice?  If so, yikes!
> 
> Even if the List is immutable, the elements may not be, so using a
>> List
 as
> a key starts to open 1 up to a lot of problems.
> 
> As others have pointed out in SO and other channels, information
>> should
> not be kept in the key.
> 
> It is perfect fine to have a "Composite" Key, but then define a
> CompositeKey class type with properly implemented equals(:Object) and
> hashCode():int methods.
> 
> For the most part, Keys should really only ever be simple Scalar
>> values
> (e.g. Long, String, etc).
> 
> -j
> 
> 
> 
> 
> On Thu, Nov 30, 2017 at 2:25 PM, Jason Huynh 
>>> wrote:
> 
>> I started work on the following plan:
>> - deprecate current "ALL_KEYS" and List passing behavior in
>> registerInterest
>> ()
>> - add registerInterestForAllKeys();
>> - add registerInterest(T... keys)
>> - add registerInterest(Iterablekeys)
>> 
>> I might be missing something here but:
>> With the addition of registerInterest(Iterable keys), I think we
 would
>> not be able to register interest a List as the key itself.  A list
>>> would
>> be
>> iterated over due to the addition of registerInterest(Iterable
>>> keys).
>> A
>> list in a list would be passed into registerInterest and again be
 iterated
>> over.  I could change the newly created registerInterest call and
>> explicitly name it something else or are we ok with Iterables not
>>> being
>> able to be registered as individual keys.
>> 
>> 
>> 
>> 
>> 
>> On Mon, Nov 20, 2017 at 9:05 AM Kirk Lund  wrote:
>> 
>>> John's approach looks best for when you need to specify keys.
>>> 
>>> For ALL_KEYS, what about an API that doesn't require a token or
>> all
>> keys:
>>> 
>>> public void registerInterestForAllKeys();
>>> 
>>> On Fri, Nov 17, 2017 at 1:24 PM, Jason Huynh 
 wrote:
>>> 
 Thanks John for the clarification!
 
 On Fri, Nov 17, 2017 at 1:12 PM John Blum 
>>> wrote:
 
> This...
> 
>> The Iterable version would handle any collection type by
>>> having
>> the
 user
> pass
> in the iterator for the collection.
> 
> Is not correct.
> 
> The Collection interface itself "extends" the
>> java.lang.Iterable
> interface (see here...
> 
 https://docs.oracle.com/javase/8/docs/api/java/util/Collection.html
 under
> "*All
> Superinterfaces*").
> 
> Therefore a user can simply to this...
> 
> *List* keys = ...
> 
> region.registerInterest(keys); *// calls the
> Region.registerInterest(:Iterable) method.*
> 
> Alternatively, this would also be allowed...
> 
> *Set* keys = ...
> 
> region.registerInterest(keys);
> 
> 
> On Fri, Nov 17, 2017 at 11:44 AM, Jason Huynh <
>>> jhu...@pivotal.io>
>>> wrote:
> 
>> Current idea is to:
>> - deprecate current "ALL_KEYS" and List passing behavior in
>> registerInterest()
>> - add registerInterestAllKeys();
>> - add registerInterest(T... keys) and
 registerIntere