Re: [DISCUSS] Hadoop3 support target?

[Josh Elser]

On 12/5/17 6:43 PM, Christopher wrote:

I was wondering about Hadoop 3 shading and whether that would help us. It
would be really nice if it could, or if there was some other class path
solution that was easy.

I think there are two major issues in this thread. The first is the API
problems. The second is the Hadoop 3 support. They are related, but I think
quickly dealing with the API issues can clarify what our options are for
Hadoop 3.


In the spirit of trying to keep these issues separate (I think 
Christopher is correct)


https://github.com/apache/accumulo/pull/332

If we switch to using the new shaded jars from Hadoop, we can avoid 
coupling these issues at all. This comes with caveats as 3.0.0-beta1 is 
busted (https://issues.apache.org/jira/browse/HADOOP-15058). Building a 
3.0.1-SNAPSHOT locally and using that let me run all of the unit tests 
which is promising.


Going to kick off the ITs and see how they fare.

Re: [DISCUSS] Hadoop3 support target?

[Christopher]

On Wed, Dec 6, 2017 at 3:37 PM Keith Turner  wrote:

> On Wed, Dec 6, 2017 at 2:28 PM, Christopher  wrote:
> > On Wed, Dec 6, 2017 at 2:10 PM Josh Elser  wrote:
> >
> >>
> >>
> >> On 12/6/17 2:06 PM, Christopher wrote:
> >> > On Wed, Dec 6, 2017 at 1:55 PM Keith Turner  wrote:
> >> >
> >> >> On Wed, Dec 6, 2017 at 1:43 PM, Josh Elser 
> wrote:
> >> >>>
> >> >>>
> >> >>> On 12/6/17 12:17 PM, Keith Turner wrote:
> >> 
> >>  On Wed, Dec 6, 2017 at 11:56 AM, Josh Elser
> >> wrote:
> >> >
> >> > Maybe a difference in interpretation:
> >> >
> >> > I was seeing 1a as being source-compatible still. My assumption
> was
> >> >> that
> >> > "Deprecate ClientConfiguration" meant that it would remain in the
> >> > codebase
> >> > -- "replace" as in "replace expected user invocation", not
> removal of
> >> >> the
> >> > old ClientConfiguration and addition of a new ClientConfig class.
> >> 
> >>  Ok, if we deprecate ClientConfiguration, leave it in 2.0, and drop
> the
> >>  extends from ClientConfiguration in 2.0.  Then I am not sure what
> the
> >>  benefit of introducing the new ClientConfig type is?
> >> >>>
> >> >>>
> >> >>> I read this as leaving the extends in ClientConfiguration and
> dropping
> >> >> that
> >> >>> in the new ClientConfig. Agree, I wouldn't see the point in changing
> >> the
> >> >>> parent class of ClientConfiguration (as that would break things).
> >> >>
> >> >>
> >> >> I don't think we can leave ClientConfiguration as deprecated and
> >> >> extending commons config in Accumulo 2.0.  This leaves commons config
> >> >> 1 in the API.
> >> >>
> >> >> Personally I am not in favor of dropping ClientConfiguration in 2.0,
> >> >> which is why I was in favor option b.
> >> >>
> >> >
> >> > In the absence of any further input from others, I'll follow along
> with
> >> > whatever you and Josh can agree on. Although I lean towards option
> 1.a, I
> >> > don't feel strongly about either option. We can also do a vote if
> neither
> >> > of you is able (or willing) to convince the other of your preference.
> >>
> >> I don't feel strongly enough either way to raise a stink. Color me
> >> surprised that Keith is the one to encourage quick removals from API :)
> >>
> >> If he's OK with it, I'm fine with it. I was trying to err on the side of
> >> less breakage.
> >>
> >
> > Sorry to rehash this, but your wording about Keith encouraging quick
> > removals confused me, so I want to make sure we're on the same page.
> >
> > As I understand it, Keith favors the option (1.b) to remove the
> superclass
> > from ClientConfiguration in 2.0, rather than remove the whole class. This
> > results in no source breakage as long as nobody used inherited methods
> from
> > the superclass or assigned (or cast, implicitly or explicitly) it to the
> > superclass type. It does have binary breakage in 2.0. So, I don't think
> > Keith is advocating for quick removals, but the opposite, which has
> > *complicated* compatibility expectations but possibly less breakage under
> > specific conditions.
>
> This email helped me see a subtle difference I did not see.  Its about
> whether code compiled against 1.9 (w/ not deprecation warnings) will
> work in 2.0.
>
> For option 1.a
>
>  * IF user compiles against 1.9 AND resolves all deprecation warnings then
> their
>code is guaranteed to work with 2.0.
>  * Can not have source compat for code that used ClientConfiguration when
> going
>from 1.6,1.7,1.8 to 2.0 and skipping step above.
>
> For option 1.b
>
>  * IF user compiles against 1.9 AND resolves all deprecation warnings then
> their
>code MAY work with 2.0. Using inherited commons config methods goes
>undetected.
>  * May have source compat for code that used ClientConfiguration when
> going from
>1.6,1.7,1.8 to 2.0.
>
> What prevents source compat for 1.6,1.7,1.8 to 2.0 for option 1.a is
> dropping ClientConfiguration type in 2.0
>
> Another option to consider is 1.c where the ClientConfiguration type
> is not dropped in 2.0.  The following would be done for 1.c
>
>  * Deprecate ZooKeeperInstance constuctor that takes commons config type
> in 1.9
>  * Deprecate ClientConfiguration type in 1.9
>  * Add ClientConfig type in 1.9
>  * Add ZooKeeperInstance constructor that takes ClientConfig type in 1.9
>  * Add ZooKeeperInstance constructor that takes ClientConfiguration type
> in 1.9
>  * Drop ZooKeeperInstance constuctor that takes commons config type in 2.0
>  * Drop extends commons config type from ClientConfiguration in 2.0.
>  * Do NOT drop ClientConfiguration type in 2.0
>
> For option 1.c
>
>  * IF user compiles against 1.9 AND resolves all deprecation warnings then
> their
>code is guaranteed to work with 2.0.
>  * May have source compat for code that used ClientConfiguration when
> going from
>1.6,1.7,1.8 to 2.0.
>
> >
> > The 

Re: [DISCUSS] Hadoop3 support target?

[Keith Turner]

On Wed, Dec 6, 2017 at 2:28 PM, Christopher  wrote:
> On Wed, Dec 6, 2017 at 2:10 PM Josh Elser  wrote:
>
>>
>>
>> On 12/6/17 2:06 PM, Christopher wrote:
>> > On Wed, Dec 6, 2017 at 1:55 PM Keith Turner  wrote:
>> >
>> >> On Wed, Dec 6, 2017 at 1:43 PM, Josh Elser  wrote:
>> >>>
>> >>>
>> >>> On 12/6/17 12:17 PM, Keith Turner wrote:
>> 
>>  On Wed, Dec 6, 2017 at 11:56 AM, Josh Elser
>> wrote:
>> >
>> > Maybe a difference in interpretation:
>> >
>> > I was seeing 1a as being source-compatible still. My assumption was
>> >> that
>> > "Deprecate ClientConfiguration" meant that it would remain in the
>> > codebase
>> > -- "replace" as in "replace expected user invocation", not removal of
>> >> the
>> > old ClientConfiguration and addition of a new ClientConfig class.
>> 
>>  Ok, if we deprecate ClientConfiguration, leave it in 2.0, and drop the
>>  extends from ClientConfiguration in 2.0.  Then I am not sure what the
>>  benefit of introducing the new ClientConfig type is?
>> >>>
>> >>>
>> >>> I read this as leaving the extends in ClientConfiguration and dropping
>> >> that
>> >>> in the new ClientConfig. Agree, I wouldn't see the point in changing
>> the
>> >>> parent class of ClientConfiguration (as that would break things).
>> >>
>> >>
>> >> I don't think we can leave ClientConfiguration as deprecated and
>> >> extending commons config in Accumulo 2.0.  This leaves commons config
>> >> 1 in the API.
>> >>
>> >> Personally I am not in favor of dropping ClientConfiguration in 2.0,
>> >> which is why I was in favor option b.
>> >>
>> >
>> > In the absence of any further input from others, I'll follow along with
>> > whatever you and Josh can agree on. Although I lean towards option 1.a, I
>> > don't feel strongly about either option. We can also do a vote if neither
>> > of you is able (or willing) to convince the other of your preference.
>>
>> I don't feel strongly enough either way to raise a stink. Color me
>> surprised that Keith is the one to encourage quick removals from API :)
>>
>> If he's OK with it, I'm fine with it. I was trying to err on the side of
>> less breakage.
>>
>
> Sorry to rehash this, but your wording about Keith encouraging quick
> removals confused me, so I want to make sure we're on the same page.
>
> As I understand it, Keith favors the option (1.b) to remove the superclass
> from ClientConfiguration in 2.0, rather than remove the whole class. This
> results in no source breakage as long as nobody used inherited methods from
> the superclass or assigned (or cast, implicitly or explicitly) it to the
> superclass type. It does have binary breakage in 2.0. So, I don't think
> Keith is advocating for quick removals, but the opposite, which has
> *complicated* compatibility expectations but possibly less breakage under
> specific conditions.

This email helped me see a subtle difference I did not see.  Its about
whether code compiled against 1.9 (w/ not deprecation warnings) will
work in 2.0.

For option 1.a

 * IF user compiles against 1.9 AND resolves all deprecation warnings then their
   code is guaranteed to work with 2.0.
 * Can not have source compat for code that used ClientConfiguration when going
   from 1.6,1.7,1.8 to 2.0 and skipping step above.

For option 1.b

 * IF user compiles against 1.9 AND resolves all deprecation warnings then their
   code MAY work with 2.0. Using inherited commons config methods goes
   undetected.
 * May have source compat for code that used ClientConfiguration when going from
   1.6,1.7,1.8 to 2.0.

What prevents source compat for 1.6,1.7,1.8 to 2.0 for option 1.a is
dropping ClientConfiguration type in 2.0

Another option to consider is 1.c where the ClientConfiguration type
is not dropped in 2.0.  The following would be done for 1.c

 * Deprecate ZooKeeperInstance constuctor that takes commons config type in 1.9
 * Deprecate ClientConfiguration type in 1.9
 * Add ClientConfig type in 1.9
 * Add ZooKeeperInstance constructor that takes ClientConfig type in 1.9
 * Add ZooKeeperInstance constructor that takes ClientConfiguration type in 1.9
 * Drop ZooKeeperInstance constuctor that takes commons config type in 2.0
 * Drop extends commons config type from ClientConfiguration in 2.0.
 * Do NOT drop ClientConfiguration type in 2.0

For option 1.c

 * IF user compiles against 1.9 AND resolves all deprecation warnings then their
   code is guaranteed to work with 2.0.
 * May have source compat for code that used ClientConfiguration when going from
   1.6,1.7,1.8 to 2.0.

>
> The other option (1.a) would result in no source or binary breakage, except
> if the deprecation warnings are ignored, in which case, there would be a
> source and binary breakage in 2.0. This option may also create churn if we
> make further improve the entry point to Accumulo in 2.0 or later. The
> potential for breakage is a 

Re: [DISCUSS] Hadoop3 support target?

[Christopher]

On Wed, Dec 6, 2017 at 2:10 PM Josh Elser  wrote:

>
>
> On 12/6/17 2:06 PM, Christopher wrote:
> > On Wed, Dec 6, 2017 at 1:55 PM Keith Turner  wrote:
> >
> >> On Wed, Dec 6, 2017 at 1:43 PM, Josh Elser  wrote:
> >>>
> >>>
> >>> On 12/6/17 12:17 PM, Keith Turner wrote:
> 
>  On Wed, Dec 6, 2017 at 11:56 AM, Josh Elser
> wrote:
> >
> > Maybe a difference in interpretation:
> >
> > I was seeing 1a as being source-compatible still. My assumption was
> >> that
> > "Deprecate ClientConfiguration" meant that it would remain in the
> > codebase
> > -- "replace" as in "replace expected user invocation", not removal of
> >> the
> > old ClientConfiguration and addition of a new ClientConfig class.
> 
>  Ok, if we deprecate ClientConfiguration, leave it in 2.0, and drop the
>  extends from ClientConfiguration in 2.0.  Then I am not sure what the
>  benefit of introducing the new ClientConfig type is?
> >>>
> >>>
> >>> I read this as leaving the extends in ClientConfiguration and dropping
> >> that
> >>> in the new ClientConfig. Agree, I wouldn't see the point in changing
> the
> >>> parent class of ClientConfiguration (as that would break things).
> >>
> >>
> >> I don't think we can leave ClientConfiguration as deprecated and
> >> extending commons config in Accumulo 2.0.  This leaves commons config
> >> 1 in the API.
> >>
> >> Personally I am not in favor of dropping ClientConfiguration in 2.0,
> >> which is why I was in favor option b.
> >>
> >
> > In the absence of any further input from others, I'll follow along with
> > whatever you and Josh can agree on. Although I lean towards option 1.a, I
> > don't feel strongly about either option. We can also do a vote if neither
> > of you is able (or willing) to convince the other of your preference.
>
> I don't feel strongly enough either way to raise a stink. Color me
> surprised that Keith is the one to encourage quick removals from API :)
>
> If he's OK with it, I'm fine with it. I was trying to err on the side of
> less breakage.
>

Sorry to rehash this, but your wording about Keith encouraging quick
removals confused me, so I want to make sure we're on the same page.

As I understand it, Keith favors the option (1.b) to remove the superclass
from ClientConfiguration in 2.0, rather than remove the whole class. This
results in no source breakage as long as nobody used inherited methods from
the superclass or assigned (or cast, implicitly or explicitly) it to the
superclass type. It does have binary breakage in 2.0. So, I don't think
Keith is advocating for quick removals, but the opposite, which has
*complicated* compatibility expectations but possibly less breakage under
specific conditions.

The other option (1.a) would result in no source or binary breakage, except
if the deprecation warnings are ignored, in which case, there would be a
source and binary breakage in 2.0. This option may also create churn if we
make further improve the entry point to Accumulo in 2.0 or later. The
potential for breakage is a bit more predictable and well-defined in this
option.

Re: [DISCUSS] Hadoop3 support target?

[Josh Elser]

On 12/6/17 2:06 PM, Christopher wrote:

On Wed, Dec 6, 2017 at 1:55 PM Keith Turner  wrote:


On Wed, Dec 6, 2017 at 1:43 PM, Josh Elser  wrote:



On 12/6/17 12:17 PM, Keith Turner wrote:


On Wed, Dec 6, 2017 at 11:56 AM, Josh Elser  wrote:


Maybe a difference in interpretation:

I was seeing 1a as being source-compatible still. My assumption was

that

"Deprecate ClientConfiguration" meant that it would remain in the
codebase
-- "replace" as in "replace expected user invocation", not removal of

the

old ClientConfiguration and addition of a new ClientConfig class.


Ok, if we deprecate ClientConfiguration, leave it in 2.0, and drop the
extends from ClientConfiguration in 2.0.  Then I am not sure what the
benefit of introducing the new ClientConfig type is?



I read this as leaving the extends in ClientConfiguration and dropping

that

in the new ClientConfig. Agree, I wouldn't see the point in changing the
parent class of ClientConfiguration (as that would break things).



I don't think we can leave ClientConfiguration as deprecated and
extending commons config in Accumulo 2.0.  This leaves commons config
1 in the API.

Personally I am not in favor of dropping ClientConfiguration in 2.0,
which is why I was in favor option b.



In the absence of any further input from others, I'll follow along with
whatever you and Josh can agree on. Although I lean towards option 1.a, I
don't feel strongly about either option. We can also do a vote if neither
of you is able (or willing) to convince the other of your preference.


I don't feel strongly enough either way to raise a stink. Color me 
surprised that Keith is the one to encourage quick removals from API :)


If he's OK with it, I'm fine with it. I was trying to err on the side of 
less breakage.

Re: [DISCUSS] Hadoop3 support target?

[Christopher]

On Wed, Dec 6, 2017 at 1:55 PM Keith Turner  wrote:

> On Wed, Dec 6, 2017 at 1:43 PM, Josh Elser  wrote:
> >
> >
> > On 12/6/17 12:17 PM, Keith Turner wrote:
> >>
> >> On Wed, Dec 6, 2017 at 11:56 AM, Josh Elser  wrote:
> >>>
> >>> Maybe a difference in interpretation:
> >>>
> >>> I was seeing 1a as being source-compatible still. My assumption was
> that
> >>> "Deprecate ClientConfiguration" meant that it would remain in the
> >>> codebase
> >>> -- "replace" as in "replace expected user invocation", not removal of
> the
> >>> old ClientConfiguration and addition of a new ClientConfig class.
> >>
> >> Ok, if we deprecate ClientConfiguration, leave it in 2.0, and drop the
> >> extends from ClientConfiguration in 2.0.  Then I am not sure what the
> >> benefit of introducing the new ClientConfig type is?
> >
> >
> > I read this as leaving the extends in ClientConfiguration and dropping
> that
> > in the new ClientConfig. Agree, I wouldn't see the point in changing the
> > parent class of ClientConfiguration (as that would break things).
>
>
> I don't think we can leave ClientConfiguration as deprecated and
> extending commons config in Accumulo 2.0.  This leaves commons config
> 1 in the API.
>
> Personally I am not in favor of dropping ClientConfiguration in 2.0,
> which is why I was in favor option b.
>

In the absence of any further input from others, I'll follow along with
whatever you and Josh can agree on. Although I lean towards option 1.a, I
don't feel strongly about either option. We can also do a vote if neither
of you is able (or willing) to convince the other of your preference.

Re: [DISCUSS] Hadoop3 support target?

[Keith Turner]

On Wed, Dec 6, 2017 at 1:43 PM, Josh Elser  wrote:
>
>
> On 12/6/17 12:17 PM, Keith Turner wrote:
>>
>> On Wed, Dec 6, 2017 at 11:56 AM, Josh Elser  wrote:
>>>
>>> Maybe a difference in interpretation:
>>>
>>> I was seeing 1a as being source-compatible still. My assumption was that
>>> "Deprecate ClientConfiguration" meant that it would remain in the
>>> codebase
>>> -- "replace" as in "replace expected user invocation", not removal of the
>>> old ClientConfiguration and addition of a new ClientConfig class.
>>
>> Ok, if we deprecate ClientConfiguration, leave it in 2.0, and drop the
>> extends from ClientConfiguration in 2.0.  Then I am not sure what the
>> benefit of introducing the new ClientConfig type is?
>
>
> I read this as leaving the extends in ClientConfiguration and dropping that
> in the new ClientConfig. Agree, I wouldn't see the point in changing the
> parent class of ClientConfiguration (as that would break things).


I don't think we can leave ClientConfiguration as deprecated and
extending commons config in Accumulo 2.0.  This leaves commons config
1 in the API.

Personally I am not in favor of dropping ClientConfiguration in 2.0,
which is why I was in favor option b.

Re: [DISCUSS] Hadoop3 support target?

[Christopher]

On Wed, Dec 6, 2017 at 1:43 PM Josh Elser  wrote:

>
>
> On 12/6/17 12:17 PM, Keith Turner wrote:
> > On Wed, Dec 6, 2017 at 11:56 AM, Josh Elser  wrote:
> >> Maybe a difference in interpretation:
> >>
> >> I was seeing 1a as being source-compatible still. My assumption was that
> >> "Deprecate ClientConfiguration" meant that it would remain in the
> codebase
> >> -- "replace" as in "replace expected user invocation", not removal of
> the
> >> old ClientConfiguration and addition of a new ClientConfig class.
> > Ok, if we deprecate ClientConfiguration, leave it in 2.0, and drop the
> > extends from ClientConfiguration in 2.0.  Then I am not sure what the
> > benefit of introducing the new ClientConfig type is?
>
> I read this as leaving the extends in ClientConfiguration and dropping
> that in the new ClientConfig. Agree, I wouldn't see the point in
> changing the parent class of ClientConfiguration (as that would break
> things).
>

The intention of 1.a was definitely to leave in the ClientConfiguration
(deprecated) as-is until it is dropped entirely in 2.0. So, there'd be full
source and binary compatibility while it exists, but it would go through a
deprecation cycle and be dropped in 2.0, so users would have to switch to
the replacement ClientConfig before 2.0.

Re: [DISCUSS] Hadoop3 support target?

[Josh Elser]

On 12/6/17 12:17 PM, Keith Turner wrote:

On Wed, Dec 6, 2017 at 11:56 AM, Josh Elser  wrote:

Maybe a difference in interpretation:

I was seeing 1a as being source-compatible still. My assumption was that
"Deprecate ClientConfiguration" meant that it would remain in the codebase
-- "replace" as in "replace expected user invocation", not removal of the
old ClientConfiguration and addition of a new ClientConfig class.

Ok, if we deprecate ClientConfiguration, leave it in 2.0, and drop the
extends from ClientConfiguration in 2.0.  Then I am not sure what the
benefit of introducing the new ClientConfig type is?


I read this as leaving the extends in ClientConfiguration and dropping 
that in the new ClientConfig. Agree, I wouldn't see the point in 
changing the parent class of ClientConfiguration (as that would break 
things).

Re: [DISCUSS] Hadoop3 support target?

[Keith Turner]

On Wed, Dec 6, 2017 at 11:56 AM, Josh Elser  wrote:
> Maybe a difference in interpretation:
>
> I was seeing 1a as being source-compatible still. My assumption was that
> "Deprecate ClientConfiguration" meant that it would remain in the codebase
> -- "replace" as in "replace expected user invocation", not removal of the
> old ClientConfiguration and addition of a new ClientConfig class.

Ok, if we deprecate ClientConfiguration, leave it in 2.0, and drop the
extends from ClientConfiguration in 2.0.  Then I am not sure what the
benefit of introducing the new ClientConfig type is?


>
>
> On 12/6/17 11:29 AM, Keith Turner wrote:
>>
>> On Wed, Dec 6, 2017 at 11:28 AM, Josh Elser  wrote:
>>>
>>> 1.a sounds better to me.
>>
>>
>> why?
>>
>>>
>>> A would be the ideal solution, I think B is the next best if A doesn't
>>> work.
>>>
>>> I need to get the Hadoop3 compatibility fixed, so I'll be investigating
>>> the
>>> Hadoop shaded artifacts this week.
>>>
>>>
>>> On 12/5/17 6:43 PM, Christopher wrote:


 I was wondering about Hadoop 3 shading and whether that would help us.
 It
 would be really nice if it could, or if there was some other class path
 solution that was easy.

 I think there are two major issues in this thread. The first is the API
 problems. The second is the Hadoop 3 support. They are related, but I
 think
 quickly dealing with the API issues can clarify what our options are for
 Hadoop 3.




 To fix the API, I would like to get consensus on proceeding with this
 path:

 1. Rename 1.8.2-SNAPSHOT to 1.9.0-SNAPSHOT and deprecate the existing
 ZooKeeperInstance constructor which takes a Configuration
   a) Deprecate ClientConfiguration and replace with ClientConfig (or
 a
 better name) which does not extend Configuration or have API leak
 problems,
 and add a new ZKI constructor for this
   b) Ignore extends for now, and drop it from ClientConfiguration in
 2.0
 with a break (can't deprecate superclass), and add new ZKI constructor
 for
 more specific ClientConfiguration next to deprecated one
 2. Drop deprecated stuff from 2.0 branch (and extends, if option 1.b was
 chosen)
 3. Plan a 1.9.0 release instead of 1.8.2

 I prefer 1.a over 1.b, personally, but I've been tossing back and forth.
 I
 would need input on which is best. There are pros and cons to both,
 regarding churn, and source and binary compatibility.




 Once we deal with the API, our options for Hadoop 3 become:

 A. Use Hadoop 3 shaded artifacts or some other class path solution (such
 as
 getting lucky identifying a version of commons-beanutils that works for
 both)
 B. Shade in 1.9 with a breaking change
 C. Create a 1.9 version named 2.0, so we can do a breaking change
 without
 semver violation; shade in this version
 D. Shade in the branch we're currently calling 2.0

 I think we can defer that decision pending some further
 investigation/experimentation into what works, and deal with it after
 dealing with steps 1-3 above (but soon after, hopefully).



 On Tue, Dec 5, 2017 at 3:58 PM Josh Elser  wrote:

> Another potential suggestion I forgot about: we try to just move to the
> Hadoop shaded artifacts. This would invalidate the need to do more, but
> I have no idea how "battle-tested" those artifacts are.
>
> On 12/5/17 3:52 PM, Keith Turner wrote:
>>
>>
>> If we do the following.
>>
>> * Drop ZooKeeperInstance.ZooKeeperInstance(Configuration config)
>
>
> method.
>>
>>
>> * Drop extends from ClientConfig
>> * Add a method ZooKeeperInstance.ZooKeeperInstance(ClientConfig
>> config)
>>
>> Then this will not be binary compatible, so it will still be painful
>> in many cases.   It may be source compatible.
>>
>> For example the following will be source (but not binary) compatible.
>>
>>  ClientConfiguration cc = new ClientConfiguration(file);
>>  //when compiled against older version of Accumulo will bind to
>> method with commons config signature
>>  //when recompiled will bind to clientconfig version of method
>>  ZooKeeperInstance zki = new ZooKeeperInstance(cc);
>>
>> The following would not be source or binary compatible.
>>
>>  Configuration cc = new ClientConfiguration(file);
>>  ZooKeeperInstance zki = new ZooKeeperInstance(cc);
>>
>>
>> On Tue, Dec 5, 2017 at 3:40 PM, Josh Elser  wrote:
>>>
>>>
>>>
>>>
>>> On 12/5/17 3:28 PM, Keith Turner wrote:



 On Tue, Dec 5, 2017 at 2:53 PM, Josh Elser
 wrote:
>
>
>

Re: [DISCUSS] Hadoop3 support target?

[Josh Elser]

Maybe a difference in interpretation:

I was seeing 1a as being source-compatible still. My assumption was that 
"Deprecate ClientConfiguration" meant that it would remain in the 
codebase -- "replace" as in "replace expected user invocation", not 
removal of the old ClientConfiguration and addition of a new 
ClientConfig class.


On 12/6/17 11:29 AM, Keith Turner wrote:

On Wed, Dec 6, 2017 at 11:28 AM, Josh Elser  wrote:

1.a sounds better to me.


why?



A would be the ideal solution, I think B is the next best if A doesn't work.

I need to get the Hadoop3 compatibility fixed, so I'll be investigating the
Hadoop shaded artifacts this week.


On 12/5/17 6:43 PM, Christopher wrote:


I was wondering about Hadoop 3 shading and whether that would help us. It
would be really nice if it could, or if there was some other class path
solution that was easy.

I think there are two major issues in this thread. The first is the API
problems. The second is the Hadoop 3 support. They are related, but I
think
quickly dealing with the API issues can clarify what our options are for
Hadoop 3.




To fix the API, I would like to get consensus on proceeding with this
path:

1. Rename 1.8.2-SNAPSHOT to 1.9.0-SNAPSHOT and deprecate the existing
ZooKeeperInstance constructor which takes a Configuration
  a) Deprecate ClientConfiguration and replace with ClientConfig (or a
better name) which does not extend Configuration or have API leak
problems,
and add a new ZKI constructor for this
  b) Ignore extends for now, and drop it from ClientConfiguration in
2.0
with a break (can't deprecate superclass), and add new ZKI constructor for
more specific ClientConfiguration next to deprecated one
2. Drop deprecated stuff from 2.0 branch (and extends, if option 1.b was
chosen)
3. Plan a 1.9.0 release instead of 1.8.2

I prefer 1.a over 1.b, personally, but I've been tossing back and forth. I
would need input on which is best. There are pros and cons to both,
regarding churn, and source and binary compatibility.




Once we deal with the API, our options for Hadoop 3 become:

A. Use Hadoop 3 shaded artifacts or some other class path solution (such
as
getting lucky identifying a version of commons-beanutils that works for
both)
B. Shade in 1.9 with a breaking change
C. Create a 1.9 version named 2.0, so we can do a breaking change without
semver violation; shade in this version
D. Shade in the branch we're currently calling 2.0

I think we can defer that decision pending some further
investigation/experimentation into what works, and deal with it after
dealing with steps 1-3 above (but soon after, hopefully).



On Tue, Dec 5, 2017 at 3:58 PM Josh Elser  wrote:


Another potential suggestion I forgot about: we try to just move to the
Hadoop shaded artifacts. This would invalidate the need to do more, but
I have no idea how "battle-tested" those artifacts are.

On 12/5/17 3:52 PM, Keith Turner wrote:


If we do the following.

* Drop ZooKeeperInstance.ZooKeeperInstance(Configuration config)


method.


* Drop extends from ClientConfig
* Add a method ZooKeeperInstance.ZooKeeperInstance(ClientConfig
config)

Then this will not be binary compatible, so it will still be painful
in many cases.   It may be source compatible.

For example the following will be source (but not binary) compatible.

 ClientConfiguration cc = new ClientConfiguration(file);
 //when compiled against older version of Accumulo will bind to
method with commons config signature
 //when recompiled will bind to clientconfig version of method
 ZooKeeperInstance zki = new ZooKeeperInstance(cc);

The following would not be source or binary compatible.

 Configuration cc = new ClientConfiguration(file);
 ZooKeeperInstance zki = new ZooKeeperInstance(cc);


On Tue, Dec 5, 2017 at 3:40 PM, Josh Elser  wrote:




On 12/5/17 3:28 PM, Keith Turner wrote:



On Tue, Dec 5, 2017 at 2:53 PM, Josh Elser  wrote:



Interesting. What makes you want to deprecate ClientConfig entirely?

I'd be worried about removing without sufficient thought of


replacement


around. It would be a bit "churn-y" to introduce yet another way that
clients have to connect (since it was introduced in 1.6-ish?).
Working
around the ClientConfig changes was irritating for the downstream
integrations (Hive, most notably).



Ok maybe thats a bad idea, not looking to cause pain.  Here were some
of my goals.

 * Remove commons config from API completely via deprecation cycle.
 * Introduce API that supports putting all props needed to connect
to
Accumulo in an API.

I suppose if we want to keep ClientConfig class in API, then there is
no way to remove commons config via a deprecation cycle??  We can't
deprecate the extension of commons config, all we can do is just drop
it at some point.



My line of thinking is that the majority of the time, we're creating a
ClientConfiguration by one 

Re: [DISCUSS] Hadoop3 support target?

[Keith Turner]

On Wed, Dec 6, 2017 at 11:28 AM, Josh Elser  wrote:
> 1.a sounds better to me.

why?

>
> A would be the ideal solution, I think B is the next best if A doesn't work.
>
> I need to get the Hadoop3 compatibility fixed, so I'll be investigating the
> Hadoop shaded artifacts this week.
>
>
> On 12/5/17 6:43 PM, Christopher wrote:
>>
>> I was wondering about Hadoop 3 shading and whether that would help us. It
>> would be really nice if it could, or if there was some other class path
>> solution that was easy.
>>
>> I think there are two major issues in this thread. The first is the API
>> problems. The second is the Hadoop 3 support. They are related, but I
>> think
>> quickly dealing with the API issues can clarify what our options are for
>> Hadoop 3.
>>
>>
>>
>>
>> To fix the API, I would like to get consensus on proceeding with this
>> path:
>>
>> 1. Rename 1.8.2-SNAPSHOT to 1.9.0-SNAPSHOT and deprecate the existing
>> ZooKeeperInstance constructor which takes a Configuration
>>  a) Deprecate ClientConfiguration and replace with ClientConfig (or a
>> better name) which does not extend Configuration or have API leak
>> problems,
>> and add a new ZKI constructor for this
>>  b) Ignore extends for now, and drop it from ClientConfiguration in
>> 2.0
>> with a break (can't deprecate superclass), and add new ZKI constructor for
>> more specific ClientConfiguration next to deprecated one
>> 2. Drop deprecated stuff from 2.0 branch (and extends, if option 1.b was
>> chosen)
>> 3. Plan a 1.9.0 release instead of 1.8.2
>>
>> I prefer 1.a over 1.b, personally, but I've been tossing back and forth. I
>> would need input on which is best. There are pros and cons to both,
>> regarding churn, and source and binary compatibility.
>>
>>
>>
>>
>> Once we deal with the API, our options for Hadoop 3 become:
>>
>> A. Use Hadoop 3 shaded artifacts or some other class path solution (such
>> as
>> getting lucky identifying a version of commons-beanutils that works for
>> both)
>> B. Shade in 1.9 with a breaking change
>> C. Create a 1.9 version named 2.0, so we can do a breaking change without
>> semver violation; shade in this version
>> D. Shade in the branch we're currently calling 2.0
>>
>> I think we can defer that decision pending some further
>> investigation/experimentation into what works, and deal with it after
>> dealing with steps 1-3 above (but soon after, hopefully).
>>
>>
>>
>> On Tue, Dec 5, 2017 at 3:58 PM Josh Elser  wrote:
>>
>>> Another potential suggestion I forgot about: we try to just move to the
>>> Hadoop shaded artifacts. This would invalidate the need to do more, but
>>> I have no idea how "battle-tested" those artifacts are.
>>>
>>> On 12/5/17 3:52 PM, Keith Turner wrote:

 If we do the following.

* Drop ZooKeeperInstance.ZooKeeperInstance(Configuration config)
>>>
>>> method.

* Drop extends from ClientConfig
* Add a method ZooKeeperInstance.ZooKeeperInstance(ClientConfig
 config)

 Then this will not be binary compatible, so it will still be painful
 in many cases.   It may be source compatible.

 For example the following will be source (but not binary) compatible.

 ClientConfiguration cc = new ClientConfiguration(file);
 //when compiled against older version of Accumulo will bind to
 method with commons config signature
 //when recompiled will bind to clientconfig version of method
 ZooKeeperInstance zki = new ZooKeeperInstance(cc);

 The following would not be source or binary compatible.

 Configuration cc = new ClientConfiguration(file);
 ZooKeeperInstance zki = new ZooKeeperInstance(cc);


 On Tue, Dec 5, 2017 at 3:40 PM, Josh Elser  wrote:
>
>
>
> On 12/5/17 3:28 PM, Keith Turner wrote:
>>
>>
>> On Tue, Dec 5, 2017 at 2:53 PM, Josh Elser  wrote:
>>>
>>>
>>> Interesting. What makes you want to deprecate ClientConfig entirely?
>>>
>>> I'd be worried about removing without sufficient thought of
>>>
>>> replacement
>>>
>>> around. It would be a bit "churn-y" to introduce yet another way that
>>> clients have to connect (since it was introduced in 1.6-ish?).
>>> Working
>>> around the ClientConfig changes was irritating for the downstream
>>> integrations (Hive, most notably).
>>
>>
>> Ok maybe thats a bad idea, not looking to cause pain.  Here were some
>> of my goals.
>>
>> * Remove commons config from API completely via deprecation cycle.
>> * Introduce API that supports putting all props needed to connect
>> to
>> Accumulo in an API.
>>
>> I suppose if we want to keep ClientConfig class in API, then there is
>> no way to remove commons config via a deprecation cycle??  We can't
>> deprecate the extension of commons config, all we can do 

Re: [DISCUSS] Hadoop3 support target?

[Keith Turner]

On Tue, Dec 5, 2017 at 6:43 PM, Christopher  wrote:
> I was wondering about Hadoop 3 shading and whether that would help us. It
> would be really nice if it could, or if there was some other class path
> solution that was easy.
>
> I think there are two major issues in this thread. The first is the API
> problems. The second is the Hadoop 3 support. They are related, but I think
> quickly dealing with the API issues can clarify what our options are for
> Hadoop 3.
>
>
>
>
> To fix the API, I would like to get consensus on proceeding with this path:
>
> 1. Rename 1.8.2-SNAPSHOT to 1.9.0-SNAPSHOT and deprecate the existing
> ZooKeeperInstance constructor which takes a Configuration
> a) Deprecate ClientConfiguration and replace with ClientConfig (or a
> better name) which does not extend Configuration or have API leak problems,
> and add a new ZKI constructor for this
> b) Ignore extends for now, and drop it from ClientConfiguration in 2.0
> with a break (can't deprecate superclass), and add new ZKI constructor for
> more specific ClientConfiguration next to deprecated one
> 2. Drop deprecated stuff from 2.0 branch (and extends, if option 1.b was
> chosen)
> 3. Plan a 1.9.0 release instead of 1.8.2
>
> I prefer 1.a over 1.b, personally, but I've been tossing back and forth. I
> would need input on which is best. There are pros and cons to both,
> regarding churn, and source and binary compatibility.

For 1.a there is no way to maintain source or binary compat going from
1.x where x<9 to 2.y.

For 1.b it seems that you can get get source compat (but not binary)
in some cases when going from 1.x where x<9 to 2.y.

Based on this it seems that 1.b may be better because its source
compat sometimes, where 1.a is never source compat.

>
>
>
>
> Once we deal with the API, our options for Hadoop 3 become:
>
> A. Use Hadoop 3 shaded artifacts or some other class path solution (such as
> getting lucky identifying a version of commons-beanutils that works for
> both)
> B. Shade in 1.9 with a breaking change
> C. Create a 1.9 version named 2.0, so we can do a breaking change without
> semver violation; shade in this version
> D. Shade in the branch we're currently calling 2.0
>
> I think we can defer that decision pending some further
> investigation/experimentation into what works, and deal with it after
> dealing with steps 1-3 above (but soon after, hopefully).
>
>
>
> On Tue, Dec 5, 2017 at 3:58 PM Josh Elser  wrote:
>
>> Another potential suggestion I forgot about: we try to just move to the
>> Hadoop shaded artifacts. This would invalidate the need to do more, but
>> I have no idea how "battle-tested" those artifacts are.
>>
>> On 12/5/17 3:52 PM, Keith Turner wrote:
>> > If we do the following.
>> >
>> >   * Drop ZooKeeperInstance.ZooKeeperInstance(Configuration config)
>> method.
>> >   * Drop extends from ClientConfig
>> >   * Add a method ZooKeeperInstance.ZooKeeperInstance(ClientConfig config)
>> >
>> > Then this will not be binary compatible, so it will still be painful
>> > in many cases.   It may be source compatible.
>> >
>> > For example the following will be source (but not binary) compatible.
>> >
>> >ClientConfiguration cc = new ClientConfiguration(file);
>> >//when compiled against older version of Accumulo will bind to
>> > method with commons config signature
>> >//when recompiled will bind to clientconfig version of method
>> >ZooKeeperInstance zki = new ZooKeeperInstance(cc);
>> >
>> > The following would not be source or binary compatible.
>> >
>> >Configuration cc = new ClientConfiguration(file);
>> >ZooKeeperInstance zki = new ZooKeeperInstance(cc);
>> >
>> >
>> > On Tue, Dec 5, 2017 at 3:40 PM, Josh Elser  wrote:
>> >>
>> >>
>> >> On 12/5/17 3:28 PM, Keith Turner wrote:
>> >>>
>> >>> On Tue, Dec 5, 2017 at 2:53 PM, Josh Elser  wrote:
>> 
>>  Interesting. What makes you want to deprecate ClientConfig entirely?
>> 
>>  I'd be worried about removing without sufficient thought of
>> replacement
>>  around. It would be a bit "churn-y" to introduce yet another way that
>>  clients have to connect (since it was introduced in 1.6-ish?). Working
>>  around the ClientConfig changes was irritating for the downstream
>>  integrations (Hive, most notably).
>> >>>
>> >>> Ok maybe thats a bad idea, not looking to cause pain.  Here were some
>> >>> of my goals.
>> >>>
>> >>>* Remove commons config from API completely via deprecation cycle.
>> >>>* Introduce API that supports putting all props needed to connect to
>> >>> Accumulo in an API.
>> >>>
>> >>> I suppose if we want to keep ClientConfig class in API, then there is
>> >>> no way to remove commons config via a deprecation cycle??  We can't
>> >>> deprecate the extension of commons config, all we can do is just drop
>> >>> it at some point.
>> >>>
>> >>
>> >> My line of thinking is that the majority of the 

Re: [DISCUSS] Hadoop3 support target?

[Josh Elser]

1.a sounds better to me.

A would be the ideal solution, I think B is the next best if A doesn't work.

I need to get the Hadoop3 compatibility fixed, so I'll be investigating 
the Hadoop shaded artifacts this week.


On 12/5/17 6:43 PM, Christopher wrote:

I was wondering about Hadoop 3 shading and whether that would help us. It
would be really nice if it could, or if there was some other class path
solution that was easy.

I think there are two major issues in this thread. The first is the API
problems. The second is the Hadoop 3 support. They are related, but I think
quickly dealing with the API issues can clarify what our options are for
Hadoop 3.




To fix the API, I would like to get consensus on proceeding with this path:

1. Rename 1.8.2-SNAPSHOT to 1.9.0-SNAPSHOT and deprecate the existing
ZooKeeperInstance constructor which takes a Configuration
 a) Deprecate ClientConfiguration and replace with ClientConfig (or a
better name) which does not extend Configuration or have API leak problems,
and add a new ZKI constructor for this
 b) Ignore extends for now, and drop it from ClientConfiguration in 2.0
with a break (can't deprecate superclass), and add new ZKI constructor for
more specific ClientConfiguration next to deprecated one
2. Drop deprecated stuff from 2.0 branch (and extends, if option 1.b was
chosen)
3. Plan a 1.9.0 release instead of 1.8.2

I prefer 1.a over 1.b, personally, but I've been tossing back and forth. I
would need input on which is best. There are pros and cons to both,
regarding churn, and source and binary compatibility.




Once we deal with the API, our options for Hadoop 3 become:

A. Use Hadoop 3 shaded artifacts or some other class path solution (such as
getting lucky identifying a version of commons-beanutils that works for
both)
B. Shade in 1.9 with a breaking change
C. Create a 1.9 version named 2.0, so we can do a breaking change without
semver violation; shade in this version
D. Shade in the branch we're currently calling 2.0

I think we can defer that decision pending some further
investigation/experimentation into what works, and deal with it after
dealing with steps 1-3 above (but soon after, hopefully).



On Tue, Dec 5, 2017 at 3:58 PM Josh Elser  wrote:


Another potential suggestion I forgot about: we try to just move to the
Hadoop shaded artifacts. This would invalidate the need to do more, but
I have no idea how "battle-tested" those artifacts are.

On 12/5/17 3:52 PM, Keith Turner wrote:

If we do the following.

   * Drop ZooKeeperInstance.ZooKeeperInstance(Configuration config)

method.

   * Drop extends from ClientConfig
   * Add a method ZooKeeperInstance.ZooKeeperInstance(ClientConfig config)

Then this will not be binary compatible, so it will still be painful
in many cases.   It may be source compatible.

For example the following will be source (but not binary) compatible.

ClientConfiguration cc = new ClientConfiguration(file);
//when compiled against older version of Accumulo will bind to
method with commons config signature
//when recompiled will bind to clientconfig version of method
ZooKeeperInstance zki = new ZooKeeperInstance(cc);

The following would not be source or binary compatible.

Configuration cc = new ClientConfiguration(file);
ZooKeeperInstance zki = new ZooKeeperInstance(cc);


On Tue, Dec 5, 2017 at 3:40 PM, Josh Elser  wrote:



On 12/5/17 3:28 PM, Keith Turner wrote:


On Tue, Dec 5, 2017 at 2:53 PM, Josh Elser  wrote:


Interesting. What makes you want to deprecate ClientConfig entirely?

I'd be worried about removing without sufficient thought of

replacement

around. It would be a bit "churn-y" to introduce yet another way that
clients have to connect (since it was introduced in 1.6-ish?). Working
around the ClientConfig changes was irritating for the downstream
integrations (Hive, most notably).


Ok maybe thats a bad idea, not looking to cause pain.  Here were some
of my goals.

* Remove commons config from API completely via deprecation cycle.
* Introduce API that supports putting all props needed to connect to
Accumulo in an API.

I suppose if we want to keep ClientConfig class in API, then there is
no way to remove commons config via a deprecation cycle??  We can't
deprecate the extension of commons config, all we can do is just drop
it at some point.



My line of thinking is that the majority of the time, we're creating a
ClientConfiguration by one of:

* ClientConfiguration#loadDefault()
* new ClientConfiguration(String)
* new ClientConfiguration(File)

Granted, we also inherit/expose a few other things (notably extending
CompositeConfiguration and throwing ConfigurationException). I would be
comfortable with dropping those w/o deprecation. I have not seen

evidence

from anyone that they are widely in use by folks (although I've not
explicitly asked, either).

Re: [DISCUSS] Hadoop3 support target?

[Christopher]

I was wondering about Hadoop 3 shading and whether that would help us. It
would be really nice if it could, or if there was some other class path
solution that was easy.

I think there are two major issues in this thread. The first is the API
problems. The second is the Hadoop 3 support. They are related, but I think
quickly dealing with the API issues can clarify what our options are for
Hadoop 3.




To fix the API, I would like to get consensus on proceeding with this path:

1. Rename 1.8.2-SNAPSHOT to 1.9.0-SNAPSHOT and deprecate the existing
ZooKeeperInstance constructor which takes a Configuration
a) Deprecate ClientConfiguration and replace with ClientConfig (or a
better name) which does not extend Configuration or have API leak problems,
and add a new ZKI constructor for this
b) Ignore extends for now, and drop it from ClientConfiguration in 2.0
with a break (can't deprecate superclass), and add new ZKI constructor for
more specific ClientConfiguration next to deprecated one
2. Drop deprecated stuff from 2.0 branch (and extends, if option 1.b was
chosen)
3. Plan a 1.9.0 release instead of 1.8.2

I prefer 1.a over 1.b, personally, but I've been tossing back and forth. I
would need input on which is best. There are pros and cons to both,
regarding churn, and source and binary compatibility.




Once we deal with the API, our options for Hadoop 3 become:

A. Use Hadoop 3 shaded artifacts or some other class path solution (such as
getting lucky identifying a version of commons-beanutils that works for
both)
B. Shade in 1.9 with a breaking change
C. Create a 1.9 version named 2.0, so we can do a breaking change without
semver violation; shade in this version
D. Shade in the branch we're currently calling 2.0

I think we can defer that decision pending some further
investigation/experimentation into what works, and deal with it after
dealing with steps 1-3 above (but soon after, hopefully).



On Tue, Dec 5, 2017 at 3:58 PM Josh Elser  wrote:

> Another potential suggestion I forgot about: we try to just move to the
> Hadoop shaded artifacts. This would invalidate the need to do more, but
> I have no idea how "battle-tested" those artifacts are.
>
> On 12/5/17 3:52 PM, Keith Turner wrote:
> > If we do the following.
> >
> >   * Drop ZooKeeperInstance.ZooKeeperInstance(Configuration config)
> method.
> >   * Drop extends from ClientConfig
> >   * Add a method ZooKeeperInstance.ZooKeeperInstance(ClientConfig config)
> >
> > Then this will not be binary compatible, so it will still be painful
> > in many cases.   It may be source compatible.
> >
> > For example the following will be source (but not binary) compatible.
> >
> >ClientConfiguration cc = new ClientConfiguration(file);
> >//when compiled against older version of Accumulo will bind to
> > method with commons config signature
> >//when recompiled will bind to clientconfig version of method
> >ZooKeeperInstance zki = new ZooKeeperInstance(cc);
> >
> > The following would not be source or binary compatible.
> >
> >Configuration cc = new ClientConfiguration(file);
> >ZooKeeperInstance zki = new ZooKeeperInstance(cc);
> >
> >
> > On Tue, Dec 5, 2017 at 3:40 PM, Josh Elser  wrote:
> >>
> >>
> >> On 12/5/17 3:28 PM, Keith Turner wrote:
> >>>
> >>> On Tue, Dec 5, 2017 at 2:53 PM, Josh Elser  wrote:
> 
>  Interesting. What makes you want to deprecate ClientConfig entirely?
> 
>  I'd be worried about removing without sufficient thought of
> replacement
>  around. It would be a bit "churn-y" to introduce yet another way that
>  clients have to connect (since it was introduced in 1.6-ish?). Working
>  around the ClientConfig changes was irritating for the downstream
>  integrations (Hive, most notably).
> >>>
> >>> Ok maybe thats a bad idea, not looking to cause pain.  Here were some
> >>> of my goals.
> >>>
> >>>* Remove commons config from API completely via deprecation cycle.
> >>>* Introduce API that supports putting all props needed to connect to
> >>> Accumulo in an API.
> >>>
> >>> I suppose if we want to keep ClientConfig class in API, then there is
> >>> no way to remove commons config via a deprecation cycle??  We can't
> >>> deprecate the extension of commons config, all we can do is just drop
> >>> it at some point.
> >>>
> >>
> >> My line of thinking is that the majority of the time, we're creating a
> >> ClientConfiguration by one of:
> >>
> >> * ClientConfiguration#loadDefault()
> >> * new ClientConfiguration(String)
> >> * new ClientConfiguration(File)
> >>
> >> Granted, we also inherit/expose a few other things (notably extending
> >> CompositeConfiguration and throwing ConfigurationException). I would be
> >> comfortable with dropping those w/o deprecation. I have not seen
> evidence
> >> from anyone that they are widely in use by folks (although I've not
> >> explicitly asked, either).
>

Re: [DISCUSS] Hadoop3 support target?

[Keith Turner]

Another option for the sake of discussion, is to have a hadoop 3
version.  For example


 
org.apache.accumulo
accumulo-core
1.8.2-hadoop3
  

I think this horrible in too many ways to enumerate, not advocating
for it.  Just trying to exhaustively think through the solution space.


On Tue, Dec 5, 2017 at 3:58 PM, Josh Elser  wrote:
> Another potential suggestion I forgot about: we try to just move to the
> Hadoop shaded artifacts. This would invalidate the need to do more, but I
> have no idea how "battle-tested" those artifacts are.
>
>
> On 12/5/17 3:52 PM, Keith Turner wrote:
>>
>> If we do the following.
>>
>>   * Drop ZooKeeperInstance.ZooKeeperInstance(Configuration config) method.
>>   * Drop extends from ClientConfig
>>   * Add a method ZooKeeperInstance.ZooKeeperInstance(ClientConfig config)
>>
>> Then this will not be binary compatible, so it will still be painful
>> in many cases.   It may be source compatible.
>>
>> For example the following will be source (but not binary) compatible.
>>
>>ClientConfiguration cc = new ClientConfiguration(file);
>>//when compiled against older version of Accumulo will bind to
>> method with commons config signature
>>//when recompiled will bind to clientconfig version of method
>>ZooKeeperInstance zki = new ZooKeeperInstance(cc);
>>
>> The following would not be source or binary compatible.
>>
>>Configuration cc = new ClientConfiguration(file);
>>ZooKeeperInstance zki = new ZooKeeperInstance(cc);
>>
>>
>> On Tue, Dec 5, 2017 at 3:40 PM, Josh Elser  wrote:
>>>
>>>
>>>
>>> On 12/5/17 3:28 PM, Keith Turner wrote:


 On Tue, Dec 5, 2017 at 2:53 PM, Josh Elser  wrote:
>
>
> Interesting. What makes you want to deprecate ClientConfig entirely?
>
> I'd be worried about removing without sufficient thought of replacement
> around. It would be a bit "churn-y" to introduce yet another way that
> clients have to connect (since it was introduced in 1.6-ish?). Working
> around the ClientConfig changes was irritating for the downstream
> integrations (Hive, most notably).


 Ok maybe thats a bad idea, not looking to cause pain.  Here were some
 of my goals.

* Remove commons config from API completely via deprecation cycle.
* Introduce API that supports putting all props needed to connect to
 Accumulo in an API.

 I suppose if we want to keep ClientConfig class in API, then there is
 no way to remove commons config via a deprecation cycle??  We can't
 deprecate the extension of commons config, all we can do is just drop
 it at some point.

>>>
>>> My line of thinking is that the majority of the time, we're creating a
>>> ClientConfiguration by one of:
>>>
>>> * ClientConfiguration#loadDefault()
>>> * new ClientConfiguration(String)
>>> * new ClientConfiguration(File)
>>>
>>> Granted, we also inherit/expose a few other things (notably extending
>>> CompositeConfiguration and throwing ConfigurationException). I would be
>>> comfortable with dropping those w/o deprecation. I have not seen evidence
>>> from anyone that they are widely in use by folks (although I've not
>>> explicitly asked, either).

Re: [DISCUSS] Hadoop3 support target?

[Josh Elser]

Another potential suggestion I forgot about: we try to just move to the 
Hadoop shaded artifacts. This would invalidate the need to do more, but 
I have no idea how "battle-tested" those artifacts are.


On 12/5/17 3:52 PM, Keith Turner wrote:

If we do the following.

  * Drop ZooKeeperInstance.ZooKeeperInstance(Configuration config) method.
  * Drop extends from ClientConfig
  * Add a method ZooKeeperInstance.ZooKeeperInstance(ClientConfig config)

Then this will not be binary compatible, so it will still be painful
in many cases.   It may be source compatible.

For example the following will be source (but not binary) compatible.

   ClientConfiguration cc = new ClientConfiguration(file);
   //when compiled against older version of Accumulo will bind to
method with commons config signature
   //when recompiled will bind to clientconfig version of method
   ZooKeeperInstance zki = new ZooKeeperInstance(cc);

The following would not be source or binary compatible.

   Configuration cc = new ClientConfiguration(file);
   ZooKeeperInstance zki = new ZooKeeperInstance(cc);


On Tue, Dec 5, 2017 at 3:40 PM, Josh Elser  wrote:



On 12/5/17 3:28 PM, Keith Turner wrote:


On Tue, Dec 5, 2017 at 2:53 PM, Josh Elser  wrote:


Interesting. What makes you want to deprecate ClientConfig entirely?

I'd be worried about removing without sufficient thought of replacement
around. It would be a bit "churn-y" to introduce yet another way that
clients have to connect (since it was introduced in 1.6-ish?). Working
around the ClientConfig changes was irritating for the downstream
integrations (Hive, most notably).


Ok maybe thats a bad idea, not looking to cause pain.  Here were some
of my goals.

   * Remove commons config from API completely via deprecation cycle.
   * Introduce API that supports putting all props needed to connect to
Accumulo in an API.

I suppose if we want to keep ClientConfig class in API, then there is
no way to remove commons config via a deprecation cycle??  We can't
deprecate the extension of commons config, all we can do is just drop
it at some point.



My line of thinking is that the majority of the time, we're creating a
ClientConfiguration by one of:

* ClientConfiguration#loadDefault()
* new ClientConfiguration(String)
* new ClientConfiguration(File)

Granted, we also inherit/expose a few other things (notably extending
CompositeConfiguration and throwing ConfigurationException). I would be
comfortable with dropping those w/o deprecation. I have not seen evidence
from anyone that they are widely in use by folks (although I've not
explicitly asked, either).

Re: [DISCUSS] Hadoop3 support target?

[Christopher]

On Tue, Dec 5, 2017 at 3:40 PM Josh Elser  wrote:

>
>
> On 12/5/17 3:28 PM, Keith Turner wrote:
> > On Tue, Dec 5, 2017 at 2:53 PM, Josh Elser  wrote:
> >> Interesting. What makes you want to deprecate ClientConfig entirely?
> >>
> >> I'd be worried about removing without sufficient thought of replacement
> >> around. It would be a bit "churn-y" to introduce yet another way that
> >> clients have to connect (since it was introduced in 1.6-ish?). Working
> >> around the ClientConfig changes was irritating for the downstream
> >> integrations (Hive, most notably).
> > Ok maybe thats a bad idea, not looking to cause pain.  Here were some
> > of my goals.
> >
> >   * Remove commons config from API completely via deprecation cycle.
> >   * Introduce API that supports putting all props needed to connect to
> > Accumulo in an API.
> >
> > I suppose if we want to keep ClientConfig class in API, then there is
> > no way to remove commons config via a deprecation cycle??  We can't
> > deprecate the extension of commons config, all we can do is just drop
> > it at some point.
> >
>
> My line of thinking is that the majority of the time, we're creating a
> ClientConfiguration by one of:
>
> * ClientConfiguration#loadDefault()
> * new ClientConfiguration(String)
> * new ClientConfiguration(File)
>
> Granted, we also inherit/expose a few other things (notably extending
> CompositeConfiguration and throwing ConfigurationException). I would be
> comfortable with dropping those w/o deprecation. I have not seen
> evidence from anyone that they are widely in use by folks (although I've
> not explicitly asked, either).
>

I would also be comfortable with that. I think the main API that we need to
deal with is the ZooKeeperInstance(Configuration) constructor. It should be
ZooKeeperInstance(ClientConfiguration) instead. If we assume that the only
way this method was used was with ClientConfiguration objects, then simply
changing the parameter type will continue to be source compatible. I don't
know if we want to assume that. Either way, it won't be binary compatible.

If we're not concerned with binary compatibility, and are willing to make
that assumption about this constructor only (or usually) being used with
ClientConfiguration, then we can just change the method signature now,
without deprecation cycles or a new alternative method.

Re: [DISCUSS] Hadoop3 support target?

[Keith Turner]

If we do the following.

 * Drop ZooKeeperInstance.ZooKeeperInstance(Configuration config) method.
 * Drop extends from ClientConfig
 * Add a method ZooKeeperInstance.ZooKeeperInstance(ClientConfig config)

Then this will not be binary compatible, so it will still be painful
in many cases.   It may be source compatible.

For example the following will be source (but not binary) compatible.

  ClientConfiguration cc = new ClientConfiguration(file);
  //when compiled against older version of Accumulo will bind to
method with commons config signature
  //when recompiled will bind to clientconfig version of method
  ZooKeeperInstance zki = new ZooKeeperInstance(cc);

The following would not be source or binary compatible.

  Configuration cc = new ClientConfiguration(file);
  ZooKeeperInstance zki = new ZooKeeperInstance(cc);


On Tue, Dec 5, 2017 at 3:40 PM, Josh Elser  wrote:
>
>
> On 12/5/17 3:28 PM, Keith Turner wrote:
>>
>> On Tue, Dec 5, 2017 at 2:53 PM, Josh Elser  wrote:
>>>
>>> Interesting. What makes you want to deprecate ClientConfig entirely?
>>>
>>> I'd be worried about removing without sufficient thought of replacement
>>> around. It would be a bit "churn-y" to introduce yet another way that
>>> clients have to connect (since it was introduced in 1.6-ish?). Working
>>> around the ClientConfig changes was irritating for the downstream
>>> integrations (Hive, most notably).
>>
>> Ok maybe thats a bad idea, not looking to cause pain.  Here were some
>> of my goals.
>>
>>   * Remove commons config from API completely via deprecation cycle.
>>   * Introduce API that supports putting all props needed to connect to
>> Accumulo in an API.
>>
>> I suppose if we want to keep ClientConfig class in API, then there is
>> no way to remove commons config via a deprecation cycle??  We can't
>> deprecate the extension of commons config, all we can do is just drop
>> it at some point.
>>
>
> My line of thinking is that the majority of the time, we're creating a
> ClientConfiguration by one of:
>
> * ClientConfiguration#loadDefault()
> * new ClientConfiguration(String)
> * new ClientConfiguration(File)
>
> Granted, we also inherit/expose a few other things (notably extending
> CompositeConfiguration and throwing ConfigurationException). I would be
> comfortable with dropping those w/o deprecation. I have not seen evidence
> from anyone that they are widely in use by folks (although I've not
> explicitly asked, either).

Re: [DISCUSS] Hadoop3 support target?

[Josh Elser]

On 12/5/17 3:28 PM, Keith Turner wrote:

On Tue, Dec 5, 2017 at 2:53 PM, Josh Elser  wrote:

Interesting. What makes you want to deprecate ClientConfig entirely?

I'd be worried about removing without sufficient thought of replacement
around. It would be a bit "churn-y" to introduce yet another way that
clients have to connect (since it was introduced in 1.6-ish?). Working
around the ClientConfig changes was irritating for the downstream
integrations (Hive, most notably).

Ok maybe thats a bad idea, not looking to cause pain.  Here were some
of my goals.

  * Remove commons config from API completely via deprecation cycle.
  * Introduce API that supports putting all props needed to connect to
Accumulo in an API.

I suppose if we want to keep ClientConfig class in API, then there is
no way to remove commons config via a deprecation cycle??  We can't
deprecate the extension of commons config, all we can do is just drop
it at some point.



My line of thinking is that the majority of the time, we're creating a 
ClientConfiguration by one of:


* ClientConfiguration#loadDefault()
* new ClientConfiguration(String)
* new ClientConfiguration(File)

Granted, we also inherit/expose a few other things (notably extending 
CompositeConfiguration and throwing ConfigurationException). I would be 
comfortable with dropping those w/o deprecation. I have not seen 
evidence from anyone that they are widely in use by folks (although I've 
not explicitly asked, either).

Re: [DISCUSS] Hadoop3 support target?

[Keith Turner]

On Tue, Dec 5, 2017 at 2:53 PM, Josh Elser <els...@apache.org> wrote:
> Interesting. What makes you want to deprecate ClientConfig entirely?
>
> I'd be worried about removing without sufficient thought of replacement
> around. It would be a bit "churn-y" to introduce yet another way that
> clients have to connect (since it was introduced in 1.6-ish?). Working
> around the ClientConfig changes was irritating for the downstream
> integrations (Hive, most notably).

Ok maybe thats a bad idea, not looking to cause pain.  Here were some
of my goals.

 * Remove commons config from API completely via deprecation cycle.
 * Introduce API that supports putting all props needed to connect to
Accumulo in an API.

I suppose if we want to keep ClientConfig class in API, then there is
no way to remove commons config via a deprecation cycle??  We can't
deprecate the extension of commons config, all we can do is just drop
it at some point.

>
>
> On 12/5/17 1:13 PM, Keith Turner wrote:
>>
>> I was thinking of a slightly different path forward.
>>
>>   * Add new entry point and deprecate clientconfig in 1.9
>>   * Branch 1.9 off 1.8
>>   * Stop releasing 1.8.x in favor of 1.9.x (they are the same except
>> for new API)
>>   * Release 1.9 ASAP
>>   * Drop clientconfig in 2.0.0
>>   * Release 2.0.0 early next year... maybe target March
>>
>> On Tue, Dec 5, 2017 at 12:51 PM, Josh Elser <els...@apache.org> wrote:
>>>
>>> Ok, a bridge version seems to be a general path forward. Generally this
>>> would be...
>>>
>>> * 1.8 gets relevant commons-config classes/methods deprecated
>>> * 1.9 is 1.8 with those deprecation points removed
>>> * 1.9 has commons-config shaded (maybe?)
>>>
>>> IMO, it's critical that we remove the commons-config stuff from our
>>> public
>>> API (shame this somehow was let in to begin).
>>>
>>> I think shading our use of commons-config would be a good idea and lessen
>>> our ClientConfiguration scope to being able to read from a file. Trying
>>> to
>>> support the breadth of what commons-configuration can do will just get us
>>> into more trouble.
>>>
>>>
>>> On 12/5/17 12:18 PM, Keith Turner wrote:
>>>>
>>>>
>>>> If we are going to deprecate, then it would be nice to have a
>>>> replacement.  One thing that has irked me about the current Accumulo
>>>> entry point is that one can not specify everything needed to connect
>>>> to in a single props file.  Specifically, credentials can not be
>>>> specified.  It would be really nice to have a new entry point that
>>>> allows this.
>>>>
>>>> We could release a 1.9 bridge version.  This version would be based on
>>>> 1.8 and only include a new entry point. Base it on 1.8 in order to
>>>> allow a low risk upgrade for anyone currently using 1.8.  Once people
>>>> start using 1.9 they can have code that uses the old and new entry
>>>> point running at the same time.  In 2.0 we can drop the problematic
>>>> entry point.
>>>>
>>>> Below is a commit to 1.8 where I was experimenting with a new entry
>>>> point.
>>>>
>>>>
>>>>
>>>> https://github.com/keith-turner/accumulo/commit/1c07fa62e9c57bde7e60907595d50f898d03c9d5
>>>>
>>>> This new API would need review, its rough and there are some things I
>>>> don't like about it.  Just sharing for discussion of general concept,
>>>> not advocating for this specific API.
>>>>
>>>> On Mon, Dec 4, 2017 at 6:27 PM, Dave Marion <dmario...@gmail.com> wrote:
>>>>>
>>>>>
>>>>> There is no reason that you can't mark the offending API methods as
>>>>> deprecated in a 1.8.x release, then immediately branch off of that to
>>>>> create
>>>>> a 2.0 and remove the method. Alternatively, we could decide to forego
>>>>> the
>>>>> semver rules for a specific release and make sure to point it out in
>>>>> the
>>>>> release notes.
>>>>>
>>>>> -Original Message-
>>>>> From: Josh Elser [mailto:els...@apache.org]
>>>>> Sent: Monday, December 4, 2017 6:19 PM
>>>>> To: dev@accumulo.apache.org
>>>>> Subject: Re: [DISCUSS] Hadoop3 support target?
>>>>>
>>>>> Also, just to be clear for everyone else:
>>>>>
>>>>&g

Re: [DISCUSS] Hadoop3 support target?

[Josh Elser]

Interesting. What makes you want to deprecate ClientConfig entirely?

I'd be worried about removing without sufficient thought of replacement 
around. It would be a bit "churn-y" to introduce yet another way that 
clients have to connect (since it was introduced in 1.6-ish?). Working 
around the ClientConfig changes was irritating for the downstream 
integrations (Hive, most notably).


On 12/5/17 1:13 PM, Keith Turner wrote:

I was thinking of a slightly different path forward.

  * Add new entry point and deprecate clientconfig in 1.9
  * Branch 1.9 off 1.8
  * Stop releasing 1.8.x in favor of 1.9.x (they are the same except
for new API)
  * Release 1.9 ASAP
  * Drop clientconfig in 2.0.0
  * Release 2.0.0 early next year... maybe target March

On Tue, Dec 5, 2017 at 12:51 PM, Josh Elser <els...@apache.org> wrote:

Ok, a bridge version seems to be a general path forward. Generally this
would be...

* 1.8 gets relevant commons-config classes/methods deprecated
* 1.9 is 1.8 with those deprecation points removed
* 1.9 has commons-config shaded (maybe?)

IMO, it's critical that we remove the commons-config stuff from our public
API (shame this somehow was let in to begin).

I think shading our use of commons-config would be a good idea and lessen
our ClientConfiguration scope to being able to read from a file. Trying to
support the breadth of what commons-configuration can do will just get us
into more trouble.


On 12/5/17 12:18 PM, Keith Turner wrote:


If we are going to deprecate, then it would be nice to have a
replacement.  One thing that has irked me about the current Accumulo
entry point is that one can not specify everything needed to connect
to in a single props file.  Specifically, credentials can not be
specified.  It would be really nice to have a new entry point that
allows this.

We could release a 1.9 bridge version.  This version would be based on
1.8 and only include a new entry point. Base it on 1.8 in order to
allow a low risk upgrade for anyone currently using 1.8.  Once people
start using 1.9 they can have code that uses the old and new entry
point running at the same time.  In 2.0 we can drop the problematic
entry point.

Below is a commit to 1.8 where I was experimenting with a new entry point.


https://github.com/keith-turner/accumulo/commit/1c07fa62e9c57bde7e60907595d50f898d03c9d5

This new API would need review, its rough and there are some things I
don't like about it.  Just sharing for discussion of general concept,
not advocating for this specific API.

On Mon, Dec 4, 2017 at 6:27 PM, Dave Marion <dmario...@gmail.com> wrote:


There is no reason that you can't mark the offending API methods as
deprecated in a 1.8.x release, then immediately branch off of that to create
a 2.0 and remove the method. Alternatively, we could decide to forego the
semver rules for a specific release and make sure to point it out in the
release notes.

-Original Message-
From: Josh Elser [mailto:els...@apache.org]
Sent: Monday, December 4, 2017 6:19 PM
To: dev@accumulo.apache.org
Subject: Re: [DISCUSS] Hadoop3 support target?

Also, just to be clear for everyone else:

This means that we have *no roadmap* at all for Hadoop 3 support because
Accumulo 2.0 is in a state of languish.

This is a severe enough problem to me that I would consider breaking API
compatibility and fixing the API leak in 1.7/1.8. I'm curious what people
other than Christopher think (assuming from his comments/JIRA work that he
disagrees with me).

On 12/4/17 6:12 PM, Christopher wrote:


Agreed.

On Mon, Dec 4, 2017 at 6:01 PM Josh Elser <els...@apache.org> wrote:


Ah, I'm seeing now -- didn't check my inbox appropriately.

I think the fact that code that we don't own has somehow been allowed
to be public API is the smell. That's something that needs to be
rectified sooner than later. By that measure, it can *only* land on
Accumulo 2.0 (which is going to be a major issue for the project).

On 12/4/17 5:58 PM, Josh Elser wrote:


Sorry, I don't follow. Why do you think 4611/4753 is a show-stopper?
Cuz, uh... I made it work already :)

Thanks for the JIRA cleanup. Forgot about that one.

On 12/4/17 5:55 PM, Christopher wrote:


I don't think we can support it with 1.8 or earlier, because of
some serious incompatibilities (namely, ACCUMULO-4611/4753) I think
people are still patching 1.7, so I don't think we've "officially"
EOL'd it.
I think 2.0 could require Hadoop 3, if Hadoop 3 is sufficiently
stable.

On Mon, Dec 4, 2017 at 1:14 PM Josh Elser <els...@apache.org> wrote:


What branch do we want to consider Hadoop3 support?

There is a 3.0.0-beta1 release that's been out for a while, and
Hadoop PMC has already done a 3.0.0 RC0. I think it's the right
time to start considering this.

In my poking so far, I've filed ACCUMULO-4753 which I'm working
through now. This does raise the question: where do we want to say
we support Hadoop3? 1.8 or 2.0? (have we "officially" de

Re: [DISCUSS] Hadoop3 support target?

[Keith Turner]

I was thinking of a slightly different path forward.

 * Add new entry point and deprecate clientconfig in 1.9
 * Branch 1.9 off 1.8
 * Stop releasing 1.8.x in favor of 1.9.x (they are the same except
for new API)
 * Release 1.9 ASAP
 * Drop clientconfig in 2.0.0
 * Release 2.0.0 early next year... maybe target March

On Tue, Dec 5, 2017 at 12:51 PM, Josh Elser <els...@apache.org> wrote:
> Ok, a bridge version seems to be a general path forward. Generally this
> would be...
>
> * 1.8 gets relevant commons-config classes/methods deprecated
> * 1.9 is 1.8 with those deprecation points removed
> * 1.9 has commons-config shaded (maybe?)
>
> IMO, it's critical that we remove the commons-config stuff from our public
> API (shame this somehow was let in to begin).
>
> I think shading our use of commons-config would be a good idea and lessen
> our ClientConfiguration scope to being able to read from a file. Trying to
> support the breadth of what commons-configuration can do will just get us
> into more trouble.
>
>
> On 12/5/17 12:18 PM, Keith Turner wrote:
>>
>> If we are going to deprecate, then it would be nice to have a
>> replacement.  One thing that has irked me about the current Accumulo
>> entry point is that one can not specify everything needed to connect
>> to in a single props file.  Specifically, credentials can not be
>> specified.  It would be really nice to have a new entry point that
>> allows this.
>>
>> We could release a 1.9 bridge version.  This version would be based on
>> 1.8 and only include a new entry point. Base it on 1.8 in order to
>> allow a low risk upgrade for anyone currently using 1.8.  Once people
>> start using 1.9 they can have code that uses the old and new entry
>> point running at the same time.  In 2.0 we can drop the problematic
>> entry point.
>>
>> Below is a commit to 1.8 where I was experimenting with a new entry point.
>>
>>
>> https://github.com/keith-turner/accumulo/commit/1c07fa62e9c57bde7e60907595d50f898d03c9d5
>>
>> This new API would need review, its rough and there are some things I
>> don't like about it.  Just sharing for discussion of general concept,
>> not advocating for this specific API.
>>
>> On Mon, Dec 4, 2017 at 6:27 PM, Dave Marion <dmario...@gmail.com> wrote:
>>>
>>> There is no reason that you can't mark the offending API methods as
>>> deprecated in a 1.8.x release, then immediately branch off of that to create
>>> a 2.0 and remove the method. Alternatively, we could decide to forego the
>>> semver rules for a specific release and make sure to point it out in the
>>> release notes.
>>>
>>> -Original Message-
>>> From: Josh Elser [mailto:els...@apache.org]
>>> Sent: Monday, December 4, 2017 6:19 PM
>>> To: dev@accumulo.apache.org
>>> Subject: Re: [DISCUSS] Hadoop3 support target?
>>>
>>> Also, just to be clear for everyone else:
>>>
>>> This means that we have *no roadmap* at all for Hadoop 3 support because
>>> Accumulo 2.0 is in a state of languish.
>>>
>>> This is a severe enough problem to me that I would consider breaking API
>>> compatibility and fixing the API leak in 1.7/1.8. I'm curious what people
>>> other than Christopher think (assuming from his comments/JIRA work that he
>>> disagrees with me).
>>>
>>> On 12/4/17 6:12 PM, Christopher wrote:
>>>>
>>>> Agreed.
>>>>
>>>> On Mon, Dec 4, 2017 at 6:01 PM Josh Elser <els...@apache.org> wrote:
>>>>
>>>>> Ah, I'm seeing now -- didn't check my inbox appropriately.
>>>>>
>>>>> I think the fact that code that we don't own has somehow been allowed
>>>>> to be public API is the smell. That's something that needs to be
>>>>> rectified sooner than later. By that measure, it can *only* land on
>>>>> Accumulo 2.0 (which is going to be a major issue for the project).
>>>>>
>>>>> On 12/4/17 5:58 PM, Josh Elser wrote:
>>>>>>
>>>>>> Sorry, I don't follow. Why do you think 4611/4753 is a show-stopper?
>>>>>> Cuz, uh... I made it work already :)
>>>>>>
>>>>>> Thanks for the JIRA cleanup. Forgot about that one.
>>>>>>
>>>>>> On 12/4/17 5:55 PM, Christopher wrote:
>>>>>>>
>>>>>>> I don't think we can support it with 1.8 or earlier, because of
>>>>>>> some serious incompatibilities (namely,

Re: [DISCUSS] Hadoop3 support target?

[Josh Elser]

Ok, a bridge version seems to be a general path forward. Generally this 
would be...


* 1.8 gets relevant commons-config classes/methods deprecated
* 1.9 is 1.8 with those deprecation points removed
* 1.9 has commons-config shaded (maybe?)

IMO, it's critical that we remove the commons-config stuff from our 
public API (shame this somehow was let in to begin).


I think shading our use of commons-config would be a good idea and 
lessen our ClientConfiguration scope to being able to read from a file. 
Trying to support the breadth of what commons-configuration can do will 
just get us into more trouble.


On 12/5/17 12:18 PM, Keith Turner wrote:

If we are going to deprecate, then it would be nice to have a
replacement.  One thing that has irked me about the current Accumulo
entry point is that one can not specify everything needed to connect
to in a single props file.  Specifically, credentials can not be
specified.  It would be really nice to have a new entry point that
allows this.

We could release a 1.9 bridge version.  This version would be based on
1.8 and only include a new entry point. Base it on 1.8 in order to
allow a low risk upgrade for anyone currently using 1.8.  Once people
start using 1.9 they can have code that uses the old and new entry
point running at the same time.  In 2.0 we can drop the problematic
entry point.

Below is a commit to 1.8 where I was experimenting with a new entry point.

https://github.com/keith-turner/accumulo/commit/1c07fa62e9c57bde7e60907595d50f898d03c9d5

This new API would need review, its rough and there are some things I
don't like about it.  Just sharing for discussion of general concept,
not advocating for this specific API.

On Mon, Dec 4, 2017 at 6:27 PM, Dave Marion <dmario...@gmail.com> wrote:

There is no reason that you can't mark the offending API methods as deprecated 
in a 1.8.x release, then immediately branch off of that to create a 2.0 and 
remove the method. Alternatively, we could decide to forego the semver rules 
for a specific release and make sure to point it out in the release notes.

-Original Message-
From: Josh Elser [mailto:els...@apache.org]
Sent: Monday, December 4, 2017 6:19 PM
To: dev@accumulo.apache.org
Subject: Re: [DISCUSS] Hadoop3 support target?

Also, just to be clear for everyone else:

This means that we have *no roadmap* at all for Hadoop 3 support because 
Accumulo 2.0 is in a state of languish.

This is a severe enough problem to me that I would consider breaking API 
compatibility and fixing the API leak in 1.7/1.8. I'm curious what people other 
than Christopher think (assuming from his comments/JIRA work that he disagrees 
with me).

On 12/4/17 6:12 PM, Christopher wrote:

Agreed.

On Mon, Dec 4, 2017 at 6:01 PM Josh Elser <els...@apache.org> wrote:


Ah, I'm seeing now -- didn't check my inbox appropriately.

I think the fact that code that we don't own has somehow been allowed
to be public API is the smell. That's something that needs to be
rectified sooner than later. By that measure, it can *only* land on
Accumulo 2.0 (which is going to be a major issue for the project).

On 12/4/17 5:58 PM, Josh Elser wrote:

Sorry, I don't follow. Why do you think 4611/4753 is a show-stopper?
Cuz, uh... I made it work already :)

Thanks for the JIRA cleanup. Forgot about that one.

On 12/4/17 5:55 PM, Christopher wrote:

I don't think we can support it with 1.8 or earlier, because of
some serious incompatibilities (namely, ACCUMULO-4611/4753) I think
people are still patching 1.7, so I don't think we've "officially"
EOL'd it.
I think 2.0 could require Hadoop 3, if Hadoop 3 is sufficiently stable.

On Mon, Dec 4, 2017 at 1:14 PM Josh Elser <els...@apache.org> wrote:


What branch do we want to consider Hadoop3 support?

There is a 3.0.0-beta1 release that's been out for a while, and
Hadoop PMC has already done a 3.0.0 RC0. I think it's the right
time to start considering this.

In my poking so far, I've filed ACCUMULO-4753 which I'm working
through now. This does raise the question: where do we want to say
we support Hadoop3? 1.8 or 2.0? (have we "officially" deprecated
1.7?)

- Josh

https://issues.apache.org/jira/browse/ACCUMULO-4753

Re: [DISCUSS] Hadoop3 support target?

[Keith Turner]

If we are going to deprecate, then it would be nice to have a
replacement.  One thing that has irked me about the current Accumulo
entry point is that one can not specify everything needed to connect
to in a single props file.  Specifically, credentials can not be
specified.  It would be really nice to have a new entry point that
allows this.

We could release a 1.9 bridge version.  This version would be based on
1.8 and only include a new entry point. Base it on 1.8 in order to
allow a low risk upgrade for anyone currently using 1.8.  Once people
start using 1.9 they can have code that uses the old and new entry
point running at the same time.  In 2.0 we can drop the problematic
entry point.

Below is a commit to 1.8 where I was experimenting with a new entry point.

https://github.com/keith-turner/accumulo/commit/1c07fa62e9c57bde7e60907595d50f898d03c9d5

This new API would need review, its rough and there are some things I
don't like about it.  Just sharing for discussion of general concept,
not advocating for this specific API.

On Mon, Dec 4, 2017 at 6:27 PM, Dave Marion <dmario...@gmail.com> wrote:
> There is no reason that you can't mark the offending API methods as 
> deprecated in a 1.8.x release, then immediately branch off of that to create 
> a 2.0 and remove the method. Alternatively, we could decide to forego the 
> semver rules for a specific release and make sure to point it out in the 
> release notes.
>
> -Original Message-
> From: Josh Elser [mailto:els...@apache.org]
> Sent: Monday, December 4, 2017 6:19 PM
> To: dev@accumulo.apache.org
> Subject: Re: [DISCUSS] Hadoop3 support target?
>
> Also, just to be clear for everyone else:
>
> This means that we have *no roadmap* at all for Hadoop 3 support because 
> Accumulo 2.0 is in a state of languish.
>
> This is a severe enough problem to me that I would consider breaking API 
> compatibility and fixing the API leak in 1.7/1.8. I'm curious what people 
> other than Christopher think (assuming from his comments/JIRA work that he 
> disagrees with me).
>
> On 12/4/17 6:12 PM, Christopher wrote:
>> Agreed.
>>
>> On Mon, Dec 4, 2017 at 6:01 PM Josh Elser <els...@apache.org> wrote:
>>
>>> Ah, I'm seeing now -- didn't check my inbox appropriately.
>>>
>>> I think the fact that code that we don't own has somehow been allowed
>>> to be public API is the smell. That's something that needs to be
>>> rectified sooner than later. By that measure, it can *only* land on
>>> Accumulo 2.0 (which is going to be a major issue for the project).
>>>
>>> On 12/4/17 5:58 PM, Josh Elser wrote:
>>>> Sorry, I don't follow. Why do you think 4611/4753 is a show-stopper?
>>>> Cuz, uh... I made it work already :)
>>>>
>>>> Thanks for the JIRA cleanup. Forgot about that one.
>>>>
>>>> On 12/4/17 5:55 PM, Christopher wrote:
>>>>> I don't think we can support it with 1.8 or earlier, because of
>>>>> some serious incompatibilities (namely, ACCUMULO-4611/4753) I think
>>>>> people are still patching 1.7, so I don't think we've "officially"
>>>>> EOL'd it.
>>>>> I think 2.0 could require Hadoop 3, if Hadoop 3 is sufficiently stable.
>>>>>
>>>>> On Mon, Dec 4, 2017 at 1:14 PM Josh Elser <els...@apache.org> wrote:
>>>>>
>>>>>> What branch do we want to consider Hadoop3 support?
>>>>>>
>>>>>> There is a 3.0.0-beta1 release that's been out for a while, and
>>>>>> Hadoop PMC has already done a 3.0.0 RC0. I think it's the right
>>>>>> time to start considering this.
>>>>>>
>>>>>> In my poking so far, I've filed ACCUMULO-4753 which I'm working
>>>>>> through now. This does raise the question: where do we want to say
>>>>>> we support Hadoop3? 1.8 or 2.0? (have we "officially" deprecated
>>>>>> 1.7?)
>>>>>>
>>>>>> - Josh
>>>>>>
>>>>>> https://issues.apache.org/jira/browse/ACCUMULO-4753
>>>>>>
>>>>>
>>>
>>
>

Re: [DISCUSS] Hadoop3 support target?

[Christopher]

I'm not certain what I'm supposed to be in disagreement with. I think
you've presented a fair assessment of the situation, and I agree with the
severity of the issue.

My comments in JIRA about shading not working was simply an observation
that we also need to consider the API breakage, which the shading wouldn't
fix.

If we're resigned to do the API breakage in 1.8, we can make that the
"bridge" version (supporting both Hadoop 2 and 3) by shading. Or, we can
wait until 2.0 and shade there to make that the "bridge" version (perhaps
even locking in a release timeline for 2.0... which hasn't seemed urgent
until now). Either way, shading seems the only way forward in order to
resolve this particular dependency issue.

The only other path I can see would be to not have a "bridge" version at
all and instead require upgrading Accumulo simultaneously with Hadoop. I
kind of like that option, but I don't think it's realistic for our
audiences, as it doesn't allow them to manage their upgrade risks, so the
shaded "bridge" version seems like the better option.

On Mon, Dec 4, 2017 at 6:19 PM Josh Elser  wrote:

> Also, just to be clear for everyone else:
>
> This means that we have *no roadmap* at all for Hadoop 3 support because
> Accumulo 2.0 is in a state of languish.
>
> This is a severe enough problem to me that I would consider breaking API
> compatibility and fixing the API leak in 1.7/1.8. I'm curious what
> people other than Christopher think (assuming from his comments/JIRA
> work that he disagrees with me).
>
> On 12/4/17 6:12 PM, Christopher wrote:
> > Agreed.
> >
> > On Mon, Dec 4, 2017 at 6:01 PM Josh Elser  wrote:
> >
> >> Ah, I'm seeing now -- didn't check my inbox appropriately.
> >>
> >> I think the fact that code that we don't own has somehow been allowed to
> >> be public API is the smell. That's something that needs to be rectified
> >> sooner than later. By that measure, it can *only* land on Accumulo 2.0
> >> (which is going to be a major issue for the project).
> >>
> >> On 12/4/17 5:58 PM, Josh Elser wrote:
> >>> Sorry, I don't follow. Why do you think 4611/4753 is a show-stopper?
> >>> Cuz, uh... I made it work already :)
> >>>
> >>> Thanks for the JIRA cleanup. Forgot about that one.
> >>>
> >>> On 12/4/17 5:55 PM, Christopher wrote:
>  I don't think we can support it with 1.8 or earlier, because of some
>  serious incompatibilities (namely, ACCUMULO-4611/4753)
>  I think people are still patching 1.7, so I don't think we've
>  "officially"
>  EOL'd it.
>  I think 2.0 could require Hadoop 3, if Hadoop 3 is sufficiently
> stable.
> 
>  On Mon, Dec 4, 2017 at 1:14 PM Josh Elser  wrote:
> 
> > What branch do we want to consider Hadoop3 support?
> >
> > There is a 3.0.0-beta1 release that's been out for a while, and
> Hadoop
> > PMC has already done a 3.0.0 RC0. I think it's the right time to
> start
> > considering this.
> >
> > In my poking so far, I've filed ACCUMULO-4753 which I'm working
> through
> > now. This does raise the question: where do we want to say we support
> > Hadoop3? 1.8 or 2.0? (have we "officially" deprecated 1.7?)
> >
> > - Josh
> >
> > https://issues.apache.org/jira/browse/ACCUMULO-4753
> >
> 
> >>
> >
>

RE: [DISCUSS] Hadoop3 support target?

[Dave Marion]

There is no reason that you can't mark the offending API methods as deprecated 
in a 1.8.x release, then immediately branch off of that to create a 2.0 and 
remove the method. Alternatively, we could decide to forego the semver rules 
for a specific release and make sure to point it out in the release notes.

-Original Message-
From: Josh Elser [mailto:els...@apache.org] 
Sent: Monday, December 4, 2017 6:19 PM
To: dev@accumulo.apache.org
Subject: Re: [DISCUSS] Hadoop3 support target?

Also, just to be clear for everyone else:

This means that we have *no roadmap* at all for Hadoop 3 support because 
Accumulo 2.0 is in a state of languish.

This is a severe enough problem to me that I would consider breaking API 
compatibility and fixing the API leak in 1.7/1.8. I'm curious what people other 
than Christopher think (assuming from his comments/JIRA work that he disagrees 
with me).

On 12/4/17 6:12 PM, Christopher wrote:
> Agreed.
> 
> On Mon, Dec 4, 2017 at 6:01 PM Josh Elser <els...@apache.org> wrote:
> 
>> Ah, I'm seeing now -- didn't check my inbox appropriately.
>>
>> I think the fact that code that we don't own has somehow been allowed 
>> to be public API is the smell. That's something that needs to be 
>> rectified sooner than later. By that measure, it can *only* land on 
>> Accumulo 2.0 (which is going to be a major issue for the project).
>>
>> On 12/4/17 5:58 PM, Josh Elser wrote:
>>> Sorry, I don't follow. Why do you think 4611/4753 is a show-stopper?
>>> Cuz, uh... I made it work already :)
>>>
>>> Thanks for the JIRA cleanup. Forgot about that one.
>>>
>>> On 12/4/17 5:55 PM, Christopher wrote:
>>>> I don't think we can support it with 1.8 or earlier, because of 
>>>> some serious incompatibilities (namely, ACCUMULO-4611/4753) I think 
>>>> people are still patching 1.7, so I don't think we've "officially"
>>>> EOL'd it.
>>>> I think 2.0 could require Hadoop 3, if Hadoop 3 is sufficiently stable.
>>>>
>>>> On Mon, Dec 4, 2017 at 1:14 PM Josh Elser <els...@apache.org> wrote:
>>>>
>>>>> What branch do we want to consider Hadoop3 support?
>>>>>
>>>>> There is a 3.0.0-beta1 release that's been out for a while, and 
>>>>> Hadoop PMC has already done a 3.0.0 RC0. I think it's the right 
>>>>> time to start considering this.
>>>>>
>>>>> In my poking so far, I've filed ACCUMULO-4753 which I'm working 
>>>>> through now. This does raise the question: where do we want to say 
>>>>> we support Hadoop3? 1.8 or 2.0? (have we "officially" deprecated 
>>>>> 1.7?)
>>>>>
>>>>> - Josh
>>>>>
>>>>> https://issues.apache.org/jira/browse/ACCUMULO-4753
>>>>>
>>>>
>>
> 

Re: [DISCUSS] Hadoop3 support target?

[Josh Elser]

Also, just to be clear for everyone else:

This means that we have *no roadmap* at all for Hadoop 3 support because 
Accumulo 2.0 is in a state of languish.


This is a severe enough problem to me that I would consider breaking API 
compatibility and fixing the API leak in 1.7/1.8. I'm curious what 
people other than Christopher think (assuming from his comments/JIRA 
work that he disagrees with me).


On 12/4/17 6:12 PM, Christopher wrote:

Agreed.

On Mon, Dec 4, 2017 at 6:01 PM Josh Elser  wrote:


Ah, I'm seeing now -- didn't check my inbox appropriately.

I think the fact that code that we don't own has somehow been allowed to
be public API is the smell. That's something that needs to be rectified
sooner than later. By that measure, it can *only* land on Accumulo 2.0
(which is going to be a major issue for the project).

On 12/4/17 5:58 PM, Josh Elser wrote:

Sorry, I don't follow. Why do you think 4611/4753 is a show-stopper?
Cuz, uh... I made it work already :)

Thanks for the JIRA cleanup. Forgot about that one.

On 12/4/17 5:55 PM, Christopher wrote:

I don't think we can support it with 1.8 or earlier, because of some
serious incompatibilities (namely, ACCUMULO-4611/4753)
I think people are still patching 1.7, so I don't think we've
"officially"
EOL'd it.
I think 2.0 could require Hadoop 3, if Hadoop 3 is sufficiently stable.

On Mon, Dec 4, 2017 at 1:14 PM Josh Elser  wrote:


What branch do we want to consider Hadoop3 support?

There is a 3.0.0-beta1 release that's been out for a while, and Hadoop
PMC has already done a 3.0.0 RC0. I think it's the right time to start
considering this.

In my poking so far, I've filed ACCUMULO-4753 which I'm working through
now. This does raise the question: where do we want to say we support
Hadoop3? 1.8 or 2.0? (have we "officially" deprecated 1.7?)

- Josh

https://issues.apache.org/jira/browse/ACCUMULO-4753

Re: [DISCUSS] Hadoop3 support target?

[Christopher]

Agreed.

On Mon, Dec 4, 2017 at 6:01 PM Josh Elser  wrote:

> Ah, I'm seeing now -- didn't check my inbox appropriately.
>
> I think the fact that code that we don't own has somehow been allowed to
> be public API is the smell. That's something that needs to be rectified
> sooner than later. By that measure, it can *only* land on Accumulo 2.0
> (which is going to be a major issue for the project).
>
> On 12/4/17 5:58 PM, Josh Elser wrote:
> > Sorry, I don't follow. Why do you think 4611/4753 is a show-stopper?
> > Cuz, uh... I made it work already :)
> >
> > Thanks for the JIRA cleanup. Forgot about that one.
> >
> > On 12/4/17 5:55 PM, Christopher wrote:
> >> I don't think we can support it with 1.8 or earlier, because of some
> >> serious incompatibilities (namely, ACCUMULO-4611/4753)
> >> I think people are still patching 1.7, so I don't think we've
> >> "officially"
> >> EOL'd it.
> >> I think 2.0 could require Hadoop 3, if Hadoop 3 is sufficiently stable.
> >>
> >> On Mon, Dec 4, 2017 at 1:14 PM Josh Elser  wrote:
> >>
> >>> What branch do we want to consider Hadoop3 support?
> >>>
> >>> There is a 3.0.0-beta1 release that's been out for a while, and Hadoop
> >>> PMC has already done a 3.0.0 RC0. I think it's the right time to start
> >>> considering this.
> >>>
> >>> In my poking so far, I've filed ACCUMULO-4753 which I'm working through
> >>> now. This does raise the question: where do we want to say we support
> >>> Hadoop3? 1.8 or 2.0? (have we "officially" deprecated 1.7?)
> >>>
> >>> - Josh
> >>>
> >>> https://issues.apache.org/jira/browse/ACCUMULO-4753
> >>>
> >>
>

Re: [DISCUSS] Hadoop3 support target?

[Josh Elser]

Ah, I'm seeing now -- didn't check my inbox appropriately.

I think the fact that code that we don't own has somehow been allowed to 
be public API is the smell. That's something that needs to be rectified 
sooner than later. By that measure, it can *only* land on Accumulo 2.0 
(which is going to be a major issue for the project).


On 12/4/17 5:58 PM, Josh Elser wrote:
Sorry, I don't follow. Why do you think 4611/4753 is a show-stopper? 
Cuz, uh... I made it work already :)


Thanks for the JIRA cleanup. Forgot about that one.

On 12/4/17 5:55 PM, Christopher wrote:

I don't think we can support it with 1.8 or earlier, because of some
serious incompatibilities (namely, ACCUMULO-4611/4753)
I think people are still patching 1.7, so I don't think we've 
"officially"

EOL'd it.
I think 2.0 could require Hadoop 3, if Hadoop 3 is sufficiently stable.

On Mon, Dec 4, 2017 at 1:14 PM Josh Elser  wrote:


What branch do we want to consider Hadoop3 support?

There is a 3.0.0-beta1 release that's been out for a while, and Hadoop
PMC has already done a 3.0.0 RC0. I think it's the right time to start
considering this.

In my poking so far, I've filed ACCUMULO-4753 which I'm working through
now. This does raise the question: where do we want to say we support
Hadoop3? 1.8 or 2.0? (have we "officially" deprecated 1.7?)

- Josh

https://issues.apache.org/jira/browse/ACCUMULO-4753

Re: [DISCUSS] Hadoop3 support target?

[Josh Elser]

Sorry, I don't follow. Why do you think 4611/4753 is a show-stopper? 
Cuz, uh... I made it work already :)


Thanks for the JIRA cleanup. Forgot about that one.

On 12/4/17 5:55 PM, Christopher wrote:

I don't think we can support it with 1.8 or earlier, because of some
serious incompatibilities (namely, ACCUMULO-4611/4753)
I think people are still patching 1.7, so I don't think we've "officially"
EOL'd it.
I think 2.0 could require Hadoop 3, if Hadoop 3 is sufficiently stable.

On Mon, Dec 4, 2017 at 1:14 PM Josh Elser  wrote:


What branch do we want to consider Hadoop3 support?

There is a 3.0.0-beta1 release that's been out for a while, and Hadoop
PMC has already done a 3.0.0 RC0. I think it's the right time to start
considering this.

In my poking so far, I've filed ACCUMULO-4753 which I'm working through
now. This does raise the question: where do we want to say we support
Hadoop3? 1.8 or 2.0? (have we "officially" deprecated 1.7?)

- Josh

https://issues.apache.org/jira/browse/ACCUMULO-4753

Re: [DISCUSS] Hadoop3 support target?

[Christopher]

I don't think we can support it with 1.8 or earlier, because of some
serious incompatibilities (namely, ACCUMULO-4611/4753)
I think people are still patching 1.7, so I don't think we've "officially"
EOL'd it.
I think 2.0 could require Hadoop 3, if Hadoop 3 is sufficiently stable.

On Mon, Dec 4, 2017 at 1:14 PM Josh Elser  wrote:

> What branch do we want to consider Hadoop3 support?
>
> There is a 3.0.0-beta1 release that's been out for a while, and Hadoop
> PMC has already done a 3.0.0 RC0. I think it's the right time to start
> considering this.
>
> In my poking so far, I've filed ACCUMULO-4753 which I'm working through
> now. This does raise the question: where do we want to say we support
> Hadoop3? 1.8 or 2.0? (have we "officially" deprecated 1.7?)
>
> - Josh
>
> https://issues.apache.org/jira/browse/ACCUMULO-4753
>

[DISCUSS] Hadoop3 support target?

[Josh Elser]

What branch do we want to consider Hadoop3 support?

There is a 3.0.0-beta1 release that's been out for a while, and Hadoop 
PMC has already done a 3.0.0 RC0. I think it's the right time to start 
considering this.


In my poking so far, I've filed ACCUMULO-4753 which I'm working through 
now. This does raise the question: where do we want to say we support 
Hadoop3? 1.8 or 2.0? (have we "officially" deprecated 1.7?)


- Josh

https://issues.apache.org/jira/browse/ACCUMULO-4753