If the new interface is not in the public API, then I think adding it (without deprecating MAC) is fine.
That way it can evolve if needed and we can add it to the public API on a later release. On Fri, Mar 28, 2014 at 3:39 PM, Christopher <[email protected]> wrote: > But... without more time to fully develop the requirements for the > interface, with a few implementations, it's probably going to change > anyway. I think even adding the interface could complicate the > follow-on work. But... *shrug*.... maybe you can have guarantees that > the interface will stay as is (same package, same methods, same name, > etc.)? > > -- > Christopher L Tubbs II > http://gravatar.com/ctubbsii > > > On Fri, Mar 28, 2014 at 3:14 PM, Josh Elser <[email protected]> wrote: > > Not even the addition of a new interface, Christopher? I'd very much like > > to have an interface that we can get in 1.6.0 at a minimum. I wouldn't > even > > push for any deprecation of what's currently in place. > > On Mar 28, 2014 10:02 AM, "Christopher" <[email protected]> wrote: > > > >> I don't think any of this should be done for 1.6.0, but I like the > >> idea of creating a separate cluster interface for testing. I think it > >> should be integrated into the accumulo-maven-plugin, also. I think the > >> idea should be hammered out, and tested as a separate thing, to > >> experiment with the options, and provided as a complete feature for > >> the next major release. If it would change packaging dependencies, it > >> shouldn't even be done for 1.6.x bugfix releases. > >> > >> -- > >> Christopher L Tubbs II > >> http://gravatar.com/ctubbsii > >> > >> > >> On Fri, Mar 28, 2014 at 12:24 PM, Josh Elser <[email protected]> > wrote: > >> > Oh, I like that idea, Bill & Sean. > >> > > >> > Package: org.apache.accumulo.cluster > >> > Public API: org.apache.accumulo.cluster.AccumuloCluster > >> > MAC: org.apache.accumulo.cluster.mini.MiniAccumuloCluster (implements > >> > AccumuloCluster, allows for backwards compat) > >> > Yarn: org.apache.accumulo.cluster.yarn > >> > Docker: ... > >> > Mesos: ... > >> > > >> > etc etc etc. > >> > > >> > One question in my mind, do we keep the maven module > >> 'accumulo-minicluster'? > >> > I would imagine that if we struck the 'mini' portion from 1.6 that > would > >> > create some confusion. Would it be worth the indirection to rename > >> > accumulo-minicluster to accumulo-cluster and then create a new > >> > accumulo-minicluster module that depends on accumulo-minicluster (but > >> > contains no code itself) to preserve the 1.4 and 1.5 poms to generally > >> work > >> > with a version bump? I'm not sure if Maven would be happy with that > or do > >> > what I think it "should". > >> > > >> > > >> > On 3/28/14, 6:26 AM, Bill Havanki wrote: > >> >> > >> >> I've been watching the conversation on the side, but I wanted to > mention > >> >> that it seems the focus isn't so much on "mini" clusters anymore. > You're > >> >> thinking of programmatic cluster management, whether one node or > many. > >> The > >> >> idea of a basic cluster management interface, with MAC as an > >> >> implementation, is promising. A package name of just "cluster" could > >> work. > >> >> > >> >> Carry on :) > >> >> > >> >> Bill H > >> >> > >> >> > >> >> On Fri, Mar 28, 2014 at 12:39 AM, Sean Busbey > >> >> <[email protected]>wrote: > >> >> > >> >>> If you decide to go the mapred/mapreduce way, you could go with the > >> >>> package > >> >>> name "mini". > >> >>> > >> >>> alternatively, we can do a multi-stage change out > >> >>> > >> >>> 1) 1.6.x: introduce TestAccumuloCluster interface, @deprecate > >> >>> MiniAccumuloCluster class and make it implement TestAccumuloCluster > >> >>> > >> >>> 2) 1.6 + major: change MiniAccumuloCluster to an interface that > extends > >> >>> TestAccumuloCluster, @deprecate TestAccumuloCluster > >> >>> > >> >>> 3) 1.6 + 2 major: remove TestAccumuloCluster > >> >>> > >> >>> Or just go with TestAccumuloCluster as the interface, have > >> >>> MiniAccumuloCluster as the local pseudo distributed implementation, > and > >> >>> then call your new one something like YarnAccumuloCluster. > >> >>> > >> >>> In that case we could use the deprecation cycle to move the MAC > class > >> out > >> >>> of the public api. > >> >>> > >> >>> > >> >>> On Thu, Mar 27, 2014 at 6:48 PM, Josh Elser <[email protected]> > >> wrote: > >> >>> > >> >>>> Thoughts on if this would be an acceptable change for 1.6.0 to > >> alleviate > >> >>>> future cruft? > >> >>>> > >> >>>> Suggestions on the new package and/or class name would be greatly > >> >>>> appreciated over "NewMiniAccumuloC*". > >> >>>> > >> >>>> > >> >>>> On 3/26/14, 3:37 PM, Josh Elser wrote: > >> >>>> > >> >>>>> Those who are interested: check out > >> >>>>> https://github.com/joshelser/accumulo/commit/ > >> >>>>> 9f63cf32559ab514a69ff2c6b02acef9c9cbb4e8 > >> >>>>> > >> >>>>> > >> >>>>> tl;dr I could create some real interfaces for the cluster and > config, > >> >>>>> which are "hidden" under the covers by the 1.4 and 1.5 > >> >>>>> MiniAccumuloCluster and MiniAccumuloConfig classes. This > de-couples > >> the > >> >>>>> default implementation, gives us the ability to hide > "implementation > >> >>>>> details" if wanted, and moves us towards some factory methods > instead > >> >>>>> of > >> >>>>> calling a class directly. > >> >>>>> > >> >>>>> Thoughts? > >> >>>>> > >> >>>>> On 3/26/14, 1:21 PM, Josh Elser wrote: > >> >>>>> > >> >>>>>> Yes, very much experimental at this point. > >> >>>>>> > >> >>>>>> What I'm most concerned about is having reasonable hooks up > front, > >> not > >> >>>>>> trying to make an implementation for inclusion 1.6.0. > >> >>>>>> > >> >>>>>> Regarding additions, the implementations already contains most > >> things > >> >>>>>> I > >> >>>>>> would want to expose. I haven't come up with anything that would > be > >> >>>>>> generally returned through the "API" rather than through this > >> proposed > >> >>>>>> implementation (e.g. YARN connection information) > >> >>>>>> > >> >>>>>> On 3/26/14, 11:57 AM, Keith Turner wrote: > >> >>>>>> > >> >>>>>>> What you are trying to do sounds interesting. It also sounds > >> >>>>>>> experimental > >> >>>>>>> and in the early stages. Is there anything specific you think > >> >>>>>>> should be > >> >>>>>>> done for 1.6.0 w/ regards to MAC API? > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> On Wed, Mar 26, 2014 at 2:26 PM, Josh Elser < > [email protected]> > >> >>>>>>> wrote: > >> >>>>>>> > >> >>>>>>> On 3/26/14, 11:13 AM, Keith Turner wrote: > >> >>>>>>>> > >> >>>>>>>> > >> >>>>>>>> On Wed, Mar 26, 2014 at 2:05 PM, Josh Elser < > >> [email protected]> > >> >>>>>>>>> > >> >>>>>>>>> wrote: > >> >>>>>>>>> > >> >>>>>>>>> On 3/26/14, 10:57 AM, Keith Turner wrote: > >> >>>>>>>>> > >> >>>>>>>>>> > >> >>>>>>>>>> Can you give an example of what you are thinking of? I > don't > >> >>>>>>>>>> understand > >> >>>>>>>>>> > >> >>>>>>>>>>> you > >> >>>>>>>>>>> viewpoint either > >> >>>>>>>>>>> > >> >>>>>>>>>>> > >> >>>>>>>>>>> Sure. One limitation of MAC, in general as a testing > harness, > >> >>>>>>>>>>> is > >> >>>>>>>>>> > >> >>>>>>>>>> that it > >> >>>>>>>>>> doesn't adequately exercise multi-node implementations. You > can > >> >>>>>>>>>> run > >> >>>>>>>>>> multiple tservers, but they are all on the same host which > >> limits > >> >>> > >> >>> the > >> >>>>>>>>>> > >> >>>>>>>>>> validity of a "robust" test. This is my immediate goal. > >> >>>>>>>>>> > >> >>>>>>>>>> Multi-node deployments are capable using something like > Mesos or > >> >>>>>>>>>> Yarn. > >> >>>>>>>>>> Given that there is already functioning support to deploy > >> Accumulo > >> >>> > >> >>> on > >> >>>>>>>>>> > >> >>>>>>>>>> Yarn, > >> >>>>>>>>>> this was my goal. > >> >>>>>>>>>> > >> >>>>>>>>>> My goal is to be able to have the ability to run all of our > >> >>>>>>>>>> AbstractMacIT > >> >>>>>>>>>> implementations against "real" hardware without changing a > >> single > >> >>>>>>>>>> line of > >> >>>>>>>>>> test code (ok - maybe a line or two to do injection of the > MAC > >> >>>>>>>>>> implementation). The point is, I believe there could be a > huge > >> >>>>>>>>>> testing > >> >>>>>>>>>> gain > >> >>>>>>>>>> from being able to write tests which leverage yarn, have the > >> same > >> >>>>>>>>>> programmatic configuration API from MAC, and provide near > "real" > >> >>>>>>>>>> Accumulo > >> >>>>>>>>>> semantics. > >> >>>>>>>>>> > >> >>>>>>>>>> > >> >>>>>>>>>> Ok so you want to MAC to be an interface so that you can > >> provide > >> >>>>>>>>>> a > >> >>>>>>>>> > >> >>>>>>>>> completely different implementation? > >> >>>>>>>>> > >> >>>>>>>>> > >> >>>>>>>>> Correct. Some things would serve well in a common abstract > base > >> >>> > >> >>> (e.g. > >> >>>>>>>> > >> >>>>>>>> numTservers, siteXml configuration), but all the nonsense about > >> >>>>>>>> creating > >> >>>>>>>> directory structures and managing Processes is implementation > >> >>> > >> >>> specific. > >> >>>>>>>> > >> >>>>>>>> > >> >>>>>>>> Perhaps I could create a new interface that the current > >> >>> > >> >>> implementation > >> >>>>>>>> > >> >>>>>>>> implements which still provides the same semantics from 1.4 and > >> 1.5. > >> >>>>>>>> Let me > >> >>>>>>>> see if I can mock up what I'm thinking -- that will probably be > >> >>>>>>>> easier than > >> >>>>>>>> me trying to write it out. > >> >>>>>>>> > >> >>>>>>>> > >> >>>>>>> > >> >>> > >> >> > >> >> > >> >> > >> > > >> >
