On Fri, Mar 28, 2014 at 3:14 PM, Josh Elser <[email protected]> wrote:
> Not even the addition of a new interface, Christopher? I'd very much like > to have an interface that we can get in 1.6.0 at a minimum. I wouldn't even > push for any deprecation of what's currently in place. > W/o deprecation it seems very confusing. The intent is that users should use the new one, but the old one is not deprecated. If someone is completely new to this, how will they know which option to use? Once you get down in the weeds of working on this, do you think you might end wanting something very different? > On Mar 28, 2014 10:02 AM, "Christopher" <[email protected]> wrote: > > > I don't think any of this should be done for 1.6.0, but I like the > > idea of creating a separate cluster interface for testing. I think it > > should be integrated into the accumulo-maven-plugin, also. I think the > > idea should be hammered out, and tested as a separate thing, to > > experiment with the options, and provided as a complete feature for > > the next major release. If it would change packaging dependencies, it > > shouldn't even be done for 1.6.x bugfix releases. > > > > -- > > Christopher L Tubbs II > > http://gravatar.com/ctubbsii > > > > > > On Fri, Mar 28, 2014 at 12:24 PM, Josh Elser <[email protected]> > wrote: > > > Oh, I like that idea, Bill & Sean. > > > > > > Package: org.apache.accumulo.cluster > > > Public API: org.apache.accumulo.cluster.AccumuloCluster > > > MAC: org.apache.accumulo.cluster.mini.MiniAccumuloCluster (implements > > > AccumuloCluster, allows for backwards compat) > > > Yarn: org.apache.accumulo.cluster.yarn > > > Docker: ... > > > Mesos: ... > > > > > > etc etc etc. > > > > > > One question in my mind, do we keep the maven module > > 'accumulo-minicluster'? > > > I would imagine that if we struck the 'mini' portion from 1.6 that > would > > > create some confusion. Would it be worth the indirection to rename > > > accumulo-minicluster to accumulo-cluster and then create a new > > > accumulo-minicluster module that depends on accumulo-minicluster (but > > > contains no code itself) to preserve the 1.4 and 1.5 poms to generally > > work > > > with a version bump? I'm not sure if Maven would be happy with that or > do > > > what I think it "should". > > > > > > > > > On 3/28/14, 6:26 AM, Bill Havanki wrote: > > >> > > >> I've been watching the conversation on the side, but I wanted to > mention > > >> that it seems the focus isn't so much on "mini" clusters anymore. > You're > > >> thinking of programmatic cluster management, whether one node or many. > > The > > >> idea of a basic cluster management interface, with MAC as an > > >> implementation, is promising. A package name of just "cluster" could > > work. > > >> > > >> Carry on :) > > >> > > >> Bill H > > >> > > >> > > >> On Fri, Mar 28, 2014 at 12:39 AM, Sean Busbey > > >> <[email protected]>wrote: > > >> > > >>> If you decide to go the mapred/mapreduce way, you could go with the > > >>> package > > >>> name "mini". > > >>> > > >>> alternatively, we can do a multi-stage change out > > >>> > > >>> 1) 1.6.x: introduce TestAccumuloCluster interface, @deprecate > > >>> MiniAccumuloCluster class and make it implement TestAccumuloCluster > > >>> > > >>> 2) 1.6 + major: change MiniAccumuloCluster to an interface that > extends > > >>> TestAccumuloCluster, @deprecate TestAccumuloCluster > > >>> > > >>> 3) 1.6 + 2 major: remove TestAccumuloCluster > > >>> > > >>> Or just go with TestAccumuloCluster as the interface, have > > >>> MiniAccumuloCluster as the local pseudo distributed implementation, > and > > >>> then call your new one something like YarnAccumuloCluster. > > >>> > > >>> In that case we could use the deprecation cycle to move the MAC class > > out > > >>> of the public api. > > >>> > > >>> > > >>> On Thu, Mar 27, 2014 at 6:48 PM, Josh Elser <[email protected]> > > wrote: > > >>> > > >>>> Thoughts on if this would be an acceptable change for 1.6.0 to > > alleviate > > >>>> future cruft? > > >>>> > > >>>> Suggestions on the new package and/or class name would be greatly > > >>>> appreciated over "NewMiniAccumuloC*". > > >>>> > > >>>> > > >>>> On 3/26/14, 3:37 PM, Josh Elser wrote: > > >>>> > > >>>>> Those who are interested: check out > > >>>>> https://github.com/joshelser/accumulo/commit/ > > >>>>> 9f63cf32559ab514a69ff2c6b02acef9c9cbb4e8 > > >>>>> > > >>>>> > > >>>>> tl;dr I could create some real interfaces for the cluster and > config, > > >>>>> which are "hidden" under the covers by the 1.4 and 1.5 > > >>>>> MiniAccumuloCluster and MiniAccumuloConfig classes. This de-couples > > the > > >>>>> default implementation, gives us the ability to hide > "implementation > > >>>>> details" if wanted, and moves us towards some factory methods > instead > > >>>>> of > > >>>>> calling a class directly. > > >>>>> > > >>>>> Thoughts? > > >>>>> > > >>>>> On 3/26/14, 1:21 PM, Josh Elser wrote: > > >>>>> > > >>>>>> Yes, very much experimental at this point. > > >>>>>> > > >>>>>> What I'm most concerned about is having reasonable hooks up front, > > not > > >>>>>> trying to make an implementation for inclusion 1.6.0. > > >>>>>> > > >>>>>> Regarding additions, the implementations already contains most > > things > > >>>>>> I > > >>>>>> would want to expose. I haven't come up with anything that would > be > > >>>>>> generally returned through the "API" rather than through this > > proposed > > >>>>>> implementation (e.g. YARN connection information) > > >>>>>> > > >>>>>> On 3/26/14, 11:57 AM, Keith Turner wrote: > > >>>>>> > > >>>>>>> What you are trying to do sounds interesting. It also sounds > > >>>>>>> experimental > > >>>>>>> and in the early stages. Is there anything specific you think > > >>>>>>> should be > > >>>>>>> done for 1.6.0 w/ regards to MAC API? > > >>>>>>> > > >>>>>>> > > >>>>>>> On Wed, Mar 26, 2014 at 2:26 PM, Josh Elser < > [email protected]> > > >>>>>>> wrote: > > >>>>>>> > > >>>>>>> On 3/26/14, 11:13 AM, Keith Turner wrote: > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> On Wed, Mar 26, 2014 at 2:05 PM, Josh Elser < > > [email protected]> > > >>>>>>>>> > > >>>>>>>>> wrote: > > >>>>>>>>> > > >>>>>>>>> On 3/26/14, 10:57 AM, Keith Turner wrote: > > >>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> Can you give an example of what you are thinking of? I > don't > > >>>>>>>>>> understand > > >>>>>>>>>> > > >>>>>>>>>>> you > > >>>>>>>>>>> viewpoint either > > >>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>>> Sure. One limitation of MAC, in general as a testing > harness, > > >>>>>>>>>>> is > > >>>>>>>>>> > > >>>>>>>>>> that it > > >>>>>>>>>> doesn't adequately exercise multi-node implementations. You > can > > >>>>>>>>>> run > > >>>>>>>>>> multiple tservers, but they are all on the same host which > > limits > > >>> > > >>> the > > >>>>>>>>>> > > >>>>>>>>>> validity of a "robust" test. This is my immediate goal. > > >>>>>>>>>> > > >>>>>>>>>> Multi-node deployments are capable using something like Mesos > or > > >>>>>>>>>> Yarn. > > >>>>>>>>>> Given that there is already functioning support to deploy > > Accumulo > > >>> > > >>> on > > >>>>>>>>>> > > >>>>>>>>>> Yarn, > > >>>>>>>>>> this was my goal. > > >>>>>>>>>> > > >>>>>>>>>> My goal is to be able to have the ability to run all of our > > >>>>>>>>>> AbstractMacIT > > >>>>>>>>>> implementations against "real" hardware without changing a > > single > > >>>>>>>>>> line of > > >>>>>>>>>> test code (ok - maybe a line or two to do injection of the MAC > > >>>>>>>>>> implementation). The point is, I believe there could be a huge > > >>>>>>>>>> testing > > >>>>>>>>>> gain > > >>>>>>>>>> from being able to write tests which leverage yarn, have the > > same > > >>>>>>>>>> programmatic configuration API from MAC, and provide near > "real" > > >>>>>>>>>> Accumulo > > >>>>>>>>>> semantics. > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> Ok so you want to MAC to be an interface so that you can > > provide > > >>>>>>>>>> a > > >>>>>>>>> > > >>>>>>>>> completely different implementation? > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> Correct. Some things would serve well in a common abstract > base > > >>> > > >>> (e.g. > > >>>>>>>> > > >>>>>>>> numTservers, siteXml configuration), but all the nonsense about > > >>>>>>>> creating > > >>>>>>>> directory structures and managing Processes is implementation > > >>> > > >>> specific. > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> Perhaps I could create a new interface that the current > > >>> > > >>> implementation > > >>>>>>>> > > >>>>>>>> implements which still provides the same semantics from 1.4 and > > 1.5. > > >>>>>>>> Let me > > >>>>>>>> see if I can mock up what I'm thinking -- that will probably be > > >>>>>>>> easier than > > >>>>>>>> me trying to write it out. > > >>>>>>>> > > >>>>>>>> > > >>>>>>> > > >>> > > >> > > >> > > >> > > > > > >
