On Tue, Jun 9, 2015 at 5:29 PM, Josh Elser <[email protected]> wrote:
> On Jun 9, 2015 2:46 PM, "Keith Turner" <[email protected]> wrote: > > > > On Mon, Jun 8, 2015 at 11:42 PM, Ryan Leary <[email protected]> wrote: > > > > > Just an anecdote from someone who has been bitten by mock more than a > > > couple times. I would try to deprecate it in 1.8 and remove in 2.0 if > at > > > all possible. People really shouldn't write tests against it. > > > > > > > +1 > > > > It should be deprecated ASAP inorder to clearly communicate its status as > > unmaintained. If its deprecated in 1.8, its eligible to be dropped in > 2.0 > > but does not have to be dropped. When its dropped can be a separate > > decision. > > > > Doesn't semver require a major version of deprecation? Still def deprecate > immediately, remove eventually. > No. Christopher had proposed that more strict requirement, but we eventually just went w/ unmodified semver. The following is from the semver page. Before you completely remove the functionality in a new major release there should be at least one minor release that contains the deprecation so that users can smoothly transition to the new API. > > > > > > > > > > On Jun 8, 2015, at 10:10 PM, Sean Busbey <[email protected]> > wrote: > > > > > > > > Josh's comment below made me realize we still haven't formally > deprecated > > > > MockAccumulo. > > > > > > > > What do folks think about doing it soon-ish with an aim of removing > it in > > > > Accumulo 3.0? (that's version three, so that it can remain deprecated > for > > > > all of version 2). > > > > > > > > > > > > -Sean > > > > > > > >> On Sun, Jun 7, 2015 at 12:37 PM, Josh Elser <[email protected]> > > > wrote: > > > >> > > > >> MiniAccumulo, yes. MockAccumulo, no. In general, we've near > completely > > > >> moved away from MockAccumulo. I wouldn't be surprised if it gets > > > deprecated > > > >> and removed soon. > > > >> > > > >> > > > >> > > > > > https://github.com/apache/accumulo/blob/1.7/test/src/test/java/org/apache/accumulo/test/functional/KerberosIT.java > > > >> > > > >> Apache Directory provides a MiniKdc that can be used easily w/ > > > >> MiniAccumulo. Many of the integration tests have already been > altered to > > > >> support running w/ or w/o kerberos. > > > >> > > > >> James Hughes wrote: > > > >> > > > >>> Hi all, > > > >>> > > > >>> For GeoMesa, stats writing is quite secondary and optional, so we > can > > > >>> sort that out as a follow-on to seeing GeoMesa work with Accumulo > 1.7. > > > >>> > > > >>> I haven't had a chance to read in details yet, so forgive me if > this is > > > >>> covered in the docs. Does either Mock or MiniAccumulo provide > enough > > > >>> hooks to test out Kerberos integration effectively? I suppose I'm > > > >>> really asking what kind of testing environment a project like > GeoMesa > > > >>> would need to use to test out Accumulo 1.7. > > > >>> > > > >>> Even though MockAccumulo has a number of limitations, it is rather > > > >>> effective in unit tests which can be part of a quick build. > > > >>> > > > >>> Thanks, > > > >>> > > > >>> Jim > > > >>> > > > >>> On Sat, Jun 6, 2015 at 11:14 PM, Xu (Simon) Chen < > [email protected] > > > >>> <mailto:[email protected]>> wrote: > > > >>> > > > >>> Nope, I am running the example as what the readme file > suggested: > > > >>> > > > >>> java -cp ./target/geomesa-quickstart-1.0-SNAPSHOT.jar > > > >>> org.geomesa.QuickStart -instanceId somecloud -zookeepers > > > >>> "zoo1:2181,zoo2:2181,zoo3:2181" -user someuser -password somepwd > > > >>> -tableName sometable > > > >>> > > > >>> I'll raise this question with the geomesa folks, but you're > right > > > that > > > >>> I can ignore it for now... > > > >>> > > > >>> Thanks! > > > >>> -Simon > > > >>> > > > >>> > > > >>> On Sat, Jun 6, 2015 at 11:00 PM, Josh Elser < > [email protected] > > > >>> <mailto:[email protected]>> wrote: > > > >>>> Are you running it via `mvn exec:java` by chance or netbeans? > > > >>> > > > >>> > > > > > http://mail-archives.apache.org/mod_mbox/accumulo-user/201411.mbox/%[email protected]%3E > > > >>>> > > > >>>> If that's just a background thread writing in Stats, it might > > > >>> just be a > > > >>>> factor of how you're invoking the program and you can ignore it. > > > >>> I don't > > > >>>> know enough about the inner-workings of GeoMesa to say one way or > > > >>> the other. > > > >>>> > > > >>>> > > > >>>> Xu (Simon) Chen wrote: > > > >>>>> > > > >>>>> Josh, > > > >>>>> > > > >>>>> Everything works well, except for one thing :-) > > > >>>>> > > > >>>>> I am running geomesa-quickstart program that ingest some data > > > >>> and then > > > >>>>> perform a simple query: > > > >>>>> https://github.com/geomesa/geomesa-quickstart > > > >>>>> > > > >>>>> For some reason, the following error is emitted consistently at > > > >>> the > > > >>>>> end of the execution, after outputting the correct result: > > > >>>>> 15/06/07 00:29:22 INFO zookeeper.ZooCache: Zookeeper error, will > > > >>> retry > > > >>>>> java.lang.InterruptedException > > > >>>>> at java.lang.Object.wait(Native Method) > > > >>>>> at java.lang.Object.wait(Object.java:503) > > > >>>>> at > > > >>> org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1342) > > > >>>>> at > > > >>> org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1036) > > > >>>>> at > > > >>> > org.apache.accumulo.fate.zookeeper.ZooCache$2.run(ZooCache.java:264) > > > >>>>> at > > > >>> > org.apache.accumulo.fate.zookeeper.ZooCache.retry(ZooCache.java:162) > > > >>>>> at > > > >>>>> > org.apache.accumulo.fate.zookeeper.ZooCache.get(ZooCache.java:289) > > > >>>>> at > > > >>>>> > org.apache.accumulo.fate.zookeeper.ZooCache.get(ZooCache.java:238) > > > >>>>> at > > > >>> > > > >>> > > > > org.apache.accumulo.core.client.impl.Tables.getTableState(Tables.java:180) > > > >>>>> at > > > >>> > > > >>> > > > > > org.apache.accumulo.core.client.impl.ConnectorImpl.getTableId(ConnectorImpl.java:82) > > > >>>>> at > > > >>> > > > >>> > > > > > org.apache.accumulo.core.client.impl.ConnectorImpl.createBatchWriter(ConnectorImpl.java:128) > > > >>>>> at > > > >>> > > > >>> > > > > > org.locationtech.geomesa.core.stats.StatWriter$$anonfun$write$2.apply(StatWriter.scala:174) > > > >>>>> at > > > >>> > > > >>> > > > > > org.locationtech.geomesa.core.stats.StatWriter$$anonfun$write$2.apply(StatWriter.scala:156) > > > >>>>> at > > > >>> scala.collection.immutable.Map$Map1.foreach(Map.scala:109) > > > >>>>> at > > > >>> > > > >>> > > > > org.locationtech.geomesa.core.stats.StatWriter$.write(StatWriter.scala:156) > > > >>>>> at > > > >>> > > > >>> > > > > > org.locationtech.geomesa.core.stats.StatWriter$.drainQueue(StatWriter.scala:148) > > > >>>>> at > > > >>> > > > >>> > > > > org.locationtech.geomesa.core.stats.StatWriter$.run(StatWriter.scala:116) > > > >>>>> at > > > >>> > > > >>> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > > > >>>>> at > > > >>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > > > >>>>> at > > > >>> > > > >>> > > > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > > > >>>>> at > > > >>> > > > >>> > > > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > > > >>>>> at > > > >>> > > > >>> > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > > >>>>> at > > > >>> > > > >>> > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > > >>>>> at java.lang.Thread.run(Thread.java:745) > > > >>>>> > > > >>>>> This is more annoying than a real problem. I am new to both > > > >>> accumulo > > > >>>>> and geomesa, but I am curious what the problem might be. > > > >>>>> > > > >>>>> Thanks! > > > >>>>> -Simon > > > >>>>> > > > >>>>> > > > >>>>> On Sat, Jun 6, 2015 at 8:01 PM, Josh Elser<[email protected] > > > >>> <mailto:[email protected]>> wrote: > > > >>>>>> > > > >>>>>> Great! Glad to hear it. Please let us know how it works out! > > > >>>>>> > > > >>>>>> > > > >>>>>> Xu (Simon) Chen wrote: > > > >>>>>>> > > > >>>>>>> Josh, > > > >>>>>>> > > > >>>>>>> You're right again.. Thanks! > > > >>>>>>> > > > >>>>>>> My ansible play actually pushed client.conf to all the server > > > >>> config > > > >>>>>>> directories, but didn't do anything for the clients, and that's > > > >>> my > > > >>>>>>> problem. Now kerberos is working great for me. > > > >>>>>>> > > > >>>>>>> Thanks again! > > > >>>>>>> -Simon > > > >>>>>>> > > > >>>>>>> On Sat, Jun 6, 2015 at 5:04 PM, Josh > > > >>> Elser<[email protected] <mailto:[email protected]>> > > > >>> > > > >>>>>>> wrote: > > > >>>>>>>> > > > >>>>>>>> Simon, > > > >>>>>>>> > > > >>>>>>>> Did you create a client configuration file (~/.accumulo/config > > > >>> or > > > >>>>>>>> $ACCUMULO_CONF_DIR/client.conf)? You need to configure > > > >>> Accumulo clients > > > >>>>>>>> to > > > >>>>>>>> actually use SASL when you're trying to use Kerberos > > > >>> authentication. > > > >>>>>>>> Your > > > >>>>>>>> server is expecting that, but I would venture a guess that > > > >>> your client > > > >>>>>>>> isn't. > > > >>>>>>>> > > > >>>>>>>> See > > > >>> > > > >>> > > > > http://accumulo.apache.org/1.7/accumulo_user_manual.html#_configuration_3 > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> Xu (Simon) Chen wrote: > > > >>>>>>>>> > > > >>>>>>>>> Josh, > > > >>>>>>>>> > > > >>>>>>>>> Thanks. It makes sense... > > > >>>>>>>>> > > > >>>>>>>>> I used a KerberosToken, but my program got stuck when > > > >>> running the > > > >>>>>>>>> following: > > > >>>>>>>>> new ZooKeeperInstance(instance, > zookeepers).getConnector(user, > > > >>>>>>>>> krbToken) > > > >>>>>>>>> > > > >>>>>>>>> It looks like my client is stuck here: > > > >>> > > > >>> > > > > > https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/impl/ConnectorImpl.java#L70 > > > >>>>>>>>> failing in the receive part of > > > >>> > > > >>> > > > > > org.apache.accumulo.core.client.impl.thrift.ClientService.Client.authenticate(). > > > >>>>>>>>> > > > >>>>>>>>> On my tservers, I see the following: > > > >>>>>>>>> > > > >>>>>>>>> 2015-06-06 18:58:19,616 [server.TThreadPoolServer] ERROR: > > > >>> Error > > > >>>>>>>>> occurred during processing of message. > > > >>>>>>>>> java.lang.RuntimeException: > > > >>>>>>>>> org.apache.thrift.transport.TTransportException: > > > >>>>>>>>> java.net.SocketTimeoutException: Read timed out > > > >>>>>>>>> at > > > >>> > > > >>> > > > > > org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219) > > > >>>>>>>>> at > > > >>> > > > >>> > > > > > org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:51) > > > >>>>>>>>> at > > > >>> > > > >>> > > > > > org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:48) > > > >>>>>>>>> at > > > >>> java.security.AccessController.doPrivileged(Native > > > >>>>>>>>> Method) > > > >>>>>>>>> at > > > >>> javax.security.auth.Subject.doAs(Subject.java:356) > > > >>>>>>>>> at > > > >>> > > > >>> > > > > > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1622) > > > >>>>>>>>> at > > > >>> > > > >>> > > > > > org.apache.accumulo.core.rpc.UGIAssumingTransportFactory.getTransport(UGIAssumingTransportFactory.java:48) > > > >>>>>>>>> at > > > >>> > > > >>> > > > > > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:208) > > > >>>>>>>>> at > > > >>> > > > >>> > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > > >>>>>>>>> at > > > >>> > > > >>> > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > > >>>>>>>>> at > > > >>> > > > >>> > > > > org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35) > > > >>>>>>>>> at java.lang.Thread.run(Thread.java:745) > > > >>>>>>>>> Caused by: org.apache.thrift.transport.TTransportException: > > > >>>>>>>>> java.net.SocketTimeoutException: Read timed out > > > >>>>>>>>> at > > > >>> > > > >>> > > > > > org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129) > > > >>>>>>>>> at > > > >>> > org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) > > > >>>>>>>>> at > > > >>> > > > >>> > > > > > org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:182) > > > >>>>>>>>> at > > > >>> > > > >>> > > > > > org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125) > > > >>>>>>>>> at > > > >>> > > > >>> > > > > org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253) > > > >>>>>>>>> at > > > >>> > > > >>> > > > > > org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41) > > > >>>>>>>>> at > > > >>> > > > >>> > > > > > org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216) > > > >>>>>>>>> ... 11 more > > > >>>>>>>>> Caused by: java.net.SocketTimeoutException: Read timed out > > > >>>>>>>>> at java.net.SocketInputStream.socketRead0(Native > > > >>> Method) > > > >>>>>>>>> at > > > >>>>>>>>> java.net.SocketInputStream.read(SocketInputStream.java:152) > > > >>>>>>>>> at > > > >>>>>>>>> java.net.SocketInputStream.read(SocketInputStream.java:122) > > > >>>>>>>>> at > > > >>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273) > > > >>>>>>>>> at > > > >>>>>>>>> > java.io.BufferedInputStream.read(BufferedInputStream.java:334) > > > >>>>>>>>> at > > > >>> > > > >>> > > > > > org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127) > > > >>>>>>>>> ... 17 more > > > >>>>>>>>> > > > >>>>>>>>> Any ideas why? > > > >>>>>>>>> > > > >>>>>>>>> Thanks. > > > >>>>>>>>> -Simon > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> On Sat, Jun 6, 2015 at 2:19 PM, Josh > > > >>> Elser<[email protected] <mailto:[email protected]>> > > > >>> > > > >>>>>>>>> wrote: > > > >>>>>>>>>> > > > >>>>>>>>>> Make sure you read the JavaDoc on DelegationToken: > > > >>>>>>>>>> > > > >>>>>>>>>> <snip> > > > >>>>>>>>>> Obtain a delegation token by calling {@link > > > >>> > > > >>> > > > > > SecurityOperations#getDelegationToken(org.apache.accumulo.core.client.admin.DelegationTokenConfig)} > > > >>>>>>>>>> </snip> > > > >>>>>>>>>> > > > >>>>>>>>>> You cannot create a usable DelegationToken as the client > > > >>> itself. > > > >>>>>>>>>> > > > >>>>>>>>>> Anyways, DelegationTokens are only relevant in cases where > > > >>> the client > > > >>>>>>>>>> Kerberos credentials are unavailable. The most common case > > > >>> is running > > > >>>>>>>>>> MapReduce jobs. If you are just interacting with Accumulo > > > >>> through the > > > >>>>>>>>>> Java > > > >>>>>>>>>> API, the KerberosToken is all you need to use. > > > >>>>>>>>>> > > > >>>>>>>>>> The user-manual likely just needs to be updated. I believe > > > >>> the > > > >>>>>>>>>> DelegationTokenConfig was added after I wrote the initial > > > >>>>>>>>>> documentation. > > > >>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>>> Xu (Simon) Chen wrote: > > > >>>>>>>>>>> > > > >>>>>>>>>>> Hi folks, > > > >>>>>>>>>>> > > > >>>>>>>>>>> The latest kerberos doc seems to indicate that > > > >>> getDelegationToken > > > >>>>>>>>>>> can > > > >>>>>>>>>>> be > > > >>>>>>>>>>> called without any parameters: > > > >>> > > > >>> > > > > > https://github.com/apache/accumulo/blob/1.7/docs/src/main/asciidoc/chapters/kerberos.txt#L410 > > > >>>>>>>>>>> > > > >>>>>>>>>>> Yet the source code indicates a DelegationTokenConfig > > > >>> object must be > > > >>>>>>>>>>> passed in: > > > >>> > > > >>> > > > > > https://github.com/apache/accumulo/blob/1.7/core/src/main/java/org/apache/accumulo/core/client/admin/SecurityOperations.java#L359 > > > >>>>>>>>>>> > > > >>>>>>>>>>> Any ideas on how I should construct the > > > >>> DelegationTokenConfig > > > >>>>>>>>>>> object? > > > >>>>>>>>>>> > > > >>>>>>>>>>> For context, I've been trying to get geomesa to work on my > > > >>> accumulo > > > >>>>>>>>>>> 1.7 > > > >>>>>>>>>>> with kerberos turned on. Right now, the code is somewhat > > > >>> tied to > > > >>>>>>>>>>> password auth: > > > >>> > > > >>> > > > > > https://github.com/locationtech/geomesa/blob/rc7_a1.7_h2.5/geomesa-core/src/main/scala/org/locationtech/geomesa/core/data/AccumuloDataStoreFactory.scala#L177 > > > >>>>>>>>>>> My thought is that I should get a KerberosToken first, and > > > >>> then try > > > >>>>>>>>>>> generate a DelegationToken, which is passed back for later > > > >>>>>>>>>>> interactions > > > >>>>>>>>>>> between geomesa and accumulo. > > > >>>>>>>>>>> > > > >>>>>>>>>>> Thanks. > > > >>>>>>>>>>> -Simon > > > > > > > > > > > > -- > > > > Sean > > > >
