+1 to deprecating in 1.8.0. Additionally, I would be +1 to accelerating its removal in 2.0. I think it has become a burden that is inhibiting progress in other areas. I would like to see it dispassionately ripped out in favor of properly mocked unit testing (with EasyMock or others) and MAC-based integration testing.
-- Christopher L Tubbs II http://gravatar.com/ctubbsii On Mon, Jun 8, 2015 at 11:42 PM, Ryan Leary <[email protected]> wrote: > Just an anecdote from someone who has been bitten by mock more than a couple > times. I would try to deprecate it in 1.8 and remove in 2.0 if at all > possible. People really shouldn't write tests against it. > >> On Jun 8, 2015, at 10:10 PM, Sean Busbey <[email protected]> wrote: >> >> Josh's comment below made me realize we still haven't formally deprecated >> MockAccumulo. >> >> What do folks think about doing it soon-ish with an aim of removing it in >> Accumulo 3.0? (that's version three, so that it can remain deprecated for >> all of version 2). >> >> >> -Sean >> >>> On Sun, Jun 7, 2015 at 12:37 PM, Josh Elser <[email protected]> wrote: >>> >>> MiniAccumulo, yes. MockAccumulo, no. In general, we've near completely >>> moved away from MockAccumulo. I wouldn't be surprised if it gets deprecated >>> and removed soon. >>> >>> >>> https://github.com/apache/accumulo/blob/1.7/test/src/test/java/org/apache/accumulo/test/functional/KerberosIT.java >>> >>> Apache Directory provides a MiniKdc that can be used easily w/ >>> MiniAccumulo. Many of the integration tests have already been altered to >>> support running w/ or w/o kerberos. >>> >>> James Hughes wrote: >>> >>>> Hi all, >>>> >>>> For GeoMesa, stats writing is quite secondary and optional, so we can >>>> sort that out as a follow-on to seeing GeoMesa work with Accumulo 1.7. >>>> >>>> I haven't had a chance to read in details yet, so forgive me if this is >>>> covered in the docs. Does either Mock or MiniAccumulo provide enough >>>> hooks to test out Kerberos integration effectively? I suppose I'm >>>> really asking what kind of testing environment a project like GeoMesa >>>> would need to use to test out Accumulo 1.7. >>>> >>>> Even though MockAccumulo has a number of limitations, it is rather >>>> effective in unit tests which can be part of a quick build. >>>> >>>> Thanks, >>>> >>>> Jim >>>> >>>> On Sat, Jun 6, 2015 at 11:14 PM, Xu (Simon) Chen <[email protected] >>>> <mailto:[email protected]>> wrote: >>>> >>>> Nope, I am running the example as what the readme file suggested: >>>> >>>> java -cp ./target/geomesa-quickstart-1.0-SNAPSHOT.jar >>>> org.geomesa.QuickStart -instanceId somecloud -zookeepers >>>> "zoo1:2181,zoo2:2181,zoo3:2181" -user someuser -password somepwd >>>> -tableName sometable >>>> >>>> I'll raise this question with the geomesa folks, but you're right that >>>> I can ignore it for now... >>>> >>>> Thanks! >>>> -Simon >>>> >>>> >>>> On Sat, Jun 6, 2015 at 11:00 PM, Josh Elser <[email protected] >>>> <mailto:[email protected]>> wrote: >>>>> Are you running it via `mvn exec:java` by chance or netbeans? >>>> >>>> http://mail-archives.apache.org/mod_mbox/accumulo-user/201411.mbox/%[email protected]%3E >>>>> >>>>> If that's just a background thread writing in Stats, it might >>>> just be a >>>>> factor of how you're invoking the program and you can ignore it. >>>> I don't >>>>> know enough about the inner-workings of GeoMesa to say one way or >>>> the other. >>>>> >>>>> >>>>> Xu (Simon) Chen wrote: >>>>>> >>>>>> Josh, >>>>>> >>>>>> Everything works well, except for one thing :-) >>>>>> >>>>>> I am running geomesa-quickstart program that ingest some data >>>> and then >>>>>> perform a simple query: >>>>>> https://github.com/geomesa/geomesa-quickstart >>>>>> >>>>>> For some reason, the following error is emitted consistently at >>>> the >>>>>> end of the execution, after outputting the correct result: >>>>>> 15/06/07 00:29:22 INFO zookeeper.ZooCache: Zookeeper error, will >>>> retry >>>>>> java.lang.InterruptedException >>>>>> at java.lang.Object.wait(Native Method) >>>>>> at java.lang.Object.wait(Object.java:503) >>>>>> at >>>> org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1342) >>>>>> at >>>> org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1036) >>>>>> at >>>> org.apache.accumulo.fate.zookeeper.ZooCache$2.run(ZooCache.java:264) >>>>>> at >>>> org.apache.accumulo.fate.zookeeper.ZooCache.retry(ZooCache.java:162) >>>>>> at >>>>>> org.apache.accumulo.fate.zookeeper.ZooCache.get(ZooCache.java:289) >>>>>> at >>>>>> org.apache.accumulo.fate.zookeeper.ZooCache.get(ZooCache.java:238) >>>>>> at >>>> >>>> org.apache.accumulo.core.client.impl.Tables.getTableState(Tables.java:180) >>>>>> at >>>> >>>> org.apache.accumulo.core.client.impl.ConnectorImpl.getTableId(ConnectorImpl.java:82) >>>>>> at >>>> >>>> org.apache.accumulo.core.client.impl.ConnectorImpl.createBatchWriter(ConnectorImpl.java:128) >>>>>> at >>>> >>>> org.locationtech.geomesa.core.stats.StatWriter$$anonfun$write$2.apply(StatWriter.scala:174) >>>>>> at >>>> >>>> org.locationtech.geomesa.core.stats.StatWriter$$anonfun$write$2.apply(StatWriter.scala:156) >>>>>> at >>>> scala.collection.immutable.Map$Map1.foreach(Map.scala:109) >>>>>> at >>>> >>>> org.locationtech.geomesa.core.stats.StatWriter$.write(StatWriter.scala:156) >>>>>> at >>>> >>>> org.locationtech.geomesa.core.stats.StatWriter$.drainQueue(StatWriter.scala:148) >>>>>> at >>>> >>>> org.locationtech.geomesa.core.stats.StatWriter$.run(StatWriter.scala:116) >>>>>> at >>>> >>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>>>>> at >>>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) >>>>>> at >>>> >>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) >>>>>> at >>>> >>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) >>>>>> at >>>> >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>>> at >>>> >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>> >>>>>> This is more annoying than a real problem. I am new to both >>>> accumulo >>>>>> and geomesa, but I am curious what the problem might be. >>>>>> >>>>>> Thanks! >>>>>> -Simon >>>>>> >>>>>> >>>>>> On Sat, Jun 6, 2015 at 8:01 PM, Josh Elser<[email protected] >>>> <mailto:[email protected]>> wrote: >>>>>>> >>>>>>> Great! Glad to hear it. Please let us know how it works out! >>>>>>> >>>>>>> >>>>>>> Xu (Simon) Chen wrote: >>>>>>>> >>>>>>>> Josh, >>>>>>>> >>>>>>>> You're right again.. Thanks! >>>>>>>> >>>>>>>> My ansible play actually pushed client.conf to all the server >>>> config >>>>>>>> directories, but didn't do anything for the clients, and that's >>>> my >>>>>>>> problem. Now kerberos is working great for me. >>>>>>>> >>>>>>>> Thanks again! >>>>>>>> -Simon >>>>>>>> >>>>>>>> On Sat, Jun 6, 2015 at 5:04 PM, Josh >>>> Elser<[email protected] <mailto:[email protected]>> >>>> >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Simon, >>>>>>>>> >>>>>>>>> Did you create a client configuration file (~/.accumulo/config >>>> or >>>>>>>>> $ACCUMULO_CONF_DIR/client.conf)? You need to configure >>>> Accumulo clients >>>>>>>>> to >>>>>>>>> actually use SASL when you're trying to use Kerberos >>>> authentication. >>>>>>>>> Your >>>>>>>>> server is expecting that, but I would venture a guess that >>>> your client >>>>>>>>> isn't. >>>>>>>>> >>>>>>>>> See >>>> >>>> http://accumulo.apache.org/1.7/accumulo_user_manual.html#_configuration_3 >>>>>>>>> >>>>>>>>> >>>>>>>>> Xu (Simon) Chen wrote: >>>>>>>>>> >>>>>>>>>> Josh, >>>>>>>>>> >>>>>>>>>> Thanks. It makes sense... >>>>>>>>>> >>>>>>>>>> I used a KerberosToken, but my program got stuck when >>>> running the >>>>>>>>>> following: >>>>>>>>>> new ZooKeeperInstance(instance, zookeepers).getConnector(user, >>>>>>>>>> krbToken) >>>>>>>>>> >>>>>>>>>> It looks like my client is stuck here: >>>> >>>> https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/impl/ConnectorImpl.java#L70 >>>>>>>>>> failing in the receive part of >>>> >>>> org.apache.accumulo.core.client.impl.thrift.ClientService.Client.authenticate(). >>>>>>>>>> >>>>>>>>>> On my tservers, I see the following: >>>>>>>>>> >>>>>>>>>> 2015-06-06 18:58:19,616 [server.TThreadPoolServer] ERROR: >>>> Error >>>>>>>>>> occurred during processing of message. >>>>>>>>>> java.lang.RuntimeException: >>>>>>>>>> org.apache.thrift.transport.TTransportException: >>>>>>>>>> java.net.SocketTimeoutException: Read timed out >>>>>>>>>> at >>>> >>>> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219) >>>>>>>>>> at >>>> >>>> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:51) >>>>>>>>>> at >>>> >>>> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:48) >>>>>>>>>> at >>>> java.security.AccessController.doPrivileged(Native >>>>>>>>>> Method) >>>>>>>>>> at >>>> javax.security.auth.Subject.doAs(Subject.java:356) >>>>>>>>>> at >>>> >>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1622) >>>>>>>>>> at >>>> >>>> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory.getTransport(UGIAssumingTransportFactory.java:48) >>>>>>>>>> at >>>> >>>> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:208) >>>>>>>>>> at >>>> >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>>>>>>> at >>>> >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>>>>>>> at >>>> >>>> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35) >>>>>>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>>>>>> Caused by: org.apache.thrift.transport.TTransportException: >>>>>>>>>> java.net.SocketTimeoutException: Read timed out >>>>>>>>>> at >>>> >>>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129) >>>>>>>>>> at >>>> org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) >>>>>>>>>> at >>>> >>>> org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:182) >>>>>>>>>> at >>>> >>>> org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125) >>>>>>>>>> at >>>> >>>> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253) >>>>>>>>>> at >>>> >>>> org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41) >>>>>>>>>> at >>>> >>>> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216) >>>>>>>>>> ... 11 more >>>>>>>>>> Caused by: java.net.SocketTimeoutException: Read timed out >>>>>>>>>> at java.net.SocketInputStream.socketRead0(Native >>>> Method) >>>>>>>>>> at >>>>>>>>>> java.net.SocketInputStream.read(SocketInputStream.java:152) >>>>>>>>>> at >>>>>>>>>> java.net.SocketInputStream.read(SocketInputStream.java:122) >>>>>>>>>> at >>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273) >>>>>>>>>> at >>>>>>>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334) >>>>>>>>>> at >>>> >>>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127) >>>>>>>>>> ... 17 more >>>>>>>>>> >>>>>>>>>> Any ideas why? >>>>>>>>>> >>>>>>>>>> Thanks. >>>>>>>>>> -Simon >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Sat, Jun 6, 2015 at 2:19 PM, Josh >>>> Elser<[email protected] <mailto:[email protected]>> >>>> >>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> Make sure you read the JavaDoc on DelegationToken: >>>>>>>>>>> >>>>>>>>>>> <snip> >>>>>>>>>>> Obtain a delegation token by calling {@link >>>> >>>> SecurityOperations#getDelegationToken(org.apache.accumulo.core.client.admin.DelegationTokenConfig)} >>>>>>>>>>> </snip> >>>>>>>>>>> >>>>>>>>>>> You cannot create a usable DelegationToken as the client >>>> itself. >>>>>>>>>>> >>>>>>>>>>> Anyways, DelegationTokens are only relevant in cases where >>>> the client >>>>>>>>>>> Kerberos credentials are unavailable. The most common case >>>> is running >>>>>>>>>>> MapReduce jobs. If you are just interacting with Accumulo >>>> through the >>>>>>>>>>> Java >>>>>>>>>>> API, the KerberosToken is all you need to use. >>>>>>>>>>> >>>>>>>>>>> The user-manual likely just needs to be updated. I believe >>>> the >>>>>>>>>>> DelegationTokenConfig was added after I wrote the initial >>>>>>>>>>> documentation. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Xu (Simon) Chen wrote: >>>>>>>>>>>> >>>>>>>>>>>> Hi folks, >>>>>>>>>>>> >>>>>>>>>>>> The latest kerberos doc seems to indicate that >>>> getDelegationToken >>>>>>>>>>>> can >>>>>>>>>>>> be >>>>>>>>>>>> called without any parameters: >>>> >>>> https://github.com/apache/accumulo/blob/1.7/docs/src/main/asciidoc/chapters/kerberos.txt#L410 >>>>>>>>>>>> >>>>>>>>>>>> Yet the source code indicates a DelegationTokenConfig >>>> object must be >>>>>>>>>>>> passed in: >>>> >>>> https://github.com/apache/accumulo/blob/1.7/core/src/main/java/org/apache/accumulo/core/client/admin/SecurityOperations.java#L359 >>>>>>>>>>>> >>>>>>>>>>>> Any ideas on how I should construct the >>>> DelegationTokenConfig >>>>>>>>>>>> object? >>>>>>>>>>>> >>>>>>>>>>>> For context, I've been trying to get geomesa to work on my >>>> accumulo >>>>>>>>>>>> 1.7 >>>>>>>>>>>> with kerberos turned on. Right now, the code is somewhat >>>> tied to >>>>>>>>>>>> password auth: >>>> >>>> https://github.com/locationtech/geomesa/blob/rc7_a1.7_h2.5/geomesa-core/src/main/scala/org/locationtech/geomesa/core/data/AccumuloDataStoreFactory.scala#L177 >>>>>>>>>>>> My thought is that I should get a KerberosToken first, and >>>> then try >>>>>>>>>>>> generate a DelegationToken, which is passed back for later >>>>>>>>>>>> interactions >>>>>>>>>>>> between geomesa and accumulo. >>>>>>>>>>>> >>>>>>>>>>>> Thanks. >>>>>>>>>>>> -Simon >> >> >> -- >> Sean
