For me, the startup doesn't take anywhere near as long as shutdown does. On Fri, Nov 11, 2016 at 9:37 PM, Ali Akhtar <ali.rac...@gmail.com> wrote:
> Unless I'm missing anything, there's no reason why these throwaway > processes should be shutdown gracefully. Just kill them as soon as the test > finishes. > > On Fri, Nov 11, 2016 at 9:26 PM, Ali Akhtar <ali.rac...@gmail.com> wrote: > >> Hey Eno, >> >> Thanks for the quick reply. >> >> In the meantime, is it possible to just send a sigterm / kill -9 which >> just kills the zookeeper + kafka? I can figure out how to do it if you can >> point out which class / method creates the processes / threads. >> >> Thanks. >> >> On Fri, Nov 11, 2016 at 9:24 PM, Eno Thereska <eno.there...@gmail.com> >> wrote: >> >>> Hi Ali, >>> >>> You're right, shutting down the broker and ZK is expensive. We kept the >>> number of integration tests relatively small (and pushed some more tests as >>> system tests, while doing as many as possible as unit tests). It's not just >>> the shutdown that's expensive, it's also the starting up unfortunately. >>> It's on our todo list to do something about this, but we haven't gotten >>> there yet. If someone from the community wants to have a look and help out, >>> that'd be great (with a JIRA and PR). >>> >>> About the second problem with ZK logs, this is being worked on as part >>> of removing the ZK dependency from streams and should be merged shortly: >>> https://github.com/apache/kafka/pull/1884 <https://github.com/apache/kaf >>> ka/pull/1884>. The msg you see does not affect correctness, it's just >>> annoying and it will go away. >>> >>> Thanks, >>> Eno >>> >>> >>> > On 11 Nov 2016, at 14:28, Ali Akhtar <ali.rac...@gmail.com> wrote: >>> > >>> > I have some unit tests in which I create an embedded single broker >>> kafka >>> > cluster, using : >>> > >>> > EmbeddedSingleNodeKafkaCluster.java from >>> > https://github.com/confluentinc/examples/blob/master/kafka-s >>> treams/src/test/java/io/confluent/examples/streams/kafka/ >>> EmbeddedSingleNodeKafkaCluster.java >>> > >>> > That class also creates an embedded zookeeper cluster / instance. >>> > >>> > The problem is, while the tests run pretty fast and pass, they then >>> stay >>> > stuck in the 'teardown / clean up' stage for a really long time, often >>> upto >>> > 10-20 >>> > seconds per test. >>> > >>> > As I have a lot of test classes, each class creating its own embedded >>> kafka >>> > cluster, this time can really add up during compiles. >>> > >>> > Is it possible to get these test classes to not do any clean up / >>> safety >>> > stuff, because the instances are just throwaway. Just have them kill >>> -9 the >>> > kafka / zookeeper and exit? >>> > >>> > It doesn't make any sense that tests pass within seconds, but can't >>> move on >>> > to the next test class because its cleaning up. >>> > >>> > I also have an embedded cassandra instance in these tests, but I don't >>> > think that one is the problem, as i see a lot of zookeeper logs such as >>> > these after the test runs: >>> > >>> > 133764 [main-SendThread(127.0.0.1:38846)] WARN >>> > org.apache.zookeeper.ClientCnxn - Session 0x15853c3497f0001 for >>> server >>> > null, unexpected error, closing socket connection and attempting >>> reconnect >>> > java.net.ConnectException: Connection refused >>> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >>> > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl >>> .java:717) >>> > at >>> > org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientC >>> nxnSocketNIO.java:361) >>> > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.ja >>> va:1081) >>> > >>> > >>> > Could it be that zookeeper doesn't exit and keeps retrying to connect? >>> >>> >> >
