Deleting the metastore_db directory gets me past the hive sink error rm -rf /Users/jrufus/dev/temp/apache-flume-1.6.0-src/flume-ng-sinks/flume-hive-sink/metastore_db
metastore_db/temp is present in the source structure, which should not be present i presume. I will work with Hari on uploading a new jar that doesnot have the .iml files and the metastore_db/temp directory. Thanks, Rufus On Sun, May 3, 2015 at 12:17 AM, Ashish <[email protected]> wrote: > Not able to debug the Zk issue. Ran the build from within the Kafka > Sink and it worked, so must be something else. > > So far not able to get a clean build. Deleted m2 repository to avoid > issues. Now running into Hive Sink Test issue. . Any suggestions on > Hive sink test error? > > I was expecting a few votes by now. Folks, please do vote on the release. > The source also contains Intellij iml files which are not needed IMHO > > Error trace below > > > Running org.apache.flume.sink.hive.TestHiveSink > > Tests run: 5, Failures: 0, Errors: 5, Skipped: 0, Time elapsed: 2.277 > sec <<< FAILURE! > > > testSingleWriterSimplePartitionedTable(org.apache.flume.sink.hive.TestHiveSink) > Time elapsed: 4 sec <<< ERROR! > > java.sql.SQLException: Failed to create database 'metastore_db', see > the next exception for details. > > at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown > Source) > > at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Unknown Source) > > at org.apache.derby.impl.jdbc.Util.seeNextException(Unknown Source) > > at org.apache.derby.impl.jdbc.EmbedConnection.createDatabase(Unknown > Source) > > at org.apache.derby.impl.jdbc.EmbedConnection.<init>(Unknown Source) > > at org.apache.derby.impl.jdbc.EmbedConnection30.<init>(Unknown Source) > > at org.apache.derby.impl.jdbc.EmbedConnection40.<init>(Unknown Source) > > at org.apache.derby.jdbc.Driver40.getNewEmbedConnection(Unknown Source) > > at org.apache.derby.jdbc.InternalDriver.connect(Unknown Source) > > at org.apache.derby.jdbc.EmbeddedDriver.connect(Unknown Source) > > at > org.apache.hadoop.hive.metastore.txn.TxnDbUtil.getConnection(TxnDbUtil.java:216) > > at > org.apache.hadoop.hive.metastore.txn.TxnDbUtil.cleanDb(TxnDbUtil.java:129) > > at org.apache.flume.sink.hive.TestHiveSink.<init>(TestHiveSink.java:105) > > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > > at > org.junit.runners.BlockJUnit4ClassRunner.createTest(BlockJUnit4ClassRunner.java:187) > > at > org.junit.runners.BlockJUnit4ClassRunner$1.runReflectiveCall(BlockJUnit4ClassRunner.java:236) > > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) > > at > org.junit.runners.BlockJUnit4ClassRunner.methodBlock(BlockJUnit4ClassRunner.java:233) > > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68) > > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47) > > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231) > > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60) > > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229) > > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50) > > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222) > > at org.junit.runners.ParentRunner.run(ParentRunner.java:300) > > at > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252) > > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141) > > at > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > > at java.lang.reflect.Method.invoke(Method.java:597) > > at > org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) > > at > org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) > > at > org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) > > at > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115) > > at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75) > > Caused by: java.sql.SQLException: Failed to create database > 'metastore_db', see the next exception for details. > > at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown > Source) > > at > org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown > Source) > > ... 41 more > > Caused by: java.sql.SQLException: The database directory > > '/Users/ashishpaliwal/work/trash/apache-flume-1.6.0-src/flume-ng-sinks/flume-hive-sink/metastore_db' > exists. However, it does not contain the expected 'service.properties' > file. Perhaps Derby was brought down in the middle of creating this > database. You may want to delete this directory and try creating the > database again. > > at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown > Source) > > at > org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown > Source) > > at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown > Source) > > at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source) > > at > org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown > Source) > > at > org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown > Source) > > at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown > Source) > > ... 38 more > > Caused by: ERROR XBM0A: The database directory > > '/Users/ashishpaliwal/work/trash/apache-flume-1.6.0-src/flume-ng-sinks/flume-hive-sink/metastore_db' > exists. However, it does not contain the expected 'service.properties' > file. Perhaps Derby was brought down in the middle of creating this > database. You may want to delete this directory and try creating the > database again. > > at org.apache.derby.iapi.error.StandardException.newException(Unknown > Source) > > at > org.apache.derby.impl.services.monitor.StorageFactoryService.vetService(Unknown > Source) > > at > org.apache.derby.impl.services.monitor.StorageFactoryService.access$600(Unknown > Source) > > at > org.apache.derby.impl.services.monitor.StorageFactoryService$9.run(Unknown > Source) > > at java.security.AccessController.doPrivileged(Native Method) > > at > org.apache.derby.impl.services.monitor.StorageFactoryService.createServiceRoot(Unknown > Source) > > at org.apache.derby.impl.services.monitor.BaseMonitor.bootService(Unknown > Source) > > at > org.apache.derby.impl.services.monitor.BaseMonitor.createPersistentService(Unknown > Source) > > at > org.apache.derby.iapi.services.monitor.Monitor.createPersistentService(Unknown > Source) > > ... 38 more > > On Sat, May 2, 2015 at 8:26 AM, Ashish <[email protected]> wrote: > > No luck with Java 7 either. > > > > With Java 6 and running into this error (running on Macbook Air with 4G > RAM) > > > > org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to > > zookeeper server within timeout: 1000 > > > > at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:880) > > > > at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:98) > > > > at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:84) > > > > at > kafka.consumer.ZookeeperConsumerConnector.connectZk(ZookeeperConsumerConnector.scala:156) > > > > at > kafka.consumer.ZookeeperConsumerConnector.<init>(ZookeeperConsumerConnector.scala:114) > > > > at > kafka.javaapi.consumer.ZookeeperConsumerConnector.<init>(ZookeeperConsumerConnector.scala:65) > > > > at > kafka.javaapi.consumer.ZookeeperConsumerConnector.<init>(ZookeeperConsumerConnector.scala:67) > > > > at > kafka.consumer.Consumer$.createJavaConsumerConnector(ConsumerConnector.scala:100) > > > > at > kafka.consumer.Consumer.createJavaConsumerConnector(ConsumerConnector.scala) > > > > at > org.apache.flume.sink.kafka.util.KafkaConsumer.<init>(KafkaConsumer.java:51) > > > > at > org.apache.flume.sink.kafka.util.TestUtil.getKafkaConsumer(TestUtil.java:120) > > > > at org.apache.flume.sink.kafka.util.TestUtil.prepare(TestUtil.java:145) > > > > at org.apache.flume.sink.kafka.TestKafkaSink.setup(TestKafkaSink.java:51) > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > > > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > > > > at java.lang.reflect.Method.invoke(Method.java:597) > > > > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45) > > > > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) > > > > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42) > > > > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27) > > > > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30) > > > > at org.junit.runners.ParentRunner.run(ParentRunner.java:300) > > > > at > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252) > > > > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141) > > > > at > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112) > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > > > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > > > > at java.lang.reflect.Method.invoke(Method.java:597) > > > > at > org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) > > > > at > org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) > > > > at > org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) > > > > at > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115) > > > > at > org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75) > > > > > > > > Results : > > > > > > Tests in error: > > > > org.apache.flume.sink.kafka.TestKafkaSink: Unable to connect to > > zookeeper server within timeout: 1000 > > > > On Thu, Apr 30, 2015 at 10:11 PM, Hari Shreedharan > > <[email protected]> wrote: > >> We don't really support or test against Java 8. I think there are > several > >> jiras open to fix Flume to work properly against Java 8, though I don't > >> think all of them have any work on it yet. Can you test against Java 7? > >> > >> On Thu, Apr 30, 2015 at 5:11 AM, Ashish <[email protected]> > wrote: > >> > >>> Have trouble with the build. I don't think it's specific to this > >>> release. I have seen this in past. > >>> > >>> Building with jdk1.8.0_25 > >>> Encountering the following error (tried 4-5 times), so far haven't > >>> been able to get a working build > >>> > >>> log4j:WARN No appenders could be found for logger > >>> (org.apache.flume.channel.file.FileChannel). > >>> log4j:WARN Please initialize the log4j system properly. > >>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig > >>> for more info. > >>> Attempting to shutdown background worker. > >>> Attempting to shutdown background worker. > >>> Attempting to shutdown background worker. > >>> Attempting to shutdown background worker. > >>> Attempting to shutdown background worker. > >>> Attempting to shutdown background worker. > >>> Attempting to shutdown background worker. > >>> Attempting to shutdown background worker. > >>> Attempting to shutdown background worker. > >>> Attempting to shutdown background worker. > >>> Attempting to shutdown background worker. > >>> Attempting to shutdown background worker. > >>> Attempting to shutdown background worker. > >>> Attempting to shutdown background worker. > >>> Attempting to shutdown background worker. > >>> Attempting to shutdown background worker. > >>> src : [ 1106 ms ]. > >>> sink : [ 2232 ms ]. > >>> main : [ 2234 ms ]. > >>> Max Queue size 428169 > >>> Attempting to shutdown background worker. > >>> src : [ 1 ms ]. > >>> sink : [ 6 ms ]. > >>> src : [ 168 ms ]. > >>> sink : [ 10 ms ]. > >>> sink1 : [ 24 ms ]. { items = 5000, attempts = 6 } > >>> src1 : [ 43 ms ]. > >>> src2 : [ 43 ms ]. > >>> sink3 : [ 105 ms ]. { items = 2500, attempts = 3 } > >>> sink2 : [ 105 ms ]. { items = 2500, attempts = 5 } > >>> Attempting to shutdown background worker. > >>> Exception in thread "src5" Exception in thread "src3" > >>> org.apache.flume.ChannelFullException: The channel has reached it's > >>> capacity. This might be the result of a sink on the channel having too > >>> low of batch size, a downstream system running slower than normal, or > >>> that the channel capacity is just too low. > >>> [channel=spillChannel-907f6e39-ffe1-4953-abc1-265fa339934a] > >>> at > >>> > org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:465) > >>> at > >>> > org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93) > >>> at > >>> > org.apache.flume.channel.SpillableMemoryChannel$SpillableMemoryTransaction.commitPutsToOverflow(SpillableMemoryChannel.java:490) > >>> at > >>> > org.apache.flume.channel.SpillableMemoryChannel$SpillableMemoryTransaction.putCommit(SpillableMemoryChannel.java:480) > >>> at > >>> > org.apache.flume.channel.SpillableMemoryChannel$SpillableMemoryTransaction.doCommit(SpillableMemoryChannel.java:401) > >>> at > >>> > org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151) > >>> at > >>> > org.apache.flume.channel.TestSpillableMemoryChannel.transactionalPutN(TestSpillableMemoryChannel.java:157) > >>> at > >>> > org.apache.flume.channel.TestSpillableMemoryChannel.access$000(TestSpillableMemoryChannel.java:43) > >>> at > >>> > org.apache.flume.channel.TestSpillableMemoryChannel$1.run(TestSpillableMemoryChannel.java:230) > >>> Exception in thread "src2" org.apache.flume.ChannelFullException: The > >>> channel has reached it's capacity. This might be the result of a sink > >>> on the channel having too low of batch size, a downstream system > >>> running slower than normal, or that the channel capacity is just too > >>> low. [channel=spillChannel-907f6e39-ffe1-4953-abc1-265fa339934a] > >>> at > >>> > org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:465) > >>> at > >>> > org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93) > >>> at > >>> > org.apache.flume.channel.SpillableMemoryChannel$SpillableMemoryTransaction.commitPutsToOverflow(SpillableMemoryChannel.java:490) > >>> at > >>> > org.apache.flume.channel.SpillableMemoryChannel$SpillableMemoryTransaction.putCommit(SpillableMemoryChannel.java:480) > >>> at > >>> > org.apache.flume.channel.SpillableMemoryChannel$SpillableMemoryTransaction.doCommit(SpillableMemoryChannel.java:401) > >>> at > >>> > org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151) > >>> at > >>> > org.apache.flume.channel.TestSpillableMemoryChannel.transactionalPutN(TestSpillableMemoryChannel.java:157) > >>> at > >>> > org.apache.flume.channel.TestSpillableMemoryChannel.access$000(TestSpillableMemoryChannel.java:43) > >>> at > >>> > org.apache.flume.channel.TestSpillableMemoryChannel$1.run(TestSpillableMemoryChannel.java:230) > >>> org.apache.flume.ChannelFullException: The channel has reached it's > >>> capacity. This might be the result of a sink on the channel having too > >>> low of batch size, a downstream system running slower than normal, or > >>> that the channel capacity is just too low. > >>> [channel=spillChannel-907f6e39-ffe1-4953-abc1-265fa339934a] > >>> at > >>> > org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:465) > >>> at > >>> > org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93) > >>> at > >>> > org.apache.flume.channel.SpillableMemoryChannel$SpillableMemoryTransaction.commitPutsToOverflow(SpillableMemoryChannel.java:490) > >>> at > >>> > org.apache.flume.channel.SpillableMemoryChannel$SpillableMemoryTransaction.putCommit(SpillableMemoryChannel.java:480) > >>> at > >>> > org.apache.flume.channel.SpillableMemoryChannel$SpillableMemoryTransaction.doCommit(SpillableMemoryChannel.java:401) > >>> at > >>> > org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151) > >>> at > >>> > org.apache.flume.channel.TestSpillableMemoryChannel.transactionalPutN(TestSpillableMemoryChannel.java:157) > >>> at > >>> > org.apache.flume.channel.TestSpillableMemoryChannel.access$000(TestSpillableMemoryChannel.java:43) > >>> at > >>> > org.apache.flume.channel.TestSpillableMemoryChannel$1.run(TestSpillableMemoryChannel.java:230) > >>> > >>> On Thu, Apr 30, 2015 at 1:56 AM, Johny Rufus <[email protected]> > wrote: > >>> > Hi All, > >>> > > >>> > This is the ninth release for Apache Flume as a top-level project, > >>> > version 1.6.0. We are voting on release candidate RC0. > >>> > > >>> > It fixes the following issues: > >>> > > >>> > > >>> > https://git-wip-us.apache.org/repos/asf?p=flume.git;a=blob;f=CHANGELOG;h=774aced731de1e49043c179a722e55feb69f1b29;hb=493976e20dfe14b0b611c92f3e160d4336d10af2 > >>> > > >>> > *** Please cast your vote within the next 72 hours *** > >>> > > >>> > The tarball (*.tar.gz), signature (*.asc), and checksums (*.md5, > *.sha1) > >>> > for the source and binary artifacts can be found here: > >>> > > >>> > http://people.apache.org/~hshreedharan/apache-flume-1.6.0-rc0/ > >>> > > >>> > Maven staging repo: > >>> > > >>> > * > https://repository.apache.org/content/repositories/orgapacheflume-1013/ > >>> > < > https://repository.apache.org/content/repositories/orgapacheflume-1013/ > >>> >* > >>> > > >>> > The tag to be voted on: > >>> > > >>> > > >>> > https://git-wip-us.apache.org/repos/asf?p=flume.git;a=commit;h=30af6e90603d476aa058d2736c2ae154acff4af9 > >>> > > >>> > > >>> > Flume's KEYS file containing PGP keys we use to sign the release: > >>> > http://www.apache.org/dist/flume/KEYS > >>> > > >>> > > >>> > Thanks, > >>> > Rufus > >>> > >>> > >>> > >>> -- > >>> thanks > >>> ashish > >>> > >>> Blog: http://www.ashishpaliwal.com/blog > >>> My Photo Galleries: http://www.pbase.com/ashishpaliwal > >>> > > > > > > > > -- > > thanks > > ashish > > > > Blog: http://www.ashishpaliwal.com/blog > > My Photo Galleries: http://www.pbase.com/ashishpaliwal > > > > -- > thanks > ashish > > Blog: http://www.ashishpaliwal.com/blog > My Photo Galleries: http://www.pbase.com/ashishpaliwal >
