[
https://issues.apache.org/jira/browse/QUARKS-19?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15187796#comment-15187796
]
May Wone commented on QUARKS-19:
--------------------------------
KAFKA
How to 1) Install Kafka and 2) Run Quarks Kafka manual tests on a Linux cluster
machine
Go to http://kafka.apache.org/documentation.html\#quickstart
Here’s an outline of the steps from the quick start guide:
1. Download Kafka kafka\_2.11-0.9.0.0.tgz file to directory say: ~/kafka
*tar –xvf kafka\_2.11-0.9.0.0.tgz**
*cd kafka\_2.11-0.9.0.0*
2. My notes: On cluster machine, these additional modifications were needed,
because my userid has permission issues accessing /tmp and the JVM does not
support all the options specified in the kafka scripts. (Your machine may not
need these changes.)
2a. kafka-run-class.sh – remove any unsupported JVM options (see
Troubleshooting section \#1 for an example). If zookeeper (step 3a) starts ok,
then this is not applicable.
2b. zookeeper.properties – replace *dataDir=/tmp/zookeeper* with
*dataDir=~/tmp/zookeeper* (see Trouble-shooting section \#2).
2c. Server.properties - replace *log.dirs=/tmp/kafka-logs* with
*log.dirs=tmp/kafka-logs* (see Trouble-shooting section \#3).
3. Follow the Kafka page directions to start the zookeeper and Kafka servers,
i.e., from directory kafka\_2.11-0.9.0.0:
3a. *./bin/zookeeper-server-start.sh config/zookeeper.properties*
3b. *./bin/kafka-server-start.sh config/server.properties*
4. To manually execute the Quarks \*xxxTestManual.java tests (per text in
build.xml), run:
*ant -Dtest.base.pattern='\*\*/\*TestManual.java' test*
Look for these results:
test.run:
\[mkdir\] Created dir:
/homes/hny7/maywone/git/quarks/connectors/kafka/unittests
\[mkdir\] Created dir:
/homes/hny7/maywone/git/quarks/connectors/kafka/unittests/testrun1997679054
\[jacoco:coverage\] Enhancing junit with coverage
\[junit\] Running quarks.test.connectors.kafka.KafkaStreamsTestManual
\[junit\] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
17.235 sec
Trouble-shooting
Symptom \#1:
Starting the zookeeper results in:
-bash-4.2$ ./bin/zookeeper-server-start.sh config/zookeeper.properties
<?xml version="1.0" ?>
<verbosegc xmlns="http://www.ibm.com/j9/verbosegc";
version="R28\_20160106\_1341\_B284759\_CMPRSS">
JVMJ9VM007E Command-line option unrecognised:
-Xloggc:/homes/hny7/maywone/kafka2/kafka\_2.11-0.9.0.0/bin/../logs/zookeeper-gc.log
</verbosegc>
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
Work around:
This type of error may indicate your machine’s JVM does not support this
option.
Edit kafka-run-class.sh to delete any unsupported option(s). For example,
replace
KAFKA\_GC\_LOG\_OPTS="-Xloggc:$LOG\_DIR/$GC\_LOG\_FILE\_NAME -verbose:gc
-XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps "
with KAFKA\_GC\_LOG\_OPTS="-verbose:gc -XX:+PrintGCDetails
-XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps "
-----------------------------------------------------------------------------------------
Symptom \#2
Starting the zookeeper results in:
\[2016-02-19 13:43:42,002\] ERROR Severe unrecoverable error, exiting
(org.apache.zookeeper.server.SyncRequestProcessor)
java.io.FileNotFoundException: /tmp/zookeeper/version-2/log.94 (Permission
denied)
at java.io.FileOutputStream.open(FileOutputStream.java:286)
at java.io.FileOutputStream.<init>(FileOutputStream.java:226)
at java.io.FileOutputStream.<init>(FileOutputStream.java:173)
at
org.apache.zookeeper.server.persistence.FileTxnLog.append(FileTxnLog.java:205)
at
org.apache.zookeeper.server.persistence.FileTxnSnapLog.append(FileTxnSnapLog.java:314)
at org.apache.zookeeper.server.ZKDatabase.append(ZKDatabase.java:476)
at
org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:140)
</verbosegc>
Work-around:
Modify zookeeper.properties to replace *dataDir=/tmp/zookeeper* with
*dataDir=~/tmp/zookeeper*
Apparently, this zookeeper property value specifies an absolute path.
-----------------------------------------------------------------------------------------
Symptom \#3:
Starting the Kafka server results in:
\[2016-02-19 13:51:57,161\] FATAL Fatal error during KafkaServer startup.
Prepare to shutdown (kafka.server.KafkaServer)
java.io.FileNotFoundException: /tmp/kafka-logs/.lock (Permission denied)
at java.io.RandomAccessFile.open0(Native Method)
at java.io.RandomAccessFile.open(RandomAccessFile.java:333)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:257)
at kafka.utils.FileLock.<init>(FileLock.scala:29)
at kafka.log.LogManager$$anonfun$lockLogDirs$1.apply(LogManager.scala:96)
at kafka.log.LogManager$$anonfun$lockLogDirs$1.apply(LogManager.scala:95)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at kafka.log.LogManager.lockLogDirs(LogManager.scala:95)
at kafka.log.LogManager.<init>(LogManager.scala:57)
at kafka.server.KafkaServer.createLogManager(KafkaServer.scala:589)
at kafka.server.KafkaServer.startup(KafkaServer.scala:171)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)
at kafka.Kafka$.main(Kafka.scala:67)
at kafka.Kafka.main(Kafka.scala)
Workaround:
Modify server.properties to replace *log.dirs=/tmp/kafka-l*ogs with
*log.dirs=tmp/kafka-logs*
Apparently this server property value specifies a relative path.
> Instructions for setting up Kafka and MQTT (Mosquitto) and running manual
> tests
> -------------------------------------------------------------------------------
>
> Key: QUARKS-19
> URL: https://issues.apache.org/jira/browse/QUARKS-19
> Project: Quarks
> Issue Type: Improvement
> Components: Documentation
> Reporter: May Wone
> Assignee: May Wone
> Priority: Minor
>
> Will attach instructions for setting up Kafka and Mosquitto (MQTT) and
> running the Quarks manual tests.
> I'd welcome feedback on the contents.
> Also, I'm thinking about where to put this information - my initial take is
> on a 'Quarks wiki'. Comments?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
