hit-lacus edited a comment on issue #1166: Prepare DebugTomcat and Integration Test in Hadoop3 Env URL: https://github.com/apache/kylin/pull/1166#issuecomment-603282888 ## Prepare Kafka ### Download Kafka > I met some error in starting the **Kafka** service contained in sandbox. So I choose to download another one from apache website. - kafka_2.12-1.1.0.tgz ### Start Kafka 1. Modify `$KAFKA_HOME/config/server.properties` > I don't cause any conflict with the default Kafka in HDP. ```sh # Do not use the default port listeners=PLAINTEXT://sandbox-hdp.hortonworks.com:9093 # Do not use the default storage path log.dirs=kafka-logs # Do not use the default metadata path zookeeper.connect=sandbox-hdp.hortonworks.com:2181/lacus ``` 2. Start broker ```sh cd $KAFKA_HOME nohup sh bin/kafka-server-start.sh config/server.properties & ``` 3. Test producer and consumer > Test them both in Laptop and HDP container ```sh # Create a topic sh bin/kafka-topics.sh --create --topic test02 --zookeeper 10.1.0.2:2181/lacus --partitions 3 --replication-factor 1 # Send some message sh bin/kafka-console-producer.sh --topic test02 --broker-list 10.1.0.2:9093 # Consume all these message sh bin/kafka-console-consumer.sh --topic test02 --bootstrap-server 10.1.0.2:9093 --from-beginning ``` ### Fix KYLIN-3970 > Error: org.apache.kafka.clients.consumer.ConsumerConfig.configNames()Ljava/util/Set ```sh ## remove old version kafka lib mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old ## remove mapdreduce lib cache rm -rf /data/hadoop/yarn/local/usercache/${LOGIN_NMAE}/appcache ## remove kafka-clients-0.8.2.1.jar in mapreduce.tar.gz(HDFS) hadoop fs -copyToLocal /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz . hadoop fs -mv /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz.ori tar zxf mapreduce.tar.gz rm hadoop/share/hadoop/tools/lib/kafka-clients-0.8.2.1.jar ## repackage mapreduce.tar.gz tar zcf mapreduce.tar.gz hadoop/ ## put it in right place hadoop fs -copyFromLocal mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
