hit-lacus edited a comment on issue #1166: Prepare DebugTomcat and Integration Test in Hadoop3 Env URL: https://github.com/apache/kylin/pull/1166#issuecomment-603282888 ## Prepare Kafka #### Download Kafka - kafka_2.12-1.1.0.tgz #### Start Kafka > I met some error in starting the **Kafka** service contained in sandbox. So I choose to download it from apache website. 1. Modify `$KAFKA_HOME/config/server.properties` ```sh # Do not use the default port listeners=PLAINTEXT://sandbox-hdp.hortonworks.com:9093 # Do not use the default storage path log.dirs=kafka-logs # Do not use the default metadata path zookeeper.connect=sandbox-hdp.hortonworks.com:2181/lacus ``` 2. Start broker ```sh nohup sh bin/kafka-server-start.sh config/server.properties & ``` 3. Test producer and consumer > Test them both in Laptop and HDP container ```sh # Create a topic sh bin/kafka-topics.sh --create --topic test02 --zookeeper 10.1.0.2:2181/lacus --partitions 3 --replication-factor 1 # Send some message sh bin/kafka-console-producer.sh --topic test02 --broker-list 10.1.0.2:9093 # Consume all these message sh bin/kafka-console-consumer.sh --topic test02 --bootstrap-server 10.1.0.2:9093 --from-beginning ``` ### Fix KYLIN-3970 > Error: org.apache.kafka.clients.consumer.ConsumerConfig.configNames()Ljava/util/Set ```sh ## remove old version kafka lib mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old ## remove mapdreduce lib cache rm -rf /data/hadoop/yarn/local/usercache/${LOGIN_NMAE}/appcache ## remove kafka-clients-0.8.2.1.jar in mapreduce.tar.gz(HDFS) hadoop fs -copyToLocal /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz . hadoop fs -mv /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz.ori tar zxf mapreduce.tar.gz rm hadoop/share/hadoop/tools/lib/kafka-clients-0.8.2.1.jar ## repackage mapreduce.tar.gz tar zcf mapreduce.tar.gz hadoop/ ## put it in right place hadoop fs -copyFromLocal mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
