iamrsaravana commented on issue #817:
URL:
https://github.com/apache/camel-kafka-connector/issues/817#issuecomment-752314128
Now I am able to send the data from Hono to Strimzi Kafka Connector using
Camel AMQP Source Connector.
Follow the below steps to pull data from Hono and send to Strimzi Kafka
Connector:
1) Creating Name space:
kubectl create namespace kafka
2) install latest Strimzi operator using the below command:
kubectl apply -f 'https://strimzi.io/install/latest?namespace=kafka' -n
kafka
3) creating Kafka (1 pod) and zookeeper(2 pods ) using the below yaml
configuration:
**kafka.yaml:**
apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
name: my-cluster
spec:
kafka:
version: 2.6.0
replicas: 1
listeners:
- name: plain
port: 9092
type: internal
tls: false
- name: tls
port: 9093
type: internal
tls: true
config:
offsets.topic.replication.factor: 1
transaction.state.log.replication.factor: 1
transaction.state.log.min.isr: 1
log.message.format.version: "2.6"
storage:
type: jbod
volumes:
- id: 0
type: persistent-claim
size: 10Gi
deleteClaim: false
zookeeper:
replicas: 2
storage:
type: persistent-claim
size: 10Gi
deleteClaim: false
entityOperator:
topicOperator: {}
userOperator: {}
Kuebctl apply -f kafka.yaml -n kafka.
_ps: Here I use glusterfs persistent storage._
4) Creating AMQP - Source Connector image file using below commands:
I downloaded Camel-amqp-kafka-connector, JMS Jar files from below link:
https://repo.maven.apache.org/maven2/org/apache/camel/kafkaconnector/camel-amqp-kafka-connector/0.7.0/camel-amqp-kafka-connector-0.7.0-package.tar.gz
https://downloads.apache.org/qpid/jms/0.51.0/apache-qpid-jms-0.51.0-bin.tar.gz
After downloaded the above tar and unzipped and created docker
image file using below command
**Dockerfile:**
FROM strimzi/kafka:0.20.1-kafka-2.6.0
USER root:root
RUN mkdir -p
/opt/kafka/plugins/camel-kafka-connectors/camel-amqp-kafka-connector/
COPY ./plugin/apache-qpid-jms-0.55.0/*
/opt/kafka/plugins/camel-kafka-connectors/camel-amqp-kafka-connector/
COPY ./camel-amqp-kafka-connector/*
/opt/kafka/plugins/camel-kafka-connectors/camel-amqp-kafka-connector/
USER 1001
Docker build -f ./Dockerfile -t
localhost:5000/my-connector-amqp_new .
docker push localhost:5000/my-connector-amqp_new
_ps: Here I have used local docker repository_
5) Creating Kafkaconnect using below Yaml configuration:
Kafkaconnect.yaml:
apiVersion: kafka.strimzi.io/v1beta1
kind: KafkaConnect
metadata:
name: my-connect-cluster-new
annotations:
strimzi.io/use-connector-resources: "true"
spec:
image: 10.128.0.6:5000/my-connector-amqp_new5
replicas: 1
bootstrapServers: my-cluster-kafka-bootstrap:9092
config:
group.id: connect-cluster-new
offset.storage.topic: connect-cluster-offsets
config.storage.topic: connect-cluster-configs
status.storage.topic: connect-cluster-status
config.storage.replication.factor: 1
offset.storage.replication.factor: 1
status.storage.replication.factor: 1
kubectl apply -f Kafkaconnect.yaml -n kafka
6) Creating AMQP-KafkaConnector using below yaml file:
amqp_connector.yaml
apiVersion: kafka.strimzi.io/v1alpha1
kind: KafkaConnector
metadata:
name: camelamqpsourceconnector
labels:
strimzi.io/cluster: my-connect-cluster-new
spec:
class:
org.apache.camel.kafkaconnector.amqp.CamelAmqpSourceConnector
tasksMax: 1
config:
camel.component.amqp.includeAmqpAnnotations: true
camel.component.amqp.connectionFactory:
'#class:org.apache.qpid.jms.JmsConnectionFactory'
camel.component.amqp.connectionFactory.remoteURI:
amqp://10.106.88.243:15672
camel.component.amqp.username: consumer@HONO
camel.component.amqp.password: verysecret
camel.component.amqp.testConnectionOnStartup: true
camel.source.path.destinationType: queue
#( my usecase i am reading from hono)
camel.source.path.destinationName: telemetry/LTTS_TENANT
# ( this is the place where hono writing the data)
topics: mytopic
#(kafka topics where you want to write it)
kubectl apply -f amqp_connector.yaml -n kafka.
ps: IP address given here all are my pod ip address. it may vary according
to your pod.
7 ) viewing data in strimzi kafka using the below command:
kubectl -n kafka exec my-cluster-kafka-0 -c kafka -i -t --
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic
mytopic
I Hope this may be useful for somebody who wants to pull the data directly
using AMQP Camel Connector.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
