[
https://issues.apache.org/jira/browse/CAMEL-16754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fernando updated CAMEL-16754:
------------------------------
Attachment: Screenshot_1.png
> Camel Kafka HDFS sink connector
> -------------------------------
>
> Key: CAMEL-16754
> URL: https://issues.apache.org/jira/browse/CAMEL-16754
> Project: Camel
> Issue Type: Bug
> Components: camel-hdfs, camel-kafka
> Affects Versions: 3.9.0, 3.10.0
> Reporter: Fernando
> Priority: Critical
> Attachments: Screenshot_1.png
>
> Original Estimate: 0.5h
> Remaining Estimate: 0.5h
>
> Hello,
> I'm trying to connect kafka and hdfs to store data. I set it up and it works
> correctly, but the problem arises when I save the kafka messages in hdfs as a
> file is created for each message. I would like to create a file containing
> multiple messages, but I can't solve this problem. I've change the value of
> {code:java}
> camel.sink.endpoint.splitStrategy=BYTES:1000000
> {code}
> {code:java}
> camel.sink.endpoint.splitStrategy=MESSAGES:10
> {code}
> But when I view the files in the hdfs folder, I see one file for each message
> (image adjunted).
> The full configuration of the connector is the next:
> {code:java}
> name=CamelHdfsSinkConnector
> connector.class=org.apache.camel.kafkaconnector.hdfs.CamelHdfsSinkConnector
> tasks.max=1
> # use the kafka converters that better suit your needs, these are just
> defaults:
> key.converter=org.apache.kafka.connect.storage.StringConverter
> value.converter=org.apache.kafka.connect.storage.StringConverter
> #key.converter=org.apache.kafka.connect.json.JsonConverter
> #value.converter=org.apache.kafka.connect.json.JsonConverter
> # comma separated topics to get messages from
> topics=modbus-office-topic
> # mandatory properties (for a complete properties list see the connector
> documentation):
> # HDFS host to use
> camel.sink.path.hostName=namenode
> camel.sink.path.port=9000
> camel.sink.endpoint.splitStrategy=BYTES:10000000
> # The directory path to use
> camel.sink.path.path=Example_folder
> {code}
> I am currently running hadoop version 3.1.2, I have my doubts that this is
> the problem, and I don't know if the problem is with the connector, the
> hadoop version or the connection configuration.
> Thanks for your time
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
