FernandoDorado opened a new issue #1291:
URL: https://github.com/apache/camel-kafka-connector/issues/1291
Good afternoon,
I have been doing some testing of the kafka-hdfs connector, especially on
the way it stacks files within HDFS. The used configuration for the connector
is as follows:
```json
{
"name":"CamelHdfsSinkConnector",
"config":{
"connector.class":"org.apache.camel.kafkaconnector.hdfs.CamelHdfsSinkConnector",
"task.max":2,
"key.converter":"org.apache.kafka.connect.storage.StringConverter",
"value.converter":"org.apache.kafka.connect.storage.StringConverter",
"topics":"office-topic",
"camel.sink.path.hostName": "HAcluster",
"camel.sink.endpoint.namedNodes": "namenode-1:8020,namenode-2:8020",
"camel.sink.endpoint.splitStrategy": "IDLE:100000,BYTES:10000000",
"camel.sink.path.path": "user/kafka/"
}
}
```
I have verified that messages do indeed arrive and stack correctly in HDFS.
My problem comes because these messages are stacked without any line break to
separate them in order to be able to use them for example with Hive. An example
of a msg in HDFS is:
I have seen that this was not a problem that happened only to me, as there
is a [post on
StackOverflow](https://stackoverflow.com/questions/67374326/append-a-new-line-character-at-every-record-passing-through-camelkafkaazureblobs)
with this problem.
I have tried the proposed solution, but since I am not a Java expert, I have
not obtained good results. I have modified the Apply method of
CamelTypeConverTransform, but I don't know if the post refers to this or
another one. This would be the method I have tried:
```java
@Override
public R apply(R record) {
final Schema schema = operatingSchema(record);
final Object value = operatingValue(record);
final Object convertedValue =
convertValueWithCamelTypeConverter(value);
final Object updatedValue = convertedValue + System.lineSeparator();
final Schema updatedSchema = getOrBuildRecordSchema(schema,
updatedValue);
return newRecord(record, updatedSchema, updatedValue);
}
```
A single msg has a format:
```json
{
"fields": {
"year": 2021,
"I1": 0,
"I2": 8.5143e-41,
"I3": 6.234535,
"PHI_UI1": 0,
"sec": 54,
"PHI_UI2": 8.6209e-41,
"P": 1335.8148,
"PHI_UI3": -20.375,
"Q": -584.91974,
"min": 42,
"month": 11,
"hour": 13,
"FREQ": 50,
"I_N": 5.915098,
"U1": 234.59723,
"UUID": 35697,
"day": 19,
"U2": 233.06522,
"U3": 234.024
},
"timestamp": 1637329377581,
"expires": null
}
```
An extracted portion of data from the HDFS file:
```json
{"fields":{"year":2021,"I1":0.0,"I2":4.8611E-41,"I3":6.2057714,"PHI_UI1":0.0,"sec":14,"PHI_UI2":4.104E-42,"P":1325.8503,"PHI_UI3":-20.5,"Q":-583.9375,"min":52,"month":11,"hour":13,"FREQ":49.75,"I_N":5.9074244,"U1":233.73982,"UUID":35697,"day":19,"U2":232.20639,"U3":233.972},"timestamp":1637329937662,"expires":null}{"fields":{"year":2021,"I1":0.0,"I2":4.1306E-41,"I3":6.2395077,"PHI_UI1":0.0,"sec":15,"PHI_UI2":5.0996E-41,"P":1323.7439,"PHI_UI3":-20.75,"Q":-599.9375,"min":52,"month":11,"hour":13,"FREQ":49.75,"I_N":5.9192123,"U1":233.64069,"UUID":35697,"day":19,"U2":232.09172,"U3":233.976},"timestamp":1637329938420,"expires":null}{"fields":{"year":2021,"I1":0.0,"I2":1.208E-41,"I3":6.243064,"PHI_UI1":0.0,"sec":15,"PHI_UI2":3.6516E-41,"P":1317.8503,"PHI_UI3":-20.5,"Q":-581.44635,"min":52,"month":11,"hour":13,"FREQ":49.75,"I_N":5.9074244,"U1":233.63208,"UUID":35697,"day":19,"U2":232.06783,"U3":233.976},"timestamp":1637329939102,"expires":null}
```
Where it can be seen that there is no separation between different msgs. Is
there some way to fix this problem?
Thank you in advance
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]