HarshSawarkar commented on code in PR #4650:
URL: https://github.com/apache/eventmesh/pull/4650#discussion_r1444470313
##########
eventmesh-connectors/eventmesh-connector-file/src/main/java/org/apache/eventmesh/connector/file/source/connector/FileSourceConnector.java:
##########
@@ -73,12 +93,48 @@ public String name() {
@Override
public void stop() {
-
+ try {
+ if (bufferedReader != null) {
+ bufferedReader.close();
+ }
+ } catch (Exception e) {
+ log.error("Error closing resources: {}", e.getMessage());
+ }
}
@Override
public List<ConnectRecord> poll() {
- return null;
+ List<ConnectRecord> connectRecords = new ArrayList<>();
+ try {
+ String line;
+ while ((line = bufferedReader.readLine()) != null) {
+ ConnectRecord connectRecord = new ConnectRecord(new
RecordPartition(), new RecordOffset(), System.currentTimeMillis(), line);
Review Comment:
I have addressed the implementation of fragmented reads through offset in
the updated code. This was achieved by reading the file in chunks and recording
the offset based on the last position index in the buffer array. Additionally,
I ensured that the last position is updated in each iteration to maintain
accurate tracking. Thank you for bringing this to my attention.
##########
eventmesh-connectors/eventmesh-connector-file/src/main/resources/source-config.yml:
##########
@@ -15,20 +15,13 @@
# limitations under the License.
#
-pubSubConfig:
- meshAddress: 127.0.0.1:10000
- subject: TopicTest
- idc: FT
- env: PRD
- group: fileSource
- appId: 5032
- userName: fileSourceUser
- passWord: filePassWord
connectorConfig:
connectorName: fileSource
nameserver: 127.0.0.1:9877
topic: TopicTest
commitOffsetIntervalMs: 5000
+ fileName: TopicTest-%s
+ filePath: TopicTest/%s
Review Comment:
Updated
##########
eventmesh-connectors/eventmesh-connector-file/src/main/resources/source-config.yml:
##########
@@ -15,20 +15,13 @@
# limitations under the License.
#
-pubSubConfig:
- meshAddress: 127.0.0.1:10000
- subject: TopicTest
- idc: FT
- env: PRD
- group: fileSource
- appId: 5032
- userName: fileSourceUser
- passWord: filePassWord
connectorConfig:
connectorName: fileSource
nameserver: 127.0.0.1:9877
topic: TopicTest
commitOffsetIntervalMs: 5000
+ fileName: TopicTest-%s
+ filePath: TopicTest/%s
offsetStorageConfig:
offsetStorageType: nacos
offsetStorageAddr: 127.0.0.1:8848
Review Comment:
I have implemented the commitOffsetIntervalMs in the updated code.
##########
eventmesh-connectors/eventmesh-connector-file/src/main/resources/source-config.yml:
##########
@@ -15,20 +15,13 @@
# limitations under the License.
#
-pubSubConfig:
Review Comment:
Updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
