odbozhou commented on code in PR #382:
URL: https://github.com/apache/rocketmq-connect/pull/382#discussion_r1031173503
##########
connectors/rocketmq-connect-sftp/src/main/java/org/apache/rocketmq/connect/http/sink/SftpSourceTask.java:
##########
@@ -0,0 +1,120 @@
+package org.apache.rocketmq.connect.http.sink;
+
+import io.openmessaging.KeyValue;
+import io.openmessaging.connector.api.component.task.source.SourceTask;
+import io.openmessaging.connector.api.component.task.source.SourceTaskContext;
+import io.openmessaging.connector.api.data.ConnectRecord;
+import io.openmessaging.connector.api.data.RecordOffset;
+import io.openmessaging.connector.api.data.RecordPartition;
+import org.apache.commons.lang3.StringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.nio.file.FileSystemException;
+import java.util.*;
+
+import static org.apache.rocketmq.connect.http.sink.SftpConstant.*;
+
+public class SftpSourceTask extends SourceTask {
+
+ private final Logger log =
LoggerFactory.getLogger(SftpConstant.LOGGER_NAME);
+
+ private SftpClient sftpClient;
+
+ private String filename;
+
+ private static final int MAX_NUMBER_SEND_CONNECT_RECORD_EACH_TIME = 2000;
+
+ @Override
+ public void init(SourceTaskContext sourceTaskContext) {
+ super.init(sourceTaskContext);
+ }
+
+ @Override
+ public void start(KeyValue config) {
+ String host = config.getString(SFTP_HOST_KEY);
+ int port = config.getInt(SFTP_PORT_KEY);
+ String username = config.getString(SFTP_USERNAME_KEY);
+ String password = config.getString(SFTP_PASSWORD_KEY);
+ String path = config.getString(SFTP_PATH_KEY);
+ filename = config.getString(SFTP_FILENAME_KEY);
+ sftpClient = new SftpClient(host, port, username, password, path);
+ }
+
+ @Override
+ public void stop() {
+ }
Review Comment:
Create an sftp session every time you poll or put data, and close the
connection after the batch of messages are processed. Will it affect the
processing efficiency? Is it possible to create a connection when the task is
started, and close the connection when the task is stopped? Considering the
sftp link disconnection, it is possible to re-create the connection when it is
detected that the connection is unavailable. Does this take performance and
availability into account? These are just some of my suggestions, if possible,
consider implementing them in the next version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
