manoj-mathivanan commented on code in PR #18391:
URL: https://github.com/apache/kafka/pull/18391#discussion_r1905788910
##########
tools/src/main/java/org/apache/kafka/tools/ProducerPerformance.java:
##########
@@ -194,9 +195,16 @@ static List<byte[]> readPayloadFile(String
payloadFilePath, String payloadDelimi
throw new IllegalArgumentException("File does not exist or
empty file provided.");
}
- String[] payloadList =
Files.readString(path).split(payloadDelimiter);
+ List<String> payloadList = new ArrayList<>();
+ try (Scanner payLoadScanner = new Scanner(path,
StandardCharsets.UTF_8)) {
+ //setting the delimiter while parsing the file, avoids loading
entire data in memory before split
+ payLoadScanner.useDelimiter(payloadDelimiter);
+ while (payLoadScanner.hasNext()) {
+ payloadList.add(payLoadScanner.next());
Review Comment:
@m1a2st The `payLoadScanner.next()` starts parsing from the position where
it is currently at and keeps searching for the delimiter till the end of file
as seen here:
https://github.com/openjdk/jdk/blob/f1d85ab3e61f923b4e120cf30e16109e04505b53/src/java.base/share/classes/java/util/Scanner.java#L1011
So, if the delimiter does not exist, we will have the same problem which
will happen even in the previous case where we load the entire string.
If the delimiter exists, the substrings are fetched and the new position is
updated. The entire String is not in the memory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
