manoj-mathivanan commented on code in PR #18391:
URL: https://github.com/apache/kafka/pull/18391#discussion_r2078312773
##########
tools/src/main/java/org/apache/kafka/tools/ProducerPerformance.java:
##########
@@ -194,9 +195,16 @@ static List<byte[]> readPayloadFile(String
payloadFilePath, String payloadDelimi
throw new IllegalArgumentException("File does not exist or
empty file provided.");
}
- String[] payloadList =
Files.readString(path).split(payloadDelimiter);
+ List<String> payloadList = new ArrayList<>();
+ try (Scanner payLoadScanner = new Scanner(path,
StandardCharsets.UTF_8)) {
+ //setting the delimiter while parsing the file, avoids loading
entire data in memory before split
+ payLoadScanner.useDelimiter(payloadDelimiter);
Review Comment:
The main issue is because we try to load the entire file into a single
String object which is limited. Even though the machine has memory, the String
object is limited to the size of an array as seen here:
https://github.com/openjdk/jdk/blob/f1d85ab3e61f923b4e120cf30e16109e04505b53/src/java.base/share/classes/java/lang/String.java#L568
When we use a delimiter with a Scanner, the entire file is not loaded into a
String object before splitting it.
The payLoadScanner.next() starts parsing from the position where it is
currently at and keeps searching for the delimiter as seen here:
https://github.com/openjdk/jdk/blob/f1d85ab3e61f923b4e120cf30e16109e04505b53/src/java.base/share/classes/java/util/Scanner.java#L1011
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
