robertwb commented on code in PR #33086:
URL: https://github.com/apache/beam/pull/33086#discussion_r1838542599
##########
sdks/java/core/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataInboundObserver.java:
##########
@@ -118,6 +121,35 @@ public boolean isConsumingReceivedData() {
return consumingReceivedData.get();
}
+ // Copies the elements of list to an array and removes references to
elements that
+ // have been iterated past.
+ @SuppressWarnings({"initialization", "assignment"})
+ private static <T> Iterator<T> createDiscardingIterator(List<T> list) {
+ if (list.isEmpty()) {
+ // As optimization, don't create array etc for empty list.
+ return list.iterator();
+ }
+
+ @Nullable Object[] array = list.toArray();
+ return new Iterator<T>() {
+ int i = 0;
+
+ @Override
+ public boolean hasNext() {
+ return i < array.length;
+ }
+
+ @Override
+ public T next() {
+ @SuppressWarnings("unchecked")
+ T result = (T) Preconditions.checkNotNull(array[i]);
Review Comment:
Why do we need to do this check? Isn't null a valid element of the original
list?
##########
sdks/java/core/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataInboundObserver.java:
##########
@@ -118,6 +121,35 @@ public boolean isConsumingReceivedData() {
return consumingReceivedData.get();
}
+ // Copies the elements of list to an array and removes references to
elements that
+ // have been iterated past.
+ @SuppressWarnings({"initialization", "assignment"})
+ private static <T> Iterator<T> createDiscardingIterator(List<T> list) {
+ if (list.isEmpty()) {
+ // As optimization, don't create array etc for empty list.
+ return list.iterator();
+ }
+
+ @Nullable Object[] array = list.toArray();
Review Comment:
How expensive is this copy? Also, this would reduce average memory by half,
but not help with peak memory, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
