steveloughran commented on code in PR #1139:
URL: https://github.com/apache/parquet-mr/pull/1139#discussion_r1560991914
##########
parquet-common/src/main/java/org/apache/parquet/io/SeekableInputStream.java:
##########
@@ -104,4 +106,25 @@ public abstract class SeekableInputStream extends
InputStream {
* fill the buffer, {@code buf.remaining()}
*/
public abstract void readFully(ByteBuffer buf) throws IOException;
+
+ /**
+ * Read a set of file ranges in a vectored manner.
+ *
+ * @param ranges the list of file ranges to read
+ * @param allocator the allocator to use for allocating ByteBuffers
+ * @throws UnsupportedOperationException if not available in this
class/runtime (default)
+ */
+ public void readVectored(List<ParquetFileRange> ranges, final
ByteBufferAllocator allocator) throws IOException {
+
+ throw new UnsupportedOperationException("Vectored IO is not supported for
" + this);
+ }
+
+ /**
+ * Is the {@link #readVectored(List, ByteBufferAllocator)} method available?
+ * @param allocator the allocator to use for allocating ByteBuffers
Review Comment:
aah, you've noticed that! It's because I've realised that on s3/abfs/gcs
network failures which don't recover after retries, we need a way to return the
buffer to the pool. See https://issues.apache.org/jira/browse/HADOOP-19105
I actually need to do some more in general on s3a read() recovery: we retry
on the GET calls and then keep the connection active only as long as it takes
to read the data -so no risk of stale connections is low- but I do need to add
the full resilience logic there
* abort http connections considered unrecoverable, rather than return to pool
* retry outstanding entries in each range
* maybe actually handle failures by decombining adjacent ranges.
Having parquet pass the allocator as far down as it does means that
supporting releases with the "return to pool" callback doesn't change the rest
of the code, instead the bridge would first look for the new method, falling
back to the original one if not found.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
