uranusjr commented on code in PR #38807:
URL: https://github.com/apache/airflow/pull/38807#discussion_r1555488988
##########
airflow/providers/amazon/aws/log/s3_task_handler.py:
##########
@@ -178,7 +180,13 @@ def s3_read(self, remote_log_location: str, return_error:
bool = False) -> str:
:return: the log found at the remote_log_location
"""
try:
- return self.hook.read_key(remote_log_location)
+ range: str = None
+ if page_number is not None:
+ page_size = 1024 * 100 # TODO: Create config for page_size
Review Comment:
I wonder if we should just put this in the API interface. So instead of a
page number (of fixed size pages), the API user would do `offset=0&limit=500`
and `offset=500&limit=500` and so on. (I took the argument names from SQL but
other suggestions are very much welcomed.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
