Taragolis commented on code in PR #26269:
URL: https://github.com/apache/airflow/pull/26269#discussion_r979213347
##########
airflow/providers/amazon/aws/hooks/glue.py:
##########
@@ -151,85 +153,150 @@ def print_job_logs(
job_failed: bool = False,
next_token: Optional[str] = None,
) -> Optional[str]:
- """Prints the batch of logs to the Airflow task log and returns
nextToken."""
- log_client = boto3.client('logs')
- response = {}
-
- filter_pattern = FAILURE_LOG_FILTER if job_failed else
DEFAULT_LOG_FILTER
+ """Prints the batch of Glue cloudwatch logs to the Airflow task log
and returns nextToken."""
+ credentials = self.get_credentials(region_name=self.conn_region_name)
+ log_client = boto3.client(
+ 'logs',
+ region_name=self.conn_region_name,
+ aws_access_key_id=credentials.access_key,
+ aws_secret_access_key=credentials.secret_key,
+ )
Review Comment:
> creating a new client for logs from the same AWSBaseHook class is not
possible
Yes you can not create different client type by call
`AWSBaseHook.get_client_type` since latest changes.
https://github.com/apache/airflow/blob/3b6176929ccc0351554e5331d920c4ec65b33a38/airflow/providers/amazon/aws/hooks/base_aws.py#L438-L454
But you still can create `boto.session.Session()` and can use this session
for create whatever client you wanted
https://github.com/apache/airflow/blob/3b6176929ccc0351554e5331d920c4ec65b33a38/airflow/providers/amazon/aws/hooks/base_aws.py#L432-L436
**Generic approach for boto3 client**
```python
session = boto3.session.Session(...)
client = session("awesome-client", ...)
```
**How it could be archived by `AwsGenericHook` / `AWSBaseHook` and any their
child**
```python
hook = GlueJobHook(aws_conn_id="awesome_aws_conn_id", ...)
session = self.get_session(...)
client = session("awesome-client", ...)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
