potiuk commented on pull request #11659:
URL: https://github.com/apache/airflow/pull/11659#issuecomment-712670165
Hey @ashb @mik-laj @kaxil @turbaszek - > I need your help. I have this
really strange problem and maybe you will be able to help quickly. This is the
LAST step to have stable selective tests working.
The MySQL tests in this build fail because of fernet.InvalidToken and I
can't figure it out quickly:
It's rather easy to reproduce:
```
./breeze --backend sqlite --db-reset
```
And then:
```
pytest
tests/providers/amazon/aws/hooks/test_emr.py::TestEmrHook::test_create_job_flow_extra_args
```
This fails with:
```
____________________________________________________________________________________________________________________________
TestEmrHook.test_create_job_flow_extra_args
_____________________________________________________________________________________________________________________________
self = <tests.providers.amazon.aws.hooks.test_emr.TestEmrHook
testMethod=test_create_job_flow_extra_args>
@mock_emr
def test_create_job_flow_extra_args(self):
"""
Test that we can add extra arguments to the launch call.
This is useful for when AWS add new options, such as
"SecurityConfiguration" so that we don't have to change our code
"""
client = boto3.client('emr', region_name='us-east-1')
hook = EmrHook(aws_conn_id='aws_default', emr_conn_id='emr_default')
# AmiVersion is really old and almost no one will use it anymore, but
# it's one of the "optional" request params that moto supports - it's
# coverage of EMR isn't 100% it turns out.
> cluster = hook.create_job_flow({'Name': 'test_cluster',
'ReleaseLabel': '', 'AmiVersion': '3.2'})
tests/providers/amazon/aws/hooks/test_emr.py:62:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
airflow/providers/amazon/aws/hooks/emr.py:80: in create_job_flow
config = emr_conn.extra_dejson.copy()
airflow/models/connection.py:377: in extra_dejson
if self.extra:
/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/attributes.py:358: in
__get__
retval = self.descriptor.__get__(instance, owner)
airflow/models/connection.py:292: in get_extra
return fernet.decrypt(bytes(self._extra, 'utf-8')).decode()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <cryptography.fernet.MultiFernet object at 0x7fa7aa0ebed0>
msg =
b'gAAAAABfjpjdT5OM4KF8yhsX_-nEC3-53b-XDTF4hpZVTzqVrYjEh5AHebgky5PCvt_hogmCInRNWXm5Zhe_zK-x9-TgAYxQlKZ2UwBcTaTi4uufY0ji...Ppw6xt39aGU0rQF_NbZYSXL0Ls9XceeALWIuGjEuxtpEGmlyX1bV2vAS9dU36rzffPcZj0VM8zvpbn3gNaFeZBq0aSSKPHtYOtzsWFj8qVdxQsellPHyI=',
ttl = None
def decrypt(self, msg, ttl=None):
for f in self._fernets:
try:
return f.decrypt(msg, ttl)
except InvalidToken:
pass
> raise InvalidToken
E cryptography.fernet.InvalidToken
/usr/local/lib/python3.7/site-packages/cryptography/fernet.py:183:
InvalidToken
---------------------------------------------------------------------------------------------------------------------------------------
Captured stdout setup
----------------------------------------------------------------------------------------------------------------------------------------
========================= AIRFLOW ==========================
Home of the user: /root
Airflow home /root/airflow
Skipping initializing of the DB as it was initialized already.
You can re-initialize the database by adding --with-db-init flag when
running tests.
----------------------------------------------------------------------------------------------------------------------------------------
Captured stdout call
----------------------------------------------------------------------------------------------------------------------------------------
[2020-10-20 07:59:31,472] {credentials.py:1087} INFO - Found credentials in
environment variables.
-----------------------------------------------------------------------------------------------------------------------------------------
Captured log call
------------------------------------------------------------------------------------------------------------------------------------------
INFO botocore.credentials:credentials.py:1087 Found credentials in
environment variables.
======================================================================================================================================
short test summary info
=======================================================================================================================================
FAILED
tests/providers/amazon/aws/hooks/test_emr.py::TestEmrHook::test_create_job_flow_extra_args
- cryptography.fernet.InvalidToken
```
Does it ring a bell maybe ??????
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
