mjordyn commented on issue #43717:
URL: https://github.com/apache/airflow/issues/43717#issuecomment-2710420656
> Thanks for the info @mjordyn! Can you share a snippet of where you're
putting the sleep in your ECS task? It sounds like you're putting it at the
beginning of your task, then your task does it's actual work, then you see the
logs? Or did you replace the actual work with only a sleep? If you have an
actual code snipped (or if you could reproduce what change you made) and share
it, that would be tremendous :)
Here's a highly simplified/psuedocode version of the code for one of the
tasks with the sleep addition:
```
import logging
import argparse
import time
import anotherpackage
import asecondpackage
logging.basicConfig(level="INFO")
logger = logging.getLogger(__name__)
def main():
parser = argparse.ArgumentParser()
parser.add_argument('-a', '--arg_a', help='Arg a')
parser.add_argument('-b', '--arg_b', help='Arg b')
parser.add_argument('-c', '--arg_c', help='Arg c')
parser.add_argument('-d', '--arg_d', help='Arg d')
args = vars(parser.parse_args())
time.sleep(10)
logger.info("Starting task")
logger.info("Logging something else")
var1 = args['key1']
var2 = args['key2']
var3 = args['key3']
start_time = time.time()
client = get_client()
files = get_file_list()
file_count = len(files)
for i, file in enumerate(files):
if i % 100 == 0:
logger.info(f"{(i/file_count) * 100} percent complete after
{(time.time()-start_time)/60} minutes")
process_file(file, client)
logger.info("Finished task")
```
The only difference between the first few logs showing up in the UI or not
is whether or not the `time.sleep(10)` line was present or not. That sleep did
not replace any actual processing code, it was purely an addition. When the
sleep line was present, everything showed up in the Airflow UI. When it wasn't,
the first 2 `logger.info` messages did not appear. The first `logger.info`
message within the loop sometimes did not show up (and sometimes did), but all
messages after that always did. Regardless of whether or not the sleep
statement was present, all logs always appeared in the ECS task Cloudwatch logs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
