please...?
On Thu, Feb 9, 2017 at 8:35 AM, Boris Tyukin <[email protected]> wrote:
> Hello,
>
> I am using HiveCliHook called from PythonOperator to run a series of
> queries and want to capture record counts for auditing and validation
> purposes.
>
> *I am thinking to use on_success_callback to invoke python function that
> will read the log file, produced by airflow and then parse it out using
> regex. *
>
> *I am going to use this method from models to get to the file log:*
>
> *def log_filepath(self): iso = self.execution_date.isoformat() log =
> os.path.expanduser(configuration.get('core', 'BASE_LOG_FOLDER')) return (
> "{log}/{self.dag_id}/{self.task_id}/{iso}.log".format(**locals()))*
> Is this a good strategy or there is an easier way? I wondering if someone
> did something similar.
>
> Another challenge is that the same log file contains multiple attempts and
> reruns of the same task so I guess I need to parse the file backwards.
>
> thanks,
> Boris
>
