[jira] [Commented] (AIRFLOW-1153) params in HiveOperator constructor can't be passed into Hive execution context
[
https://issues.apache.org/jira/browse/AIRFLOW-1153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393256#comment-16393256
]
Ace Haidrey commented on AIRFLOW-1153:
--
Yes I agree it's important to have the script as decoupled as possible from
Airflow or any other system. Your current options are to pass the vars in the
user defined macros or to use a BashOperator and execute your bash command.
If you have a PR you want reviewed, let me know, because we welcome changes.
> params in HiveOperator constructor can't be passed into Hive execution context
> --
>
> Key: AIRFLOW-1153
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1153
> Project: Apache Airflow
> Issue Type: Bug
> Components: hive_hooks, operators
>Affects Versions: Airflow 2.0, Airflow 1.8
>Reporter: Xianping lin
>Priority: Critical
> Labels: easyfix, newbie
>
> params parameter in HiveOperator can't be imported into Hive executation
> context.
> so the following centence won't work, because 'mynumber' doesn't work for
> sql sentence.
> test_hiveoperator = HiveOperator(
> task_id='hive_test',
> hiveconf_jinja_translate=True,
> hql = ''' use myDB;
> INSERT OVERWRITE TABLE t2
> select * from t1 where t1.x > ' ${hiveconf:mynumber}'
> ''',
> params={'mynumber': 2},
> dag=dag
> )
> this modification pass the 'params' in HiveOperator construction to Hive
> sql execution context.
> The the variable definition can pass to hive sql
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-1153) params in HiveOperator constructor can't be passed into Hive execution context
[
https://issues.apache.org/jira/browse/AIRFLOW-1153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16391975#comment-16391975
]
Ace Haidrey commented on AIRFLOW-1153:
--
[~mikem...@gmail.com] agreed that would be nice and probably how it should be..
I think you can add it to the params? But I haven't tried this myself and I
agree it's not very straightforward on how to accomplish this. Do you have
suggestions
> params in HiveOperator constructor can't be passed into Hive execution context
> --
>
> Key: AIRFLOW-1153
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1153
> Project: Apache Airflow
> Issue Type: Bug
> Components: hive_hooks, operators
>Affects Versions: Airflow 2.0, Airflow 1.8
>Reporter: Xianping lin
>Priority: Critical
> Labels: easyfix, newbie
>
> params parameter in HiveOperator can't be imported into Hive executation
> context.
> so the following centence won't work, because 'mynumber' doesn't work for
> sql sentence.
> test_hiveoperator = HiveOperator(
> task_id='hive_test',
> hiveconf_jinja_translate=True,
> hql = ''' use myDB;
> INSERT OVERWRITE TABLE t2
> select * from t1 where t1.x > ' ${hiveconf:mynumber}'
> ''',
> params={'mynumber': 2},
> dag=dag
> )
> this modification pass the 'params' in HiveOperator construction to Hive
> sql execution context.
> The the variable definition can pass to hive sql
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-2017) Allow PostgresOperator to publish output to the Airflow task logs
Ace Haidrey created AIRFLOW-2017: Summary: Allow PostgresOperator to publish output to the Airflow task logs Key: AIRFLOW-2017 URL: https://issues.apache.org/jira/browse/AIRFLOW-2017 Project: Apache Airflow Issue Type: Improvement Components: operators Affects Versions: 1.9.0 Reporter: Ace Haidrey Assignee: Ace Haidrey Attachments: Screen Shot 2018-01-19 at 4.46.25 PM.png, Screen Shot 2018-01-19 at 4.46.59 PM.png Currently we're not getting the output logs of any postgres operator that you would get otherwise. It's because the postgres conn has an attribute called [notices|[http://initd.org/psycopg/docs/connection.html#connection.notices|http://initd.org/psycopg/docs/connection.html#connection.notices.]]. We need to just print the results of this to get that output in the airflow logs, which makes it easy to debug amongst other things. I've included some images for before and after pictures. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-1153) params in HiveOperator constructor can't be passed into Hive execution context
[
https://issues.apache.org/jira/browse/AIRFLOW-1153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16333071#comment-16333071
]
Ace Haidrey commented on AIRFLOW-1153:
--
The HiveOperator was updated recently:
[https://github.com/apache/incubator-airflow/commit/eb994d683f244f63dd191a6640baaee66ffc8e29]
In your example it your hql can remain the same and then in your
user_defined_macro to the dag you just pass your kv pair so user_defined_macro
= \{'mynumber': 2}
Think this ticket can be closed now
> params in HiveOperator constructor can't be passed into Hive execution context
> --
>
> Key: AIRFLOW-1153
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1153
> Project: Apache Airflow
> Issue Type: Bug
> Components: hive_hooks, operators
>Affects Versions: Airflow 2.0, Airflow 1.8
>Reporter: Xianping lin
>Priority: Critical
> Labels: easyfix, newbie
>
> params parameter in HiveOperator can't be imported into Hive executation
> context.
> so the following centence won't work, because 'mynumber' doesn't work for
> sql sentence.
> test_hiveoperator = HiveOperator(
> task_id='hive_test',
> hiveconf_jinja_translate=True,
> hql = ''' use myDB;
> INSERT OVERWRITE TABLE t2
> select * from t1 where t1.x > ' ${hiveconf:mynumber}'
> ''',
> params={'mynumber': 2},
> dag=dag
> )
> this modification pass the 'params' in HiveOperator construction to Hive
> sql execution context.
> The the variable definition can pass to hive sql
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (AIRFLOW-1735) Log files do not show up for unscheduled dags
[
https://issues.apache.org/jira/browse/AIRFLOW-1735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ace Haidrey closed AIRFLOW-1735.
Resolution: Won't Fix
> Log files do not show up for unscheduled dags
> -
>
> Key: AIRFLOW-1735
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1735
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Ace Haidrey
>Assignee: Ace Haidrey
>Priority: Major
> Labels: www
> Attachments: Screen Shot 2017-06-21 at 10.53.28 PM.png
>
>
> I created a new dag with a schedule of {{@once}} and left it unscheduled. I
> ran different tasks a few times from the UI but the logs never showed up.
> Instead I'd see messages like the following:
> {code}
> *** Log file isn't local.
> *** Fetching here:
> http://al-tools2.savagebeast.com:18700/log/analytics_kochava_import_dag/kopandoraradioandroid1134523b395684da7_request/2017-06-21
> *** Failed to fetch log file from worker.
> {code}
> There is a log file for this task, but it seems to be at a slightly different
> path on the machine:
> {code}
> (airflow)hadoop@al-tools2:~/.virtualenvs/airflow$ ls -lth
> ./airflow_home/logs/analytics_kochava_import_dag/kopandoraradioandroid1134523b395684da7_request/2017-06-21T00\:00\:00
> -rw-rw-rw- 1 hadoop hadoop 5.2K Jun 21 21:57
> ./airflow_home/logs/analytics_kochava_import_dag/kopandoraradioandroid1134523b395684da7_request/2017-06-21T00:00:00
> {code}
> And indeed, when I change the URL to
> WEB_SERVER/admin/airflow/log?task_id=TASK_ID_request_id=DAG_ID_date=2017-06-21T00:00:00,
> I see the actual log:
> {code}
> *** Reading local log.
> [2017-06-21 21:57:20,931] {models.py:172} INFO - Filling up the DagBag from
> ~/analytics_kochava_import_dag.py
> [2017-06-21 21:57:21,307] {analytics_kochava_import_dag.py:139} INFO -
> Kochava app IDs
> ...
> [2017-06-21 21:57:26,400] {jobs.py:2083} INFO - Task exited with return code 0
> {code}
> I've added a screen shot of it below.
> *FIX:*
> I have noticed in www/view.py the issue is not have the time component
> 00:00:00 and we can get that quite easily if it doesnt have it. You can see
> in this PR I made, but basically if we set
> {code}
> dttm = dateutil.parser.parse(request.args.get('execution_date'))
> execution_date = dttm.isoformat()
> {code}
> This will fix it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-1735) Log files do not show up for unscheduled dags
[
https://issues.apache.org/jira/browse/AIRFLOW-1735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16332748#comment-16332748
]
Ace Haidrey commented on AIRFLOW-1735:
--
Closing the PR because I believe it's now obsolete. Other changes to parse the
datetime have gone in:
https://github.com/apache/incubator-airflow/pull/2704
> Log files do not show up for unscheduled dags
> -
>
> Key: AIRFLOW-1735
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1735
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Ace Haidrey
>Assignee: Ace Haidrey
>Priority: Major
> Labels: www
> Attachments: Screen Shot 2017-06-21 at 10.53.28 PM.png
>
>
> I created a new dag with a schedule of {{@once}} and left it unscheduled. I
> ran different tasks a few times from the UI but the logs never showed up.
> Instead I'd see messages like the following:
> {code}
> *** Log file isn't local.
> *** Fetching here:
> http://al-tools2.savagebeast.com:18700/log/analytics_kochava_import_dag/kopandoraradioandroid1134523b395684da7_request/2017-06-21
> *** Failed to fetch log file from worker.
> {code}
> There is a log file for this task, but it seems to be at a slightly different
> path on the machine:
> {code}
> (airflow)hadoop@al-tools2:~/.virtualenvs/airflow$ ls -lth
> ./airflow_home/logs/analytics_kochava_import_dag/kopandoraradioandroid1134523b395684da7_request/2017-06-21T00\:00\:00
> -rw-rw-rw- 1 hadoop hadoop 5.2K Jun 21 21:57
> ./airflow_home/logs/analytics_kochava_import_dag/kopandoraradioandroid1134523b395684da7_request/2017-06-21T00:00:00
> {code}
> And indeed, when I change the URL to
> WEB_SERVER/admin/airflow/log?task_id=TASK_ID_request_id=DAG_ID_date=2017-06-21T00:00:00,
> I see the actual log:
> {code}
> *** Reading local log.
> [2017-06-21 21:57:20,931] {models.py:172} INFO - Filling up the DagBag from
> ~/analytics_kochava_import_dag.py
> [2017-06-21 21:57:21,307] {analytics_kochava_import_dag.py:139} INFO -
> Kochava app IDs
> ...
> [2017-06-21 21:57:26,400] {jobs.py:2083} INFO - Task exited with return code 0
> {code}
> I've added a screen shot of it below.
> *FIX:*
> I have noticed in www/view.py the issue is not have the time component
> 00:00:00 and we can get that quite easily if it doesnt have it. You can see
> in this PR I made, but basically if we set
> {code}
> dttm = dateutil.parser.parse(request.args.get('execution_date'))
> execution_date = dttm.isoformat()
> {code}
> This will fix it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (AIRFLOW-1995) add on kill method to sqoop operator
[ https://issues.apache.org/jira/browse/AIRFLOW-1995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ace Haidrey closed AIRFLOW-1995. > add on kill method to sqoop operator > > > Key: AIRFLOW-1995 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1995 > Project: Apache Airflow > Issue Type: Bug > Components: contrib >Affects Versions: 1.9.0 >Reporter: Ace Haidrey >Assignee: Ace Haidrey > Labels: sqoop > Fix For: 2.0.0 > > > The sqoop operator behind the scenes sets off a bash command, but one bug > arises due to the fact that the on kill method is not implemented. It needs > to be implemented in order to kill the process for a sqoop task, otherwise > weird things happen trying to end the task. > CC [~Fokko] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (AIRFLOW-1770) Add option to file and hiveconfs in HiveOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-1770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ace Haidrey closed AIRFLOW-1770. > Add option to file and hiveconfs in HiveOperator > > > Key: AIRFLOW-1770 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1770 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Ace Haidrey >Assignee: Alan Ma > Labels: operators > Fix For: 1.10.0, 1.9.1, 2.0.0 > > > The HiveOperator as it currently stands is not flexible enough to accept a > hive file and operate on that. You need to read in the contents and pass it > and if you do that you need to change the way hiveconfs are in your file to > jinja templating. > Many teams already have their existing sql/hql files and don't want to > convert them to make them as portable and decoupled as possible. > To accomplish this all we need to do is add the option to pass a hql_file and > hiveconfs to the HiveOperator. We change the code in the execute to throw an > error if both a hql_file and an hql statement are passed. If just hql_file > the simplest way without changing the code of the hive hook is to just read > the content of the hql_file and set it to be the self.hql. The hiveconfs get > passed directly to the run_cli method and we can combine them with the > already passed in hiveconfs. > If we want to make it optional to pass in the context as hiveconfs we can add > that too as related to AIRFLOW-788. > I've included some simple tests to show it all works how we expect. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (AIRFLOW-511) DagRun Failure: Retry, Email, Callbacks
[ https://issues.apache.org/jira/browse/AIRFLOW-511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ace Haidrey reassigned AIRFLOW-511: --- Assignee: Ace Haidrey > DagRun Failure: Retry, Email, Callbacks > --- > > Key: AIRFLOW-511 > URL: https://issues.apache.org/jira/browse/AIRFLOW-511 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Rob Froetscher >Assignee: Ace Haidrey >Priority: Minor > > Are there any plans to have retry, email, or callbacks on failure of a DAG > run? Would you guys be open to someone implementing that? Right now > particularly with dagrun_timeout, there is not much insight that the dag > actually stopped. > Pseudocode: > https://github.com/apache/incubator-airflow/compare/master...rfroetscher:dagrun_failure -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (AIRFLOW-1995) add on kill method to sqoop operator
Ace Haidrey created AIRFLOW-1995: Summary: add on kill method to sqoop operator Key: AIRFLOW-1995 URL: https://issues.apache.org/jira/browse/AIRFLOW-1995 Project: Apache Airflow Issue Type: Bug Components: contrib Affects Versions: 1.9.0 Reporter: Ace Haidrey Assignee: Ace Haidrey The sqoop operator behind the scenes sets off a bash command, but one bug arises due to the fact that the on kill method is not implemented. It needs to be implemented in order to kill the process for a sqoop task, otherwise weird things happen trying to end the task. CC [~Fokko] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (AIRFLOW-511) DagRun Failure: Retry, Email, Callbacks
[ https://issues.apache.org/jira/browse/AIRFLOW-511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323179#comment-16323179 ] Ace Haidrey edited comment on AIRFLOW-511 at 1/12/18 12:07 AM: --- Have a simple PR for this. https://github.com/apache/incubator-airflow/pull/2934 was (Author: ahaidrey): Have a simple PR for this coming soon. It fixes the issues we described by adding on success on failure callbacks on the dag level. > DagRun Failure: Retry, Email, Callbacks > --- > > Key: AIRFLOW-511 > URL: https://issues.apache.org/jira/browse/AIRFLOW-511 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Rob Froetscher >Priority: Minor > > Are there any plans to have retry, email, or callbacks on failure of a DAG > run? Would you guys be open to someone implementing that? Right now > particularly with dagrun_timeout, there is not much insight that the dag > actually stopped. > Pseudocode: > https://github.com/apache/incubator-airflow/compare/master...rfroetscher:dagrun_failure -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AIRFLOW-511) DagRun Failure: Retry, Email, Callbacks
[ https://issues.apache.org/jira/browse/AIRFLOW-511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323179#comment-16323179 ] Ace Haidrey commented on AIRFLOW-511: - Have a simple PR for this coming soon. It fixes the issues we described by adding on success on failure callbacks on the dag level. > DagRun Failure: Retry, Email, Callbacks > --- > > Key: AIRFLOW-511 > URL: https://issues.apache.org/jira/browse/AIRFLOW-511 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Rob Froetscher >Priority: Minor > > Are there any plans to have retry, email, or callbacks on failure of a DAG > run? Would you guys be open to someone implementing that? Right now > particularly with dagrun_timeout, there is not much insight that the dag > actually stopped. > Pseudocode: > https://github.com/apache/incubator-airflow/compare/master...rfroetscher:dagrun_failure -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AIRFLOW-511) DagRun Failure: Retry, Email, Callbacks
[ https://issues.apache.org/jira/browse/AIRFLOW-511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16290486#comment-16290486 ] Ace Haidrey commented on AIRFLOW-511: - That does sound like a possibility, but I just feel that is harder to ask a bunch of members from a company to think about in order to monitor. Do you not see it as a good idea to have a dagrun_timeout_callback or something of that sort? > DagRun Failure: Retry, Email, Callbacks > --- > > Key: AIRFLOW-511 > URL: https://issues.apache.org/jira/browse/AIRFLOW-511 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Rob Froetscher >Priority: Minor > > Are there any plans to have retry, email, or callbacks on failure of a DAG > run? Would you guys be open to someone implementing that? Right now > particularly with dagrun_timeout, there is not much insight that the dag > actually stopped. > Pseudocode: > https://github.com/apache/incubator-airflow/compare/master...rfroetscher:dagrun_failure -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AIRFLOW-511) DagRun Failure: Retry, Email, Callbacks
[ https://issues.apache.org/jira/browse/AIRFLOW-511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16290191#comment-16290191 ] Ace Haidrey commented on AIRFLOW-511: - [~aoen] [~bolke] [~Fokko] etc. Can one of you take a look at this request and give thoughts on how to best implement this. I can make the PR, testing, etc. if no one else will but I think this is quite an important feature. Airflow's pillars to visualize, maintain, and monitor pipelines, and not getting messages on dagrun timeouts has no solution to monitoring. > DagRun Failure: Retry, Email, Callbacks > --- > > Key: AIRFLOW-511 > URL: https://issues.apache.org/jira/browse/AIRFLOW-511 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Rob Froetscher >Priority: Minor > > Are there any plans to have retry, email, or callbacks on failure of a DAG > run? Would you guys be open to someone implementing that? Right now > particularly with dagrun_timeout, there is not much insight that the dag > actually stopped. > Pseudocode: > https://github.com/apache/incubator-airflow/compare/master...rfroetscher:dagrun_failure -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (AIRFLOW-1770) Add option to file and hiveconfs in HiveOperator
Ace Haidrey created AIRFLOW-1770: Summary: Add option to file and hiveconfs in HiveOperator Key: AIRFLOW-1770 URL: https://issues.apache.org/jira/browse/AIRFLOW-1770 Project: Apache Airflow Issue Type: Improvement Reporter: Ace Haidrey Assignee: Ace Haidrey The HiveOperator as it currently stands is not flexible enough to accept a hive file and operate on that. You need to read in the contents and pass it and if you do that you need to change the way hiveconfs are in your file to jinja templating. Many teams already have their existing sql/hql files and don't want to convert them to make them as portable and decoupled as possible. To accomplish this all we need to do is add the option to pass a hql_file and hiveconfs to the HiveOperator. We change the code in the execute to throw an error if both a hql_file and an hql statement are passed. If just hql_file the simplest way without changing the code of the hive hook is to just read the content of the hql_file and set it to be the self.hql. The hiveconfs get passed directly to the run_cli method and we can combine them with the already passed in hiveconfs. If we want to make it optional to pass in the context as hiveconfs we can add that too as related to AIRFLOW-788. I've included some simple tests to show it all works how we expect. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (AIRFLOW-788) Context unexpectedly added to hive conf
[ https://issues.apache.org/jira/browse/AIRFLOW-788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16227301#comment-16227301 ] Ace Haidrey edited comment on AIRFLOW-788 at 10/31/17 6:40 PM: --- Should we add an option in the HiveOperator to pass this or not? By default have it not pass it was (Author: ahaidrey): Should we add an option in the HiveOperator to pass this or not? > Context unexpectedly added to hive conf > --- > > Key: AIRFLOW-788 > URL: https://issues.apache.org/jira/browse/AIRFLOW-788 > Project: Apache Airflow > Issue Type: Bug > Components: hive_hooks >Reporter: Bolke de Bruin > Fix For: 1.9.0 > > > If specifying hive_conf to run_cli extra variables are added from the > context, e.g. airflow.ctx.dag.dag_id . > In secured environments this can raise the need for a configuration change as > these variables might not be whitelisted. > Secondly one could regard it as information leakage, as its is added without > the user's consent. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AIRFLOW-788) Context unexpectedly added to hive conf
[ https://issues.apache.org/jira/browse/AIRFLOW-788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16227301#comment-16227301 ] Ace Haidrey commented on AIRFLOW-788: - Should we add an option in the HiveOperator to pass this or not? > Context unexpectedly added to hive conf > --- > > Key: AIRFLOW-788 > URL: https://issues.apache.org/jira/browse/AIRFLOW-788 > Project: Apache Airflow > Issue Type: Bug > Components: hive_hooks >Reporter: Bolke de Bruin > Fix For: 1.9.0 > > > If specifying hive_conf to run_cli extra variables are added from the > context, e.g. airflow.ctx.dag.dag_id . > In secured environments this can raise the need for a configuration change as > these variables might not be whitelisted. > Secondly one could regard it as information leakage, as its is added without > the user's consent. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AIRFLOW-1734) Sqoop Operator contains logic errors & needs options to pass more sqoop options
[ https://issues.apache.org/jira/browse/AIRFLOW-1734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16225379#comment-16225379 ] Ace Haidrey commented on AIRFLOW-1734: -- Awesome, thanks [~bolke] > Sqoop Operator contains logic errors & needs options to pass more sqoop > options > --- > > Key: AIRFLOW-1734 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1734 > Project: Apache Airflow > Issue Type: Bug > Components: contrib >Reporter: Ace Haidrey >Assignee: Ace Haidrey > Labels: patch > Fix For: 1.9.0 > > > After taking a look at the Sqoop Operator I have found a number of errors > with it. I will describe them in depth on the PR I will make, but it includes > assign wrong parameters, not hiding the users raw password in the logs > (though a method to do that had been created in the hook, wrong if > conditions, and the inability to add other sqoop options that would make > sense to have the option to pass those. > I have made some remarks in past commits about it too in the case I'm > misinterpreting but regardless there are definitely errors. > https://github.com/apache/incubator-airflow/pull/2177#pullrequestreview-70046796 > Here is my PR with change by change explanation: > https://github.com/apache/incubator-airflow/pull/2703/files#diff-8e77f042c2e060bbfd60828431a91e9bL131 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (AIRFLOW-1759) Create PostgresTableSensor
Ace Haidrey created AIRFLOW-1759: Summary: Create PostgresTableSensor Key: AIRFLOW-1759 URL: https://issues.apache.org/jira/browse/AIRFLOW-1759 Project: Apache Airflow Issue Type: New Feature Components: operators Reporter: Ace Haidrey Assignee: Ace Haidrey Priority: Minor Like the HivePartitionSensor or sensors of that sort, create a PostgresTableSensor that makes sure a postgres table exists before operating on it. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AIRFLOW-1735) Log files do not show up for unscheduled dags
[
https://issues.apache.org/jira/browse/AIRFLOW-1735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ace Haidrey updated AIRFLOW-1735:
-
Description:
I created a new dag with a schedule of {{@once}} and left it unscheduled. I
ran different tasks a few times from the UI but the logs never showed up.
Instead I'd see messages like the following:
{code}
*** Log file isn't local.
*** Fetching here:
http://al-tools2.savagebeast.com:18700/log/analytics_kochava_import_dag/kopandoraradioandroid1134523b395684da7_request/2017-06-21
*** Failed to fetch log file from worker.
{code}
There is a log file for this task, but it seems to be at a slightly different
path on the machine:
{code}
(airflow)hadoop@al-tools2:~/.virtualenvs/airflow$ ls -lth
./airflow_home/logs/analytics_kochava_import_dag/kopandoraradioandroid1134523b395684da7_request/2017-06-21T00\:00\:00
-rw-rw-rw- 1 hadoop hadoop 5.2K Jun 21 21:57
./airflow_home/logs/analytics_kochava_import_dag/kopandoraradioandroid1134523b395684da7_request/2017-06-21T00:00:00
{code}
And indeed, when I change the URL to
WEB_SERVER/admin/airflow/log?task_id=TASK_ID_request_id=DAG_ID_date=2017-06-21T00:00:00,
I see the actual log:
{code}
*** Reading local log.
[2017-06-21 21:57:20,931] {models.py:172} INFO - Filling up the DagBag from
~/analytics_kochava_import_dag.py
[2017-06-21 21:57:21,307] {analytics_kochava_import_dag.py:139} INFO - Kochava
app IDs
...
[2017-06-21 21:57:26,400] {jobs.py:2083} INFO - Task exited with return code 0
{code}
I've added a screen shot of it below.
*FIX:*
I have noticed in www/view.py the issue is not have the time component 00:00:00
and we can get that quite easily if it doesnt have it. You can see in this PR I
made, but basically if we set
{code}
dttm = dateutil.parser.parse(request.args.get('execution_date'))
execution_date = dttm.isoformat()
{code}
This will fix it.
was:
I created a new dag with a schedule of {{@once}} and left it unscheduled. I
ran different tasks a few times from the UI but the logs never showed up.
Instead I'd see messages like the following:
{code}
*** Log file isn't local.
*** Fetching here:
http://al-tools2.savagebeast.com:18700/log/analytics_kochava_import_dag/kopandoraradioandroid1134523b395684da7_request/2017-06-21
*** Failed to fetch log file from worker.
{code}
There is a log file for this task, but it seems to be at a slightly different
path on the machine:
{code}
(airflow)hadoop@al-tools2:~/.virtualenvs/airflow$ ls -lth
./airflow_home/logs/analytics_kochava_import_dag/kopandoraradioandroid1134523b395684da7_request/2017-06-21T00\:00\:00
-rw-rw-rw- 1 hadoop hadoop 5.2K Jun 21 21:57
./airflow_home/logs/analytics_kochava_import_dag/kopandoraradioandroid1134523b395684da7_request/2017-06-21T00:00:00
{code}
And indeed, when I change the URL to
WEB_SERVER/admin/airflow/log?task_id=TASK_ID_request_id=DAG_ID_date=2017-06-21T00:00:00,
I see the actual log:
{code}
*** Reading local log.
[2017-06-21 21:57:20,931] {models.py:172} INFO - Filling up the DagBag from
~/analytics_kochava_import_dag.py
[2017-06-21 21:57:21,307] {analytics_kochava_import_dag.py:139} INFO - Kochava
app IDs
...
[2017-06-21 21:57:26,400] {jobs.py:2083} INFO - Task exited with return code 0
{code}
I've added a screen shot of it below.
*FIX:*
I have noticed in www/view.py the issue is not have the time component 00:00:00
and we can get that quite easily if it doesnt have it. You can see in this PR I
made.
> Log files do not show up for unscheduled dags
> -
>
> Key: AIRFLOW-1735
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1735
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Ace Haidrey
>Assignee: Ace Haidrey
> Labels: www
> Attachments: Screen Shot 2017-06-21 at 10.53.28 PM.png
>
>
> I created a new dag with a schedule of {{@once}} and left it unscheduled. I
> ran different tasks a few times from the UI but the logs never showed up.
> Instead I'd see messages like the following:
> {code}
> *** Log file isn't local.
> *** Fetching here:
> http://al-tools2.savagebeast.com:18700/log/analytics_kochava_import_dag/kopandoraradioandroid1134523b395684da7_request/2017-06-21
> *** Failed to fetch log file from worker.
> {code}
> There is a log file for this task, but it seems to be at a slightly different
> path on the machine:
> {code}
> (airflow)hadoop@al-tools2:~/.virtualenvs/airflow$ ls -lth
> ./airflow_home/logs/analytics_kochava_import_dag/kopandoraradioandroid1134523b395684da7_request/2017-06-21T00\:00\:00
> -rw-rw-rw- 1 hadoop hadoop 5.2K Jun 21 21:57
> ./airflow_home/logs/analytics_kochava_import_dag/kopandoraradioandroid1134523b395684da7_request/2017-06-21T00:00:00
> {code}
> And indeed, when I change the URL to
>
[jira] [Created] (AIRFLOW-1735) Log files do not show up for unscheduled dags
Ace Haidrey created AIRFLOW-1735:
Summary: Log files do not show up for unscheduled dags
Key: AIRFLOW-1735
URL: https://issues.apache.org/jira/browse/AIRFLOW-1735
Project: Apache Airflow
Issue Type: Bug
Reporter: Ace Haidrey
Assignee: Ace Haidrey
Attachments: Screen Shot 2017-06-21 at 10.53.28 PM.png
I created a new dag with a schedule of {{@once}} and left it unscheduled. I
ran different tasks a few times from the UI but the logs never showed up.
Instead I'd see messages like the following:
{code}
*** Log file isn't local.
*** Fetching here:
http://al-tools2.savagebeast.com:18700/log/analytics_kochava_import_dag/kopandoraradioandroid1134523b395684da7_request/2017-06-21
*** Failed to fetch log file from worker.
{code}
There is a log file for this task, but it seems to be at a slightly different
path on the machine:
{code}
(airflow)hadoop@al-tools2:~/.virtualenvs/airflow$ ls -lth
./airflow_home/logs/analytics_kochava_import_dag/kopandoraradioandroid1134523b395684da7_request/2017-06-21T00\:00\:00
-rw-rw-rw- 1 hadoop hadoop 5.2K Jun 21 21:57
./airflow_home/logs/analytics_kochava_import_dag/kopandoraradioandroid1134523b395684da7_request/2017-06-21T00:00:00
{code}
And indeed, when I change the URL to
WEB_SERVER/admin/airflow/log?task_id=TASK_ID_request_id=DAG_ID_date=2017-06-21T00:00:00,
I see the actual log:
{code}
*** Reading local log.
[2017-06-21 21:57:20,931] {models.py:172} INFO - Filling up the DagBag from
~/analytics_kochava_import_dag.py
[2017-06-21 21:57:21,307] {analytics_kochava_import_dag.py:139} INFO - Kochava
app IDs
...
[2017-06-21 21:57:26,400] {jobs.py:2083} INFO - Task exited with return code 0
{code}
I've added a screen shot of it below.
*FIX:*
I have noticed in www/view.py the issue is not have the time component 00:00:00
and we can get that quite easily if it doesnt have it. You can see in this PR I
made.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (AIRFLOW-1734) Sqoop Operator contains logic errors & needs options to pass more sqoop options
[ https://issues.apache.org/jira/browse/AIRFLOW-1734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ace Haidrey updated AIRFLOW-1734: - Description: After taking a look at the Sqoop Operator I have found a number of errors with it. I will describe them in depth on the PR I will make, but it includes assign wrong parameters, not hiding the users raw password in the logs (though a method to do that had been created in the hook, wrong if conditions, and the inability to add other sqoop options that would make sense to have the option to pass those. I have made some remarks in past commits about it too in the case I'm misinterpreting but regardless there are definitely errors. https://github.com/apache/incubator-airflow/pull/2177#pullrequestreview-70046796 Here is my PR with change by change explanation: https://github.com/apache/incubator-airflow/pull/2703/files#diff-8e77f042c2e060bbfd60828431a91e9bL131 was: After taking a look at the Sqoop Operator I have found a number of errors with it. I will describe them in depth on the PR I will make, but it includes assign wrong parameters, not hiding the users raw password in the logs (though a method to do that had been created in the hook, wrong if conditions, and the inability to add other sqoop options that would make sense to have the option to pass those. I have made some remarks in past commits about it too in the case I'm misinterpreting but regardless there are definitely errors. https://github.com/apache/incubator-airflow/pull/2177#pullrequestreview-70046796 > Sqoop Operator contains logic errors & needs options to pass more sqoop > options > --- > > Key: AIRFLOW-1734 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1734 > Project: Apache Airflow > Issue Type: Bug > Components: contrib >Reporter: Ace Haidrey >Assignee: Ace Haidrey > Labels: patch > > After taking a look at the Sqoop Operator I have found a number of errors > with it. I will describe them in depth on the PR I will make, but it includes > assign wrong parameters, not hiding the users raw password in the logs > (though a method to do that had been created in the hook, wrong if > conditions, and the inability to add other sqoop options that would make > sense to have the option to pass those. > I have made some remarks in past commits about it too in the case I'm > misinterpreting but regardless there are definitely errors. > https://github.com/apache/incubator-airflow/pull/2177#pullrequestreview-70046796 > Here is my PR with change by change explanation: > https://github.com/apache/incubator-airflow/pull/2703/files#diff-8e77f042c2e060bbfd60828431a91e9bL131 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AIRFLOW-1734) Sqoop Operator contains logic errors & needs options to pass more sqoop options
[ https://issues.apache.org/jira/browse/AIRFLOW-1734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ace Haidrey updated AIRFLOW-1734: - Description: After taking a look at the Sqoop Operator I have found a number of errors with it. I will describe them in depth on the PR I will make, but it includes assign wrong parameters, not hiding the users raw password in the logs (though a method to do that had been created in the hook, wrong if conditions, and the inability to add other sqoop options that would make sense to have the option to pass those. I have made some remarks in past commits about it too in the case I'm misinterpreting but regardless there are definitely errors. https://github.com/apache/incubator-airflow/pull/2177#pullrequestreview-70046796 was: After taking a look at the Sqoop Operator I have found a number of errors with it. I will describe them in depth on the PR I will make, but it includes passign wrong parameters, not hiding the users raw password in the logs (though a method to do that had been created in the hook, wrong if conditions, and the inability to add other sqoop options that would make sense to have the option to pass those. I have made some remarks in past commits about it too in the case I'm misinterpreting but regardless there are definitely errors. https://github.com/apache/incubator-airflow/pull/2177#pullrequestreview-70046796 > Sqoop Operator contains logic errors & needs options to pass more sqoop > options > --- > > Key: AIRFLOW-1734 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1734 > Project: Apache Airflow > Issue Type: Bug > Components: contrib >Reporter: Ace Haidrey >Assignee: Ace Haidrey > Labels: patch > > After taking a look at the Sqoop Operator I have found a number of errors > with it. I will describe them in depth on the PR I will make, but it includes > assign wrong parameters, not hiding the users raw password in the logs > (though a method to do that had been created in the hook, wrong if > conditions, and the inability to add other sqoop options that would make > sense to have the option to pass those. > I have made some remarks in past commits about it too in the case I'm > misinterpreting but regardless there are definitely errors. > https://github.com/apache/incubator-airflow/pull/2177#pullrequestreview-70046796 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (AIRFLOW-1734) Sqoop Operator contains logic errors & needs options to pass more sqoop options
Ace Haidrey created AIRFLOW-1734: Summary: Sqoop Operator contains logic errors & needs options to pass more sqoop options Key: AIRFLOW-1734 URL: https://issues.apache.org/jira/browse/AIRFLOW-1734 Project: Apache Airflow Issue Type: Bug Components: contrib Reporter: Ace Haidrey Assignee: Ace Haidrey After taking a look at the Sqoop Operator I have found a number of errors with it. I will describe them in depth on the PR I will make, but it includes passign wrong parameters, not hiding the users raw password in the logs (though a method to do that had been created in the hook, wrong if conditions, and the inability to add other sqoop options that would make sense to have the option to pass those. I have made some remarks in past commits about it too in the case I'm misinterpreting but regardless there are definitely errors. https://github.com/apache/incubator-airflow/pull/2177#pullrequestreview-70046796 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (AIRFLOW-1574) Add 'to' field to template fields in EmailOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-1574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ace Haidrey closed AIRFLOW-1574. Resolution: Fixed > Add 'to' field to template fields in EmailOperator > -- > > Key: AIRFLOW-1574 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1574 > Project: Apache Airflow > Issue Type: Bug >Reporter: Ace Haidrey >Assignee: Ace Haidrey > > The to field may sometimes want to be to be template-able when you have a DAG > that is using XCOM to find the user to send the information to (i.e. we have > a form that a user submits and based on the ldap user we send this specific > user the information). It's a rather easy fix to add the 'to' user to the > template-able options. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (AIRFLOW-1574) Add 'to' field to template fields in EmailOperator
Ace Haidrey created AIRFLOW-1574: Summary: Add 'to' field to template fields in EmailOperator Key: AIRFLOW-1574 URL: https://issues.apache.org/jira/browse/AIRFLOW-1574 Project: Apache Airflow Issue Type: Bug Reporter: Ace Haidrey Assignee: Ace Haidrey The to field may sometimes want to be to be template-able when you have a DAG that is using XCOM to find the user to send the information to (i.e. we have a form that a user submits and based on the ldap user we send this specific user the information). It's a rather easy fix to add the 'to' user to the template-able options. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (AIRFLOW-1541) Add channel to template fields of SlackAPIPostOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-1541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ace Haidrey resolved AIRFLOW-1541. -- Resolution: Fixed merged in > Add channel to template fields of SlackAPIPostOperator > -- > > Key: AIRFLOW-1541 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1541 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Ace Haidrey >Assignee: Ace Haidrey > Labels: patch > > This is important for us because we use xcom to find the slack user name and > want to send that to the SlackAPIPostOperator and template the channel name > with an xcom pull, which we can't do currently. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AIRFLOW-1508) Skipped state not part of State.task_states
[
https://issues.apache.org/jira/browse/AIRFLOW-1508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16146311#comment-16146311
]
Ace Haidrey commented on AIRFLOW-1508:
--
Agreed that currently it is not a big deal, but I still do think it's a
potential bug source in the future, because SKIPPED is a task state. I was
wondering why would it be bad if it was added in the payload? Seems like it's
safe and doesn't hurt/ is more complete (just my thoughts).
Yes our use case is at Pandora, we've built some infrastructure around Airflow
and a plugin that is a BluePrint to collect stats on our instances (we deploy
an instance per team), and then use a tool called Consul + Prometheus to
monitor. So we want the metrics of all the different task states.
It's not huge but currently have to have our looping like:
{code}
for task_state_type in State.task_states + (State.SKIPPED, ):
{code}
And I'm just hoping if future states get added we'd also pick up (being a part
of task_states) on those rather than having to go see the code ourselves.
Also appreciate the response [~erod] :)
> Skipped state not part of State.task_states
> ---
>
> Key: AIRFLOW-1508
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1508
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Ace Haidrey
>Assignee: Ace Haidrey
>
> In the airflow.state module,
> [State.task_state|https://github.com/apache/incubator-airflow/blob/master/airflow/utils/state.py#L44]
> doesn't include the {{SKIPPED}} state even though the {{TaskInstance}}
> object has it. I was wondering if this was on purpose or a bug. I would think
> it should be part of the task_state object since that makes sense and will
> help some of my workflows to not have to add this in manually.
> I'm not sure who the appropriate person to ask is so thinking I'll tag some
> people and get some feedback (hopefully that's okay)..
> CC [~criccomini] [~bolke] [~allisonwang]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (AIRFLOW-1541) Add channel to template fields of SlackAPIPostOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-1541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ace Haidrey updated AIRFLOW-1541: - Issue Type: Improvement (was: Bug) > Add channel to template fields of SlackAPIPostOperator > -- > > Key: AIRFLOW-1541 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1541 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Ace Haidrey >Assignee: Ace Haidrey > Labels: patch > > This is important for us because we use xcom to find the slack user name and > want to send that to the SlackAPIPostOperator and template the channel name > with an xcom pull, which we can't do currently. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (AIRFLOW-1541) Add channel to template fields of SlackAPIPostOperator
Ace Haidrey created AIRFLOW-1541: Summary: Add channel to template fields of SlackAPIPostOperator Key: AIRFLOW-1541 URL: https://issues.apache.org/jira/browse/AIRFLOW-1541 Project: Apache Airflow Issue Type: Bug Reporter: Ace Haidrey Assignee: Ace Haidrey This is important for us because we use xcom to find the slack user name and want to send that to the SlackAPIPostOperator and template the channel name with an xcom pull, which we can't do currently. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AIRFLOW-1508) Skipped state not part of State.task_states
[
https://issues.apache.org/jira/browse/AIRFLOW-1508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ace Haidrey updated AIRFLOW-1508:
-
Description:
In the airflow.state module,
[State.task_state|https://github.com/apache/incubator-airflow/blob/master/airflow/utils/state.py#L44]
doesn't include the {{SKIPPED}} state even though the {{TaskInstance}} object
has it. I was wondering if this was on purpose or a bug. I would think it
should be part of the task_state object since that makes sense and will help
some of my workflows to not have to add this in manually.
I'm not sure who the appropriate person to ask is so thinking I'll tag some
people and get some feedback (hopefully that's okay)..
CC [~criccomini] [~bolke] [~allisonwang]
was:
In the airflow.state module,
[State.task_state|https://github.com/apache/incubator-airflow/blob/master/airflow/utils/state.py#L44]
doesn't include the {{SKIPPED}} state even though the {{TaskInstance}} object
has it. I was wondering if this was on purpose or a bug. I would think it
should be part of the task_state object since that makes sense and will help
some of my workflows to not have to add this in manually.
I'm not sure who the appropriate person to ask is so thinking I'll tag some
people and get some feedback (hopefully that's okay)..
CC [~criccomini] [~bolke]
> Skipped state not part of State.task_states
> ---
>
> Key: AIRFLOW-1508
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1508
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Ace Haidrey
>Assignee: Ace Haidrey
>
> In the airflow.state module,
> [State.task_state|https://github.com/apache/incubator-airflow/blob/master/airflow/utils/state.py#L44]
> doesn't include the {{SKIPPED}} state even though the {{TaskInstance}}
> object has it. I was wondering if this was on purpose or a bug. I would think
> it should be part of the task_state object since that makes sense and will
> help some of my workflows to not have to add this in manually.
> I'm not sure who the appropriate person to ask is so thinking I'll tag some
> people and get some feedback (hopefully that's okay)..
> CC [~criccomini] [~bolke] [~allisonwang]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AIRFLOW-1508) Skipped state not part of State.task_states
[
https://issues.apache.org/jira/browse/AIRFLOW-1508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126620#comment-16126620
]
Ace Haidrey commented on AIRFLOW-1508:
--
https://github.com/apache/incubator-airflow/pull/2519
> Skipped state not part of State.task_states
> ---
>
> Key: AIRFLOW-1508
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1508
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Ace Haidrey
>Assignee: Ace Haidrey
>
> In the airflow.state module,
> [State.task_state|https://github.com/apache/incubator-airflow/blob/master/airflow/utils/state.py#L44]
> doesn't include the {{SKIPPED}} state even though the {{TaskInstance}}
> object has it. I was wondering if this was on purpose or a bug. I would think
> it should be part of the task_state object since that makes sense and will
> help some of my workflows to not have to add this in manually.
> I'm not sure who the appropriate person to ask is so thinking I'll tag some
> people and get some feedback (hopefully that's okay)..
> CC [~criccomini] [~bolke]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AIRFLOW-1508) Skipped state not part of State.task_states
[
https://issues.apache.org/jira/browse/AIRFLOW-1508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126607#comment-16126607
]
Ace Haidrey commented on AIRFLOW-1508:
--
After some looking into, I checked the usages of the {{State.task_states}} and
it's only used here:
https://github.com/apache/incubator-airflow/blob/master/airflow/www/views.py#L560.
This just for task stats, so it won't affect anything else in the code base
and just add another metric for skipped. I'll create a PR, but please let me
know if this was intentionally left out.
> Skipped state not part of State.task_states
> ---
>
> Key: AIRFLOW-1508
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1508
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Ace Haidrey
>Assignee: Ace Haidrey
>
> In the airflow.state module,
> [State.task_state|https://github.com/apache/incubator-airflow/blob/master/airflow/utils/state.py#L44]
> doesn't include the {{SKIPPED}} state even though the {{TaskInstance}}
> object has it. I was wondering if this was on purpose or a bug. I would think
> it should be part of the task_state object since that makes sense and will
> help some of my workflows to not have to add this in manually.
> I'm not sure who the appropriate person to ask is so thinking I'll tag some
> people and get some feedback (hopefully that's okay)..
> CC [~criccomini] [~bolke]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (AIRFLOW-1508) Skipped state not part of State.task_states
[
https://issues.apache.org/jira/browse/AIRFLOW-1508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ace Haidrey reassigned AIRFLOW-1508:
Assignee: Ace Haidrey
> Skipped state not part of State.task_states
> ---
>
> Key: AIRFLOW-1508
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1508
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Ace Haidrey
>Assignee: Ace Haidrey
>
> In the airflow.state module,
> [State.task_state|https://github.com/apache/incubator-airflow/blob/master/airflow/utils/state.py#L44]
> doesn't include the {{SKIPPED}} state even though the {{TaskInstance}}
> object has it. I was wondering if this was on purpose or a bug. I would think
> it should be part of the task_state object since that makes sense and will
> help some of my workflows to not have to add this in manually.
> I'm not sure who the appropriate person to ask is so thinking I'll tag some
> people and get some feedback (hopefully that's okay)..
> CC [~criccomini] [~bolke]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (AIRFLOW-1508) Skipped state not part of State.task_states
[
https://issues.apache.org/jira/browse/AIRFLOW-1508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ace Haidrey updated AIRFLOW-1508:
-
Description:
In the airflow.state module,
[State.task_state|https://github.com/apache/incubator-airflow/blob/master/airflow/utils/state.py#L44]
doesn't include the {{SKIPPED}} state even though the {{TaskInstance}} object
has it. I was wondering if this was on purpose or a bug. I would think it
should be part of the task_state object since that makes sense and will help
some of my workflows to not have to add this in manually.
I'm not sure who the appropriate person to ask is so thinking I'll tag some
people and get some feedback (hopefully that's okay)..
CC [~criccomini] [~bolke]
was:
In the airflow.state module,
[State.task_state|https://github.com/apache/incubator-airflow/blob/master/airflow/utils/state.py#L44]
doesn't include the {{SKIPPED}} state even though the {{TaskInstance}} object
has it. I was wondering if this was on purpose or a bug. I would think it
should be part of the task_state object since that makes sense and will help
some of my workflows to not have to add this in manually.
I'm not sure who the appropriate person to ask is so thinking I'll tag some
people and get some feedback (hopefully that's okay)..
> Skipped state not part of State.task_states
> ---
>
> Key: AIRFLOW-1508
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1508
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Ace Haidrey
>
> In the airflow.state module,
> [State.task_state|https://github.com/apache/incubator-airflow/blob/master/airflow/utils/state.py#L44]
> doesn't include the {{SKIPPED}} state even though the {{TaskInstance}}
> object has it. I was wondering if this was on purpose or a bug. I would think
> it should be part of the task_state object since that makes sense and will
> help some of my workflows to not have to add this in manually.
> I'm not sure who the appropriate person to ask is so thinking I'll tag some
> people and get some feedback (hopefully that's okay)..
> CC [~criccomini] [~bolke]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (AIRFLOW-1508) Skipped state not part of State.task_states
Ace Haidrey created AIRFLOW-1508:
Summary: Skipped state not part of State.task_states
Key: AIRFLOW-1508
URL: https://issues.apache.org/jira/browse/AIRFLOW-1508
Project: Apache Airflow
Issue Type: Bug
Reporter: Ace Haidrey
In the airflow.state module,
[State.task_state|https://github.com/apache/incubator-airflow/blob/master/airflow/utils/state.py#L44]
doesn't include the {{SKIPPED}} state even though the {{TaskInstance}} object
has it. I was wondering if this was on purpose or a bug. I would think it
should be part of the task_state object since that makes sense and will help
some of my workflows to not have to add this in manually.
I'm not sure who the appropriate person to ask is so thinking I'll tag some
people and get some feedback (hopefully that's okay)..
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AIRFLOW-1239) base_task_runner logging with unicode values on Py27
[
https://issues.apache.org/jira/browse/AIRFLOW-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126502#comment-16126502
]
Ace Haidrey commented on AIRFLOW-1239:
--
This JIRA ticket can now be closed [~deckmon]
> base_task_runner logging with unicode values on Py27
>
>
> Key: AIRFLOW-1239
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1239
> Project: Apache Airflow
> Issue Type: Bug
> Components: worker
>Reporter: yanghanbing
>
> Exception in thread Thread-1:
> Traceback (most recent call last):
> File "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
> self.run()
> File "/usr/lib64/python2.7/threading.py", line 764, in run
> self.__target(*self.__args, **self.__kwargs)
> File
> "/opt/cloudera-manager/cm-5.5.1/run/cloudera-scm-server/airflow/env/lib/python2.7/site-packages/airflow/task_runner/base_task_runner.py",
> line 95, in _read_task_logs
> self.logger.info('Subtask: {}'.format(line.rstrip('\n')))
> UnicodeEncodeError: 'ascii' codec can't encode characters in position
> 107-108: ordinal not in range(128)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (AIRFLOW-1397) Airflow 1.8.1 - No data displays in Last Run Column in Airflow UI
[ https://issues.apache.org/jira/browse/AIRFLOW-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ace Haidrey reassigned AIRFLOW-1397: Assignee: user_airflow (was: Ace Haidrey) > Airflow 1.8.1 - No data displays in Last Run Column in Airflow UI > - > > Key: AIRFLOW-1397 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1397 > Project: Apache Airflow > Issue Type: Bug > Components: DAG, ui >Affects Versions: 1.8.1 >Reporter: user_airflow >Assignee: user_airflow >Priority: Critical > > Recently upgraded Airflow version from 1.8.0 to 1.8.1. After upgrading, the > Last Run column in Airflow UI started showing as Blank for all the existing > dags. > Created a pr for this bug: > https://github.com/apache/incubator-airflow/pull/2430 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (AIRFLOW-1397) Airflow 1.8.1 - No data displays in Last Run Column in Airflow UI
[ https://issues.apache.org/jira/browse/AIRFLOW-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ace Haidrey reassigned AIRFLOW-1397: Assignee: Ace Haidrey (was: user_airflow) > Airflow 1.8.1 - No data displays in Last Run Column in Airflow UI > - > > Key: AIRFLOW-1397 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1397 > Project: Apache Airflow > Issue Type: Bug > Components: DAG, ui >Affects Versions: 1.8.1 >Reporter: user_airflow >Assignee: Ace Haidrey >Priority: Critical > > Recently upgraded Airflow version from 1.8.0 to 1.8.1. After upgrading, the > Last Run column in Airflow UI started showing as Blank for all the existing > dags. > Created a pr for this bug: > https://github.com/apache/incubator-airflow/pull/2430 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AIRFLOW-1397) Airflow 1.8.1 - No data displays in Last Run Column in Airflow UI
[ https://issues.apache.org/jira/browse/AIRFLOW-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122320#comment-16122320 ] Ace Haidrey commented on AIRFLOW-1397: -- this ticket can be closed! > Airflow 1.8.1 - No data displays in Last Run Column in Airflow UI > - > > Key: AIRFLOW-1397 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1397 > Project: Apache Airflow > Issue Type: Bug > Components: DAG, ui >Affects Versions: 1.8.1 >Reporter: user_airflow >Assignee: user_airflow >Priority: Critical > > Recently upgraded Airflow version from 1.8.0 to 1.8.1. After upgrading, the > Last Run column in Airflow UI started showing as Blank for all the existing > dags. > Created a pr for this bug: > https://github.com/apache/incubator-airflow/pull/2430 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AIRFLOW-774) dagbag_size/collect_dags/dagbag_import_errors stats incorrect
[ https://issues.apache.org/jira/browse/AIRFLOW-774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16111850#comment-16111850 ] Ace Haidrey commented on AIRFLOW-774: - Got it. That's why I'm not seeing the errors that you saw, since my version is 1.8.1. But I may try taking a stab at correcting these stats.. We are using [statsd_exporter|https://github.com/prometheus/statsd_exporter] project with prometheus to monitor these instances and another issue is the labels for the stats in Airflow have underscores vs dots to split up, but they need to be dot separated so we will try changing that as well. The statsd_exporter can be configured to translate specific dot-separated StatsD metrics into labeled Prometheus metrics via a simple mapping language. That's an aside but just giving a background of what we're trying to accomplish! > dagbag_size/collect_dags/dagbag_import_errors stats incorrect > - > > Key: AIRFLOW-774 > URL: https://issues.apache.org/jira/browse/AIRFLOW-774 > Project: Apache Airflow > Issue Type: Bug > Components: logging >Reporter: Dan Davydov > > After the multiprocessor change was made (dag folders are processed in > parallel), the number of dags reported by airflow is for each of these > subprocesses which is inaccurate, and potentially orders of magnitude less > than the actual number of dags. These individual processes stats should be > aggregated. The collect_dags/dagbag_import_errors stats should also be fixed > (time it takes to parse the dags). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (AIRFLOW-1397) Airflow 1.8.1 - No data displays in Last Run Column in Airflow UI
[ https://issues.apache.org/jira/browse/AIRFLOW-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ace Haidrey reassigned AIRFLOW-1397: Assignee: user_airflow I think once you squash your commits in that PR it can finally be merged! > Airflow 1.8.1 - No data displays in Last Run Column in Airflow UI > - > > Key: AIRFLOW-1397 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1397 > Project: Apache Airflow > Issue Type: Bug > Components: DAG, ui >Affects Versions: 1.8.1 >Reporter: user_airflow >Assignee: user_airflow >Priority: Critical > > Recently upgraded Airflow version from 1.8.0 to 1.8.1. After upgrading, the > Last Run column in Airflow UI started showing as Blank for all the existing > dags. > Created a pr for this bug: > https://github.com/apache/incubator-airflow/pull/2430 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AIRFLOW-774) dagbag_size/collect_dags/dagbag_import_errors stats incorrect
[ https://issues.apache.org/jira/browse/AIRFLOW-774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16111775#comment-16111775 ] Ace Haidrey commented on AIRFLOW-774: - Does anyone have an idea of how involved a fix to this would be, or the way to go about it. Currently our team is running to the same issues of having errors regarding stats collection. Counts are off, and more. > dagbag_size/collect_dags/dagbag_import_errors stats incorrect > - > > Key: AIRFLOW-774 > URL: https://issues.apache.org/jira/browse/AIRFLOW-774 > Project: Apache Airflow > Issue Type: Bug > Components: logging >Reporter: Dan Davydov > > After the multiprocessor change was made (dag folders are processed in > parallel), the number of dags reported by airflow is for each of these > subprocesses which is inaccurate, and potentially orders of magnitude less > than the actual number of dags. These individual processes stats should be > aggregated. The collect_dags/dagbag_import_errors stats should also be fixed > (time it takes to parse the dags). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (AIRFLOW-1445) Change HivePartitionSensor UI color to be lighter so text is visible
[ https://issues.apache.org/jira/browse/AIRFLOW-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ace Haidrey resolved AIRFLOW-1445. -- Resolution: Fixed Changes were merged > Change HivePartitionSensor UI color to be lighter so text is visible > > > Key: AIRFLOW-1445 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1445 > Project: Apache Airflow > Issue Type: Improvement > Environment: Not necessary for this ticket >Reporter: Ace Haidrey >Assignee: Ace Haidrey >Priority: Minor > Labels: easyfix > Attachments: Screen Shot 2017-07-23 at 6.19.24 PM.png, Screen Shot > 2017-07-23 at 6.30.06 PM.png, Screen Shot 2017-07-29 at 1.55.10 AM.png > > Original Estimate: 0.5h > Remaining Estimate: 0.5h > > The current shade for the hive partition sensor is too difficult to read the > text in the UI, and should be have a lighter shade. I propose using the > shade: #5b5f8e. > It's a very minor change but the screenshots show the added readability. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (AIRFLOW-1239) base_task_runner logging with unicode values on Py27
[
https://issues.apache.org/jira/browse/AIRFLOW-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106097#comment-16106097
]
Ace Haidrey edited comment on AIRFLOW-1239 at 7/29/17 11:02 AM:
Added the below to pr: https://github.com/apache/incubator-airflow/pull/2318
Agree with @AllisonWang comments. I am having these same error messages looking
like the following:
{code}
[2017-07-28 23:03:03,358] {driver.py:120} INFO - Generating grammar tables from
/usr/lib/python2.7/lib2to3/PatternGrammar.txt
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File
"/home/ahaidrey/.virtualenvs/airflow/local/lib/python2.7/site-packages/airflow/task_runner/base_task_runner.py",
line 95, in _read_task_logs
self.logger.info('Subtask: {}'.format(line.rstrip('\n')))
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2018' in position
66: ordinal not in range(128)
{code}
But after adding the line {{from __future__ import unicode_literals}} (like
Allison's suggestion) it goes away and I have something like the following in
my worker logs:
{code}
[2017-07-29 03:57:05,707] {__init__.py:57} INFO - Using executor CeleryExecutor
[2017-07-29 03:57:05,833] {driver.py:120} INFO - Generating grammar tables from
/usr/lib/python2.7/lib2to3/Grammar.txt
[2017-07-29 03:57:05,879] {driver.py:120} INFO - Generating grammar tables from
/usr/lib/python2.7/lib2to3/PatternGrammar.txt
{code}
I read this [SO
post|https://stackoverflow.com/questions/3235386/python-using-format-on-a-unicode-escaped-string]
to get that solution.
was (Author: ahaidrey):
Added the below to pr: https://github.com/apache/incubator-airflow/pull/2318
Agree with @AllisonWang comments. I am having these same error messages looking
like the following:
{code}
[2017-07-28 23:03:03,358] {driver.py:120} INFO - Generating grammar tables from
/usr/lib/python2.7/lib2to3/PatternGrammar.txt
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File
"/home/ahaidrey/.virtualenvs/airflow/local/lib/python2.7/site-packages/airflow/task_runner/base_task_runner.py",
line 95, in _read_task_logs
self.logger.info('Subtask: {}'.format(line.rstrip('\n')))
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2018' in position
66: ordinal not in range(128)
{code}
But after adding the line `from __future__ import unicode_literals` (like
Allison's suggestion) it goes away and I have something like the following in
my worker logs:
{code}
[2017-07-29 03:57:05,707] {__init__.py:57} INFO - Using executor CeleryExecutor
[2017-07-29 03:57:05,833] {driver.py:120} INFO - Generating grammar tables from
/usr/lib/python2.7/lib2to3/Grammar.txt
[2017-07-29 03:57:05,879] {driver.py:120} INFO - Generating grammar tables from
/usr/lib/python2.7/lib2to3/PatternGrammar.txt
{code}
I read this [SO
post|https://stackoverflow.com/questions/3235386/python-using-format-on-a-unicode-escaped-string]
to get that solution.
> base_task_runner logging with unicode values on Py27
>
>
> Key: AIRFLOW-1239
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1239
> Project: Apache Airflow
> Issue Type: Bug
> Components: worker
>Reporter: yanghanbing
>
> Exception in thread Thread-1:
> Traceback (most recent call last):
> File "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
> self.run()
> File "/usr/lib64/python2.7/threading.py", line 764, in run
> self.__target(*self.__args, **self.__kwargs)
> File
> "/opt/cloudera-manager/cm-5.5.1/run/cloudera-scm-server/airflow/env/lib/python2.7/site-packages/airflow/task_runner/base_task_runner.py",
> line 95, in _read_task_logs
> self.logger.info('Subtask: {}'.format(line.rstrip('\n')))
> UnicodeEncodeError: 'ascii' codec can't encode characters in position
> 107-108: ordinal not in range(128)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AIRFLOW-1239) base_task_runner logging with unicode values on Py27
[
https://issues.apache.org/jira/browse/AIRFLOW-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106097#comment-16106097
]
Ace Haidrey commented on AIRFLOW-1239:
--
Added the below to pr: https://github.com/apache/incubator-airflow/pull/2318
Agree with @AllisonWang comments. I am having these same error messages looking
like the following:
{code}
[2017-07-28 23:03:03,358] {driver.py:120} INFO - Generating grammar tables from
/usr/lib/python2.7/lib2to3/PatternGrammar.txt
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File
"/home/ahaidrey/.virtualenvs/airflow/local/lib/python2.7/site-packages/airflow/task_runner/base_task_runner.py",
line 95, in _read_task_logs
self.logger.info('Subtask: {}'.format(line.rstrip('\n')))
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2018' in position
66: ordinal not in range(128)
{code}
But after adding the line `from __future__ import unicode_literals` (like
Allison's suggestion) it goes away and I have something like the following in
my worker logs:
{code}
[2017-07-29 03:57:05,707] {__init__.py:57} INFO - Using executor CeleryExecutor
[2017-07-29 03:57:05,833] {driver.py:120} INFO - Generating grammar tables from
/usr/lib/python2.7/lib2to3/Grammar.txt
[2017-07-29 03:57:05,879] {driver.py:120} INFO - Generating grammar tables from
/usr/lib/python2.7/lib2to3/PatternGrammar.txt
{code}
I read this [SO
post|https://stackoverflow.com/questions/3235386/python-using-format-on-a-unicode-escaped-string]
to get that solution.
> base_task_runner logging with unicode values on Py27
>
>
> Key: AIRFLOW-1239
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1239
> Project: Apache Airflow
> Issue Type: Bug
> Components: worker
>Reporter: yanghanbing
>
> Exception in thread Thread-1:
> Traceback (most recent call last):
> File "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
> self.run()
> File "/usr/lib64/python2.7/threading.py", line 764, in run
> self.__target(*self.__args, **self.__kwargs)
> File
> "/opt/cloudera-manager/cm-5.5.1/run/cloudera-scm-server/airflow/env/lib/python2.7/site-packages/airflow/task_runner/base_task_runner.py",
> line 95, in _read_task_logs
> self.logger.info('Subtask: {}'.format(line.rstrip('\n')))
> UnicodeEncodeError: 'ascii' codec can't encode characters in position
> 107-108: ordinal not in range(128)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (AIRFLOW-1445) Change HivePartitionSensor UI color to be lighter so text is visible
[ https://issues.apache.org/jira/browse/AIRFLOW-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106082#comment-16106082 ] Ace Haidrey edited comment on AIRFLOW-1445 at 7/29/17 9:41 AM: --- Made the shade even lighter given [~allisonwang]'s suggestions in the PR. [^Screen Shot 2017-07-29 at 1.55.10 AM.png] was (Author: ahaidrey): Made the shade even lighter given [~allisonwang]'s suggestions in the PR. > Change HivePartitionSensor UI color to be lighter so text is visible > > > Key: AIRFLOW-1445 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1445 > Project: Apache Airflow > Issue Type: Improvement > Environment: Not necessary for this ticket >Reporter: Ace Haidrey >Assignee: Ace Haidrey >Priority: Minor > Labels: easyfix > Attachments: Screen Shot 2017-07-23 at 6.19.24 PM.png, Screen Shot > 2017-07-23 at 6.30.06 PM.png, Screen Shot 2017-07-29 at 1.55.10 AM.png > > Original Estimate: 0.5h > Remaining Estimate: 0.5h > > The current shade for the hive partition sensor is too difficult to read the > text in the UI, and should be have a lighter shade. I propose using the > shade: #5b5f8e. > It's a very minor change but the screenshots show the added readability. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AIRFLOW-1445) Change HivePartitionSensor UI color to be lighter so text is visible
[ https://issues.apache.org/jira/browse/AIRFLOW-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ace Haidrey updated AIRFLOW-1445: - Attachment: Screen Shot 2017-07-29 at 1.55.10 AM.png > Change HivePartitionSensor UI color to be lighter so text is visible > > > Key: AIRFLOW-1445 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1445 > Project: Apache Airflow > Issue Type: Improvement > Environment: Not necessary for this ticket >Reporter: Ace Haidrey >Assignee: Ace Haidrey >Priority: Minor > Labels: easyfix > Attachments: Screen Shot 2017-07-23 at 6.19.24 PM.png, Screen Shot > 2017-07-23 at 6.30.06 PM.png, Screen Shot 2017-07-29 at 1.55.10 AM.png > > Original Estimate: 0.5h > Remaining Estimate: 0.5h > > The current shade for the hive partition sensor is too difficult to read the > text in the UI, and should be have a lighter shade. I propose using the > shade: #5b5f8e. > It's a very minor change but the screenshots show the added readability. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AIRFLOW-1445) Change HivePartitionSensor UI color to be lighter so text is visible
[ https://issues.apache.org/jira/browse/AIRFLOW-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106082#comment-16106082 ] Ace Haidrey commented on AIRFLOW-1445: -- Made the shade even lighter given [~allisonwang]'s suggestions in the PR. > Change HivePartitionSensor UI color to be lighter so text is visible > > > Key: AIRFLOW-1445 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1445 > Project: Apache Airflow > Issue Type: Improvement > Environment: Not necessary for this ticket >Reporter: Ace Haidrey >Assignee: Ace Haidrey >Priority: Minor > Labels: easyfix > Attachments: Screen Shot 2017-07-23 at 6.19.24 PM.png, Screen Shot > 2017-07-23 at 6.30.06 PM.png > > Original Estimate: 0.5h > Remaining Estimate: 0.5h > > The current shade for the hive partition sensor is too difficult to read the > text in the UI, and should be have a lighter shade. I propose using the > shade: #5b5f8e. > It's a very minor change but the screenshots show the added readability. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (AIRFLOW-1445) Change HivePartitionSensor UI color to be lighter so text is visible
[ https://issues.apache.org/jira/browse/AIRFLOW-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16097870#comment-16097870 ] Ace Haidrey edited comment on AIRFLOW-1445 at 7/24/17 6:56 PM: --- https://github.com/apache/incubator-airflow/pull/2476 was (Author: ahaidrey): https://github.com/apache/incubator-airflow/pull/2473 > Change HivePartitionSensor UI color to be lighter so text is visible > > > Key: AIRFLOW-1445 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1445 > Project: Apache Airflow > Issue Type: Improvement > Environment: Not necessary for this ticket >Reporter: Ace Haidrey >Assignee: Ace Haidrey >Priority: Minor > Labels: easyfix > Attachments: Screen Shot 2017-07-23 at 6.19.24 PM.png, Screen Shot > 2017-07-23 at 6.30.06 PM.png > > Original Estimate: 0.5h > Remaining Estimate: 0.5h > > The current shade for the hive partition sensor is too difficult to read the > text in the UI, and should be have a lighter shade. I propose using the > shade: #5b5f8e. > It's a very minor change but the screenshots show the added readability. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AIRFLOW-1445) Change HivePartitionSensor UI color to be lighter so text is visible
[ https://issues.apache.org/jira/browse/AIRFLOW-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16097870#comment-16097870 ] Ace Haidrey commented on AIRFLOW-1445: -- https://github.com/apache/incubator-airflow/pull/2473 > Change HivePartitionSensor UI color to be lighter so text is visible > > > Key: AIRFLOW-1445 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1445 > Project: Apache Airflow > Issue Type: Improvement > Environment: Not necessary for this ticket >Reporter: Ace Haidrey >Assignee: Ace Haidrey >Priority: Minor > Labels: easyfix > Attachments: Screen Shot 2017-07-23 at 6.19.24 PM.png, Screen Shot > 2017-07-23 at 6.30.06 PM.png > > Original Estimate: 0.5h > Remaining Estimate: 0.5h > > The current shade for the hive partition sensor is too difficult to read the > text in the UI, and should be have a lighter shade. I propose using the > shade: #5b5f8e. > It's a very minor change but the screenshots show the added readability. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Work started] (AIRFLOW-1445) Change HivePartitionSensor UI color to be lighter so text is visible
[ https://issues.apache.org/jira/browse/AIRFLOW-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-1445 started by Ace Haidrey. > Change HivePartitionSensor UI color to be lighter so text is visible > > > Key: AIRFLOW-1445 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1445 > Project: Apache Airflow > Issue Type: Improvement > Environment: Not necessary for this ticket >Reporter: Ace Haidrey >Assignee: Ace Haidrey >Priority: Minor > Labels: easyfix > Attachments: Screen Shot 2017-07-23 at 6.19.24 PM.png, Screen Shot > 2017-07-23 at 6.30.06 PM.png > > Original Estimate: 0.5h > Remaining Estimate: 0.5h > > The current shade for the hive partition sensor is too difficult to read the > text in the UI, and should be have a lighter shade. I propose using the > shade: #5b5f8e. > It's a very minor change but the screenshots show the added readability. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (AIRFLOW-1445) Change HivePartitionSensor UI color to be lighter so text is visible
Ace Haidrey created AIRFLOW-1445: Summary: Change HivePartitionSensor UI color to be lighter so text is visible Key: AIRFLOW-1445 URL: https://issues.apache.org/jira/browse/AIRFLOW-1445 Project: Apache Airflow Issue Type: Improvement Environment: Not necessary for this ticket Reporter: Ace Haidrey Assignee: Ace Haidrey Priority: Minor Attachments: Screen Shot 2017-07-23 at 6.19.24 PM.png, Screen Shot 2017-07-23 at 6.30.06 PM.png The current shade for the hive partition sensor is too difficult to read the text in the UI, and should be have a lighter shade. I propose using the shade: #5b5f8e. It's a very minor change but the screenshots show the added readability. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
