Re: Airflow 1.8.0 Release Candidate 1

2017-02-02 Thread Boris Tyukin
Bolke, you are our hero! I am sure you put a lot of your time to make it happen On Thu, Feb 2, 2017 at 2:50 PM, Bolke de Bruin wrote: > Hi All, > > I have made the (first) RELEASE CANDIDATE of Airflow 1.8.0 available at: >

Re: Airflow 1.8.0 Release Candidate 1

2017-02-02 Thread Boris Tyukin
LOL awesome! On Thu, Feb 2, 2017 at 4:00 PM, Maxime Beauchemin < maximebeauche...@gmail.com> wrote: > The Apache mailing doesn't support images so here's a link: > > http://i.imgur.com/DUkpjZu.png > ​ > > On Thu, Feb 2, 2017 at 12:52 PM, Boris Tyukin <bo...@boristyuki

Re: Airflow Meetup @ Paypal (San Jose)

2017-01-31 Thread Boris Tyukin
; > > > > Thanks > > > > > Jayesh > > > > > > > > > > On Wed, Jan 25, 2017 at 4:52 PM, Russell Jurney < > > > > russell.jur...@gmail.com> > > > > > wrote: > > > > > > > &g

Re: Airflow 1.8.0 BETA 4

2017-01-27 Thread Boris Tyukin
was requirement to MySQL database version changes with 1.8.0? I run mysql 14.14 and it worked fine with 1.7. I just installed 1.8.4 beta 4 and got this error below when i ran airflow upgradedb command File "/usr/local/lib/python2.7/site-packages/MySQLdb/cursors.py", line 205, in execute

Re: Airflow 1.8.0 BETA 4

2017-01-27 Thread Boris Tyukin
is not cool. epel repo only has 5.1.73 as latest version and CDH distro includes 5.1.73 as well. Some companies might not allow an upgrade of mysql unless it is in one of the official linux repos. On Fri, Jan 27, 2017 at 2:14 PM, Boris Tyukin <bo...@boristyukin.com> wrote: > was re

Re: Airflow Meetup in NYC @ Blue Apron

2017-01-30 Thread Boris Tyukin
i hope you guys can share presentation slides at least for all of us who are not in NYC On Mon, Jan 30, 2017 at 7:33 PM, Maxime Beauchemin < maximebeauche...@gmail.com> wrote: > I'd love to watch, is there any way you guys can livecast or share a video > after the event? > > Looking forward to

Re: Article: The Rise of the Data Engineer

2017-01-25 Thread Boris Tyukin
Max, really really nice post and I like your style of writing - please continue sharing your experience and inspire many of us working in more traditional environments ;) I shared your post with our leadership and hopefully we will have data engineers soon on our team! As far as UI vs. coding, I

Re: Airflow Meetup @ Paypal (San Jose)

2017-01-25 Thread Boris Tyukin
any way to schedule it during Strata week? would love to attend one of airflow meetups but I am in Florida. 03/13 or 03/14 would work the best because first two days of Strata are training days and not very busy On Tue, Jan 24, 2017 at 10:33 PM, Russell Jurney wrote: >

Re: how to capture sqoop mapreduce counters

2017-01-26 Thread Boris Tyukin
he fix for bash_oeprator as well : > https://github.com/apache/incubator-airflow/pull/2026 > > please take a look. > > Thanks > Jayesh > > > > > > > On Wed, Jan 25, 2017 at 1:25 PM, Boris Tyukin <bo...@boristyukin.com> > wrote: > > > I figured

Re: Airflow Meetup @ Paypal (San Jose)

2017-01-25 Thread Boris Tyukin
l during Strata week. > > -s > > On Wed, Jan 25, 2017 at 5:48 AM, Boris Tyukin <bo...@boristyukin.com> > wrote: > > > any way to schedule it during Strata week? would love to attend one of > > airflow meetups but I am in Florida. 03/13 or 03/14 would work the bes

Re: how to capture sqoop mapreduce counters

2017-01-25 Thread Boris Tyukin
t an easy way to get job_id from BashOperator / sqoop output. I guess I could create my own operator that would capture all stdout lines not only the last one. On Tue, Jan 24, 2017 at 9:07 AM, Boris Tyukin <bo...@boristyukin.com> wrote: > Hello all, > > is there a way to capture sqoop

Re: Airflow 1.8.0 BETA 4

2017-01-27 Thread Boris Tyukin
; > Bolke > > Sent from my iPhone > > > On 27 Jan 2017, at 20:19, Boris Tyukin <bo...@boristyukin.com> wrote: > > > > sorry mysql version i have is mysql Ver 14.14 Distrib 5.1.73 > > > > I see that DATETIME(6) is only supported on MySQL 5.7 and lat

Re: Airflow 1.8.0 BETA 4

2017-01-27 Thread Boris Tyukin
s released in 2013 > * MySQL 5.1.73 was released on 2013-12-04 > > We are really not ahead of the pack with our requirements. > > Bolke > > > On 27 Jan 2017, at 21:32, Boris Tyukin <bo...@boristyukin.com> wrote: > > > > thanks for confirmation Bolke > > >

Re: Airflow 1.8.0 BETA 5

2017-01-29 Thread Boris Tyukin
I am not sure if it is my config or something, but looks like after the upgrade and start of scheduler, airflow would totally hose CPU. The reason is two new examples that start running right away - latest only and latest with trigger. Once I pause them, CPU goes back to idle. Is this because now

Re: parsing task instance log files

2017-02-10 Thread Boris Tyukin
data database per the XCom system > > <https://airflow.incubator.apache.org/concepts.html#xcoms>. Then, > > depending > > what you want to do with that, you could query those out of the metadata > > database with the ad-hoc querying or charting UIs right within Airflow,

parsing task instance log files

2017-02-09 Thread Boris Tyukin
Hello, I am using HiveCliHook called from PythonOperator to run a series of queries and want to capture record counts for auditing and validation purposes. *I am thinking to use on_success_callback to invoke python function that will read the log file, produced by airflow and then parse it out

Re: parsing task instance log files

2017-02-10 Thread Boris Tyukin
please...? On Thu, Feb 9, 2017 at 8:35 AM, Boris Tyukin <bo...@boristyukin.com> wrote: > Hello, > > I am using HiveCliHook called from PythonOperator to run a series of > queries and want to capture record counts for auditing and validation > purposes. >

Re: Flow-based Airflow?

2017-01-23 Thread Boris Tyukin
this is a good discussion. Most of traditional ETL tools (SSIS, Informatica, DataStage etc.) have both - control flow (or task dependency) and data flow. Some tools like SSIS make a clear distinction between them - you create a control flow that calls data flows as a part of overall control flow.

how to capture sqoop mapreduce counters

2017-01-24 Thread Boris Tyukin
Hello all, is there a way to capture sqoop counters either using bash or sqoop operator? Specifically I need to pull a total number of rows loaded. By looking at bash operator, I think there is an option to push the last line of output to xcom but sqoop and mapreduce output is a bit more

Re: [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc3

2017-02-11 Thread Boris Tyukin
I am running LocalExecutor and not doing crazy things but use DAG generation heavily - everything runs fine as before. As I mentioned in other threads only had a few issues: 1) had to upgrade MySQL which was a PAIN. Cloudera CDH is running old version of MySQL which was compatible with 1.7.1 but

Re: parsing task instance log files

2017-02-10 Thread Boris Tyukin
do with that, you could query those out of the metadata > database with the ad-hoc querying or charting UIs right within Airflow, or > a later task altogether. > > On Fri, Feb 10, 2017 at 8:58 AM, Boris Tyukin <bo...@boristyukin.com> > wrote: > > > please...? > > > &g

Re: Good jinja templating in airflow examples

2017-01-19 Thread Boris Tyukin
//airflow.apache.org/code.html#airflow.models.DAG > > Max > > On Wed, Jan 18, 2017 at 11:31 AM, Boris Tyukin <bo...@boristyukin.com> > wrote: > > > Hi Guilherme, > > > > I guess it depends what exactly you want to do as not everything works > with > >

Re: question about max_active_runs or how to force only one Dag run at the time

2017-01-17 Thread Boris Tyukin
se a pool, if you are fine with the “second” dag run being run > later. > > > > On 17 Jan 2017, at 16:18, Boris Tyukin <bo...@boristyukin.com> wrote: > > > > hello, > > > > I would like to prevent a DAG from scheduling / running concurrently > >

question about max_active_runs or how to force only one Dag run at the time

2017-01-17 Thread Boris Tyukin
hello, I would like to prevent a DAG from scheduling / running concurrently whether by scheduler or if triggered externally (via airflow trigger_dag or UI). I tried to set max_active_runs to 1 but i did not work for externally triggered case - airflow would still go ahead and execute multiple

Re: question about max_active_runs or how to force only one Dag run at the time

2017-01-17 Thread Boris Tyukin
), > but yes this works. So could verify if the pool exists for a certain DAG > and create it if it doesn’t. > > Bolke > > > On 17 Jan 2017, at 17:26, Boris Tyukin <bo...@boristyukin.com> wrote: > > > > interesting but I have 200 DAGs like that - I generate them > &g

Re: Good jinja templating in airflow examples

2017-01-18 Thread Boris Tyukin
Hi Guilherme, I guess it depends what exactly you want to do as not everything works with jinja. >From documentation: https://pythonhosted.org/airflow/concepts.html#jinja-templating You can use Jinja templating with every parameter that is marked as “templated” in the documentation. You can

Re: Good jinja templating in airflow examples

2017-01-19 Thread Boris Tyukin
ds': > '{{ ds }}'}) > # All of the values in the `templates_dict` should get templated by the > Airflow engine, meaning that in this context, `foo` should print the value > of the `ds`. > > Max > > On Thu, Jan 19, 2017 at 6:22 AM, Boris Tyukin <bo...@boristyukin.com> > wro

Re: Airflow 1.8.0 BETA 1

2017-01-19 Thread Boris Tyukin
I'd like to test it on my VM with the code I am working on but I do not know how to upgrade from 1.7. Can I use pip to pull it from github? maybe someone can give me directions - i am very new to python. Also will it mess my airflow.cfg or something else I need to backup? On Wed, Jan 18, 2017 at

Re: Airflow 1.8.0 BETA 2

2017-01-20 Thread Boris Tyukin
just to make sure this is the latest one, right? https://dist.apache.org/repos/dist/dev/incubator/airflow/airflow-1.8.0b2+apache.incubating.tar.gz On Fri, Jan 20, 2017 at 10:57 AM, Bolke de Bruin wrote: > Hi All, > > I have made the SECOND beta of Airflow 1.8.0 available at:

Strata San Jose

2017-01-16 Thread Boris Tyukin
Hello everyone, is anyone talking about Airflow at Strata in March? Or maybe we can plan a meetup there - would awesome!

Re: Article: Airflow and the Future of Data Engineering: A Q

2017-02-28 Thread Boris Tyukin
thanks for sharing, was a good read! go Airflow go! On Tue, Feb 28, 2017 at 1:00 PM, Maxime Beauchemin < maximebeauche...@gmail.com> wrote: > Jointly publishing this short Q with Taylor D. Edmiston of Atronomer.io > > https://medium.com/the-astronomer-journey/airflow-and-the-future-of-data- >

Re: scheduler questions

2016-10-14 Thread Boris Tyukin
gn the start and end > > times precisely as needed. As an example, for a job that runs daily at > > 7am: > > > > airflow backfill -s 2016-10-07T07 -e 2016-10-10T07 my-dag-name -m > > > > The "-m" parameter tells Airflow to mark it successful without running

Re: Backfill for tasks to be scheduled @once

2016-10-20 Thread Boris Tyukin
hello Tamara, I just created a quick example below and it worked like you would expect it to work. It ran only once. Also when I cleared the task instance execution via UI (or alternatively, deleting it and setting DagRun to running status), it would rerun it again - this would be your on-demand

Re: ETL best practices for airflow

2016-10-17 Thread Boris Tyukin
vide > discussion about it. > > On Sun, Oct 16, 2016 at 6:40 PM, Boris Tyukin <bo...@boristyukin.com> > wrote: > > > I really look forward to it, Gerard! I've read what you you wrote so far > > and I really liked it - please keep up the great job! > > > >

Best practices for dynamically generated tasks and dags

2016-10-18 Thread Boris Tyukin
Sorry again for posting in the DEV group since it is user type question but I do not think we have a user group list and I do not feel that gitter is appropriate for this sort of discussions. I am actively testing Airflow for a specific use case which is a generation of workflows/tasks for

Re: Usage of "on_failure_callback" ?

2016-10-18 Thread Boris Tyukin
hi Jason, here is an example below - in task0_python_callable i did 1/0 to raise an error and Airflow then called task0_failure_cb. I could see FAIL in the task log so my task0_failure_cb was definitely invoked. The task would still have failed status if look at UI. Are you expecting a

Re: ETL best practices for airflow

2016-10-25 Thread Boris Tyukin
ask instance sources from immutable datasets that are > persisted > >> in your backend. That allows to satisfy the guarantee that re-running > any > >> chunk of ETL at different point in time should lead to the exact same > >> result. It also usually means that you need to 1-do incremental loads, > and > >> 2- "snapshot" your dimension/referential/small tables in time to make > sure > >> that running the ETL from 26 days ago sources from the dimension > snapshot > >> as it was back then and yields the exact same result. > >> > >> Anyhow, it's a complex and important subject I should probably write > about > >> in a structured way sometime. > >> > >> Max > >> > >> On Mon, Oct 17, 2016 at 6:12 PM, Boris Tyukin <bo...@boristyukin.com> > >> wrote: > >> > >> > >> > > >

Re: Changing the crontab for a DAG

2016-10-25 Thread Boris Tyukin
yep this is what was recommeded in https://cwiki.apache.org/confluence/display/AIRFLOW/Common+Pitfalls - When needing to change your start_date and schedule interval, change the name of the dag (a.k.a. dag_id) - I follow the convention : my_dag_v1 , my_dag_v2, my_dag_v3, my_dag_v4,

Re: Best practices for dynamically generated tasks and dags

2016-10-21 Thread Boris Tyukin
Guys, would you mind to chime in and share your experience?

Re: Best practices for dynamically generated tasks and dags

2016-10-21 Thread Boris Tyukin
; resultant DAG structure is not too large or complicated, we haven't had any > issues with that or done anything special. I think it's just the fact of > the matter that if you give the scheduler a lot of work to do to determine > the DAG shape, it will take a while. > > Laura &

Re: Airflow 2.0

2016-11-21 Thread Boris Tyukin
I am still deciding between Airflow and oozie for our brand new Hadoop project but here is a few things that I did not like during my limited testing: 1) pain with scheduler/webserver restarts - things magically begin working after restart or disappear (like DAG tasks that are no longer part of

scheduler questions

2016-10-13 Thread Boris Tyukin
Hello all and thanks for such an amazing project! I have been evaluating Airflow and spent a few days reading about it and playing with it and I have a few questions that I struggle to understand. Let's say I have a simple DAG that runs once a day and it is doing a full reload of tables from the

Re: scheduler questions

2016-10-13 Thread Boris Tyukin
eturn False > > > > return True > > > > > > t0 = ShortCircuitOperator( > > > > task_id = 'short_circuit_if_not_current, > > > > provide_context = True, > > > > python_callable = skip_to_current_job, > > > >

Re: scheduler questions

2016-10-13 Thread Boris Tyukin
eturn True > > > t0 = ShortCircuitOperator( > > task_id = 'short_circuit_if_not_current, > > provide_context = True, > > python_callable = skip_to_current_job, > > dag = dag > > ) > > > -s > > > On Thu, Oct 13, 2016 at 7:46 AM, Bo

Re: scheduler questions

2016-10-13 Thread Boris Tyukin
<san...@apache.org> > wrote: > > > Yes! It does work with Depends_on_past=True. > > -s > > > > On Thu, Oct 13, 2016 at 10:28 AM, Boris Tyukin <bo...@boristyukin.com> > > wrote: > > > >> thanks so much, Sid! just a follow up question on &qu

Re: scheduler questions

2016-10-13 Thread Boris Tyukin
s strict dag chaining. > > If you just don't want the same task in the subsequent dag run to get > scheduled unless the first task completes, depends_on_past=True helps > there. This is more a cascading effect in the tree view. > -s > > On Thu, Oct 13, 2016 at

Re: A question/poll on the TaskInstance data model...

2016-10-15 Thread Boris Tyukin
% > 3A%2F%2Ftwitter.com%2Fapigee=5141814536306688= > e558dca3-da0a-4d9f-c1b3-6cb9174fcb5e> > <http://t.sidekickopen06.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nM > JW7t5XZs4WJfgqW4WJj7n3MP7VWW3LqXLC56dWRRf2H8CkP02?t=http%3A% > 2F%2Fadapt.apigee.com%2F=5141814536306688=e558dca3- > da0a

Re: Fwd: Issue with Dynamically created tasks in a DAG

2016-10-16 Thread Boris Tyukin
I opened a JIRA - looks like based on comments in other threads, it does not work properly right now. [AIRFLOW-574] Show Graph/Tree view and Task Instance logs using executed DagRun, not current

Re: ETL best practices for airflow

2016-10-16 Thread Boris Tyukin
I really look forward to it, Gerard! I've read what you you wrote so far and I really liked it - please keep up the great job! I am hoping to see some best practices for the design of incremental loads and using timestamps from source database systems (not being on UTC so still confused about it

Re: A question/poll on the TaskInstance data model...

2016-10-15 Thread Boris Tyukin
Hi Ben, is it to address the issue I just described yesterday "Issue with Dynamically created tasks in a DAG"? I was hoping someone can confirm this as a bug and if there is a JIRA to address that - otherwise I would be happy to open one. To me it is a pretty major issue and a very misleading

Re: variable scope with dynamic dags

2017-03-22 Thread Boris Tyukin
rce file. > > Jeremiah > > On Wed, Mar 22, 2017 at 8:45 AM Boris Tyukin <bo...@boristyukin.com> > wrote: > > > Hi, > > > > I have a weird question but it bugs my mind. I have some like below to > > generate dags dynamically, using Max's example cod

Re: variable scope with dynamic dags

2017-03-23 Thread Boris Tyukin
1) logging.info('end of get_dag({})'.format(i)) logging.info(hpy().heap()) return dag # payload = [1,2,3] * (2 * 10 ** 7) # 457 Mb payload = [1] * (2 * 10 ** 7) # 152 Mb for i in range(3): dag = get_dag(i+1) globals()[dag.dag_id] = dag On Wed, Mar 22, 2017 at 3:50 PM, Boris Tyukin

Re: variable scope with dynamic dags

2017-03-22 Thread Boris Tyukin
different process, and each process is going to load your DAG file when it > runs. If resource use is a concern, I suggest you look at out-of-core or > persistent storage for the object so you don't need to load the whole thing > every time. > > On Wed, Mar 22, 2017 at 11:20 AM Bo

Re: Reminder : LatestOnlyOperator

2017-03-18 Thread Boris Tyukin
o,like > > > class Oracle_LatestOnly_Operator (Oracle_Operator, LatestOnlyOperator): > > ... > > I might be overthinking this and there could be a simpler way? > Sorry, I am still learning Airflow concepts... > > Thanks. > > > > -- > Ruslan Dautkhanov > > On Sat,

Re: Reminder : LatestOnlyOperator

2017-03-17 Thread Boris Tyukin
this is nice indeed along with the new catchup option https://airflow.incubator.apache.org/scheduler.html#backfill-and-catchup Thanks Sid and Ben for adding these new options! for a complete picture, it would be nice to force only one dag run at the time. On Fri, Mar 17, 2017 at 7:33 PM,

Re: Reminder : LatestOnlyOperator

2017-03-20 Thread Boris Tyukin
It does make sense. > Although how it's different from depends_on_past task-level parameter? > In both cases, a task will be skipped if there is another TI of this task > is still running (from a previous dagrun), right? > > > Thanks, > Ruslan > > > On Sat, Mar 18, 201

variable scope with dynamic dags

2017-03-22 Thread Boris Tyukin
Hi, I have a weird question but it bugs my mind. I have some like below to generate dags dynamically, using Max's example code from FAQ. It works fine but I have one large dict (let's call it my_outer_dict) that takes over 60Mb in memory and I need to access it from all generated dags. Needless

Re: issue fetching master repo

2017-04-20 Thread Boris Tyukin
d changes in working directory) # # modified: airflow/www/static/nv.d3.js but I did not touch that file. I cannot do rebase or commit: cannot rebase: you have unstaged changes D airflow/www/static/nv.d3.js This is really weird, please help On Wed, Apr 19, 2017 at 11:19 PM, Boris Tyukin &l

Re: issue fetching master repo

2017-04-20 Thread Boris Tyukin
dbr...@gmail.com> a écrit : > > > Hi Boris, > > > > To be honest this is not an airflow question, but a git question. > > > > If you havent made any changes to the code, why don’t you delete the test > > folder and cone again? > > > > B. > &

issue fetching master repo

2017-04-19 Thread Boris Tyukin
hey guys, want to submit my first tiny PR and once I fork airflow and clone my repo get this message below: I cannot commit / rebase and I cannot find a way to remove this file. Is it only my who has this issue? git status # On branch master # Changed but not updated: # (use "git add ..." to

Re: Adding Variables and Connections via script

2017-03-07 Thread Boris Tyukin
To add to Ali's reply, there was a PR for connections and cli https://github.com/apache/incubator-airflow/pull/1802 hopefully it will make to 1.8 On Tue, Mar 7, 2017 at 4:49 PM, Nicholas Hodgkinson < nik.hodgkin...@collectivehealth.com> wrote: > I would like to be able to create a script to

Re: Airflow dependency won't change

2017-08-17 Thread Boris Tyukin
e the dag id to something else, the > dependency holds. I am very curious but why i cannot just switch the order > and don't need to change the dag id. Thanks a lot! > > > > On Wed, Aug 9, 2017 at 10:21 AM, Boris Tyukin <bo...@boristyukin.com> > wrote: > > > Hit

Re: Info needed regarding upgrading to 1.8.2

2017-08-18 Thread Boris Tyukin
cation immediately and sometimes doesn't pickup at all. And when > Scheduler doesn't pickup new dag, UI also doesn't show it, i.e. no refresh > button would be available. I am trying to debug that issue with Scheduler, > but not sure if it's a real issue at all or something else. > > > >

Re: Apache airflow usage survey

2017-06-10 Thread Boris Tyukin
you are asking some really good questions, Gerard. If we do not use it in production yet but going to, are we eligible to take it? :) Boris On Sat, Jun 10, 2017 at 6:40 AM, Gerard Toonstra wrote: > Hi all, > > I'm curious how others are using and deploying airflow. Rather

Re: airflow upgrade from v1.7.1.3 to 1.8

2017-06-09 Thread Boris Tyukin
Max, do you think it would be a good idea to document the update process under installation section? I know it is basic commands but you know have a lot of users, introduced to Python thanks to Airflow! (me!) I can create jira and PR myself.. Boris On Thu, Jun 8, 2017 at 6:46 PM, Maxime

Re: Apache Airflow welcome new committer/PMC member : Fokko Driespong (a.k.a. fokko)

2017-10-04 Thread Boris Tyukin
great news! On Wed, Oct 4, 2017 at 6:03 PM, Arthur Wiedmer wrote: > Welcome Fokko! > > On Wed, Oct 4, 2017 at 12:51 PM, Sid Anand wrote: > > > Folks, > > Please join the Apache Airflow PMC in welcoming its newest member and > > co-committer, Fokko

Re: Return results optionally from spark_sql_hook

2017-10-16 Thread Boris Tyukin
that builds a > > > > > SparkSession within Python (using the Spark pyspark api) and > fetches > > > the > > > > > count from the spark partition that you've just created. Create a > > > > > BranchPythonOperator that will invoke

Re: new logging

2017-10-16 Thread Boris Tyukin
sorry i was not clear. In DAG definition file, I would normally import logging and when use with python operator like below (see print_context). In 1.8.2 i would see in Airflow log file (for that task) both 'hi from print' and 'hi from logging'. now I can only see 'hi from print'. I installed

Re: Return results optionally from spark_sql_hook

2017-10-14 Thread Boris Tyukin
Hi Fokko, thanks for your response, really appreciate it! Basically in my case I have two Spark SQL queries: 1) the first query does INSERT OVERWRITE to a partition and may take a while for a while 2) then I run a second query right after it to get count of rows of that partition. 3) I need to

Re: spark sql hook with multiple queries

2017-10-14 Thread Boris Tyukin
ons > "-Dlog4j.configuration=file:///tmp/log4j.properties" -e "SELECT 1; SELECT > 1;" > 1 > Time taken: 1.947 seconds, Fetched 1 row(s) > 1 > Time taken: 0.032 seconds, Fetched 1 row(s) > > Can you share the exception that you are seeing? What versio

Re: Some random fun

2017-09-25 Thread Boris Tyukin
this is nerdy!! :) thanks for sharing! On Mon, Sep 25, 2017 at 3:00 PM, Dan Davydov wrote: > Haha, this is great. > > On Mon, Sep 25, 2017 at 11:37 AM Shah Altaf wrote: > > > **CupcakeSensor activated** > > > > > > > > On Mon, Sep 25, 2017 at

spark sql hook with multiple queries

2017-10-11 Thread Boris Tyukin
hi guys, tried spark_sql_hook to run a multi-statement query (two queries separated by semi-column ) and it hangs forever. If i comment out the second query, it runs fine. Anyone had the same issue? i do not see anything in the code preventing more one statement. sql = """ select * from

Re: new logging

2017-10-17 Thread Boris Tyukin
't see where you configured the default logger > > <https://docs.python.org/2/howto/logging.html#configuring-logging>. I > > think > > you'd have to at least throw in: > > logger = logging.getLogger('python_test_logging') > > and then use the logger.info(…) inste

Re: Xcom_push is broken - probably after the timezone awareness

2017-12-06 Thread Boris Tyukin
in 1.9.0 > 4. Logging.info will be supported in 1.9.0 final > > > So no reason to panic I would say ;-) > > Bolke > > Verstuurd vanaf mijn iPad > > > Op 6 dec. 2017 om 16:35 heeft Boris Tyukin <bo...@boristyukin.com> het > volgende geschreven: > > >

Re: DAG logging

2017-10-31 Thread Boris Tyukin
Chris, see my post "new logging" - apparently we cannot use logging any more and have to init log handler. On Tue, Oct 31, 2017 at 1:54 PM, Chris Riccomini wrote: > Correction: > > import logging > > class DqRowCheckOperator(BaseOperator): > ... > def execute(...): >