[jira] [Assigned] (AIRFLOW-1160) Upadte SparkSubmitOperator parameters

2017-04-30 Thread Connor Ameres (JIRA)

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Connor Ameres reassigned AIRFLOW-1160:
--

Assignee: Connor Ameres

> Upadte SparkSubmitOperator parameters
> -
>
> Key: AIRFLOW-1160
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1160
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: contrib
>Affects Versions: Airflow 1.8
>Reporter: Xi Wang
>Assignee: Connor Ameres
> Fix For: Airflow 1.8
>
>
> param executor_cores from spark_submit_hook.py is not compatible with 
> SparkSubmit, should be total-executor-cores instead.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Closed] (AIRFLOW-1155) Add Tails.com to community

2017-04-30 Thread Alan Cruickshank (JIRA)

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Cruickshank closed AIRFLOW-1155.
-
Resolution: Fixed

> Add Tails.com to community
> --
>
> Key: AIRFLOW-1155
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1155
> Project: Apache Airflow
>  Issue Type: Wish
>  Components: Documentation
>Reporter: Alan Cruickshank
>Assignee: Alex Guziel
>Priority: Trivial
> Fix For: 1.9.0
>
>
> Add to README.md
> ```
> 1. [Tails.com](https://tails.com/) 
> [[@alanmcruickshank](https://github.com/alanmcruickshank)]
> ```



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (AIRFLOW-1026) connection string using _cmd tin airflow.cfg is broken

2017-04-30 Thread Gavriel Fleischer (JIRA)

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15990257#comment-15990257
 ] 

Gavriel Fleischer commented on AIRFLOW-1026:


I see the same problem with the sqlite error. I tried to override it:

sql_alchemy_conn =

but then I get this error:
sqlalchemy.exc.ArgumentError: Could not parse rfc1738 URL from string ''


> connection string using _cmd tin airflow.cfg is broken
> --
>
> Key: AIRFLOW-1026
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1026
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: configuration
>Affects Versions: Airflow 1.8
>Reporter: Harish Singh
>Priority: Critical
> Fix For: Airflow 1.8
>
>
> sql_alchemy_conn_cmd = python ./pipeline/dags/configure.py
> I am expectiing configure.py to be invoked.
> But it just throws:
>  "cannot use sqlite with the LocalExecutor"
> The connection string that my script "configure.py" would return is something 
> like this:
> mysql+mysqldb://username:**@mysqlhostname:3306/airflowdbname
> But after debugging, I found that, my script is not getting invoked at all.
> This is my airflow.cfg:
> executor = LocalExecutor
> sql_alchemy_conn_cmd = python ./pipeline/dags/configure.py 
> sql_alchemy_pool_size = 5
> sql_alchemy_pool_recycle = 3600
> I tried not using the script and directly hardcoding the conn_url
> sql_alchemy_conn = 
> mysql+mysqldb://username:**@mysqlhostname:3306/airflowdbname
> It works.
> But  there is a regression bug if somebody wants to use "sql_alchemy_conn_cmd"



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Created] (AIRFLOW-1162) execution_date doesn't have a default value exception

2017-04-30 Thread Neer Shay (JIRA)
Neer Shay created AIRFLOW-1162:
--

 Summary: execution_date doesn't have a default value exception
 Key: AIRFLOW-1162
 URL: https://issues.apache.org/jira/browse/AIRFLOW-1162
 Project: Apache Airflow
  Issue Type: Bug
  Components: DagRun
Affects Versions: Airflow 1.8, 1.8.0
 Environment: mysql db, standard linux environment
Reporter: Neer Shay


Example DAG:
default_args = {
'owner': 'airflow',
'depends_on_past': False,
'start_date': datetime(2017, 4, 27),
'email': ['airf...@airflow.com'],
'email_on_failure': False,
'email_on_retry': False,
}

dag = DAG('tutorial_new_dag', default_args=default_args)

t1 = BashOperator(
task_id='print_date',
bash_command='date',
dag=dag)

Running this DAG through the web-ui using the "run now" button produces the 
following exception:

Traceback (most recent call last):
  File "~/.virtualenvs/airflow/local/lib/python2.7/site-packages/flask/app.py", 
line 1988, in wsgi_app
response = self.full_dispatch_request()
  File "~/.virtualenvs/airflow/local/lib/python2.7/site-packages/flask/app.py", 
line 1641, in full_dispatch_request
rv = self.handle_user_exception(e)
  File "~/.virtualenvs/airflow/local/lib/python2.7/site-packages/flask/app.py", 
line 1544, in handle_user_exception
reraise(exc_type, exc_value, tb)
  File "~/.virtualenvs/airflow/local/lib/python2.7/site-packages/flask/app.py", 
line 1639, in full_dispatch_request
rv = self.dispatch_request()
  File "~/.virtualenvs/airflow/local/lib/python2.7/site-packages/flask/app.py", 
line 1625, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
  File 
"~/.virtualenvs/airflow/local/lib/python2.7/site-packages/flask_admin/base.py", 
line 69, in inner
return self._run_view(f, *args, **kwargs)
  File 
"~/.virtualenvs/airflow/local/lib/python2.7/site-packages/flask_admin/base.py", 
line 368, in _run_view
return fn(self, *args, **kwargs)
  File 
"~/.virtualenvs/airflow/local/lib/python2.7/site-packages/flask_login.py", line 
755, in decorated_view
return func(*args, **kwargs)
  File 
"~/.virtualenvs/airflow/local/lib/python2.7/site-packages/airflow/www/utils.py",
 line 125, in wrapper
return f(*args, **kwargs)
  File 
"~/.virtualenvs/airflow/local/lib/python2.7/site-packages/airflow/www/utils.py",
 line 172, in wrapper
return f(*args, **kwargs)
  File 
"~/.virtualenvs/airflow/local/lib/python2.7/site-packages/airflow/www/views.py",
 line 986, in trigger
external_trigger=True
  File 
"~/.virtualenvs/airflow/local/lib/python2.7/site-packages/airflow/utils/db.py", 
line 53, in wrapper
result = func(*args, **kwargs)
  File 
"~/.virtualenvs/airflow/local/lib/python2.7/site-packages/airflow/models.py", 
line 3383, in create_dagrun
run.verify_integrity(session=session)
  File 
"~/.virtualenvs/airflow/local/lib/python2.7/site-packages/airflow/utils/db.py", 
line 53, in wrapper
result = func(*args, **kwargs)
  File 
"~/.virtualenvs/airflow/local/lib/python2.7/site-packages/airflow/models.py", 
line 4117, in verify_integrity
session.commit()
  File 
"~/.virtualenvs/airflow/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py",
 line 874, in commit
self.transaction.commit()
  File 
"~/.virtualenvs/airflow/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py",
 line 461, in commit
self._prepare_impl()
  File 
"~/.virtualenvs/airflow/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py",
 line 441, in _prepare_impl
self.session.flush()
  File 
"~/.virtualenvs/airflow/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py",
 line 2139, in flush
self._flush(objects)
  File 
"~/.virtualenvs/airflow/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py",
 line 2259, in _flush
transaction.rollback(_capture_exception=True)
  File 
"~/.virtualenvs/airflow/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py",
 line 66, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
  File 
"~/.virtualenvs/airflow/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py",
 line 2223, in _flush
flush_context.execute()
  File 
"~/.virtualenvs/airflow/local/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py",
 line 389, in execute
rec.execute(self)
  File 
"~/.virtualenvs/airflow/local/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py",
 line 548, in execute
uow
  File 
"~/.virtualenvs/airflow/local/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py",
 line 181, in save_obj
mapper, table, insert)
  File 
"~/.virtualenvs/airflow/local/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py",
 line 835, in _emit_insert_statements
execute(statement, params)
  File 
"~/.virtualenvs/airflow/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
 line 945, in execute
return meth(self, multiparams, params)
  File 
"~/.virtualenvs/airflow/local/lib/python2.7/site-packages/sqlalchemy/sql/elements.py",
 line