[jira] [Commented] (AIRFLOW-811) Bash_operator dont read multiline output
[
https://issues.apache.org/jira/browse/AIRFLOW-811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16236594#comment-16236594
]
Jayesh commented on AIRFLOW-811:
this turn out to be more changes than simply taking care of bug in for loop.
https://github.com/apache/incubator-airflow/pull/2026
> Bash_operator dont read multiline output
>
>
> Key: AIRFLOW-811
> URL: https://issues.apache.org/jira/browse/AIRFLOW-811
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Jayesh
>Assignee: Jayesh
>Priority: Minor
>
> following piece of code is the root cause of it.
> {code}
> line = ''
> for line in iter(sp.stdout.readline, b''):
> line = line.decode(self.output_encoding).strip()
> logging.info(line)
> {code}
> I plan to fix it using string buffer instead of just 1 line string variable
> here.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (AIRFLOW-811) Bash_operator dont read multiline output
Jayesh created AIRFLOW-811:
--
Summary: Bash_operator dont read multiline output
Key: AIRFLOW-811
URL: https://issues.apache.org/jira/browse/AIRFLOW-811
Project: Apache Airflow
Issue Type: Bug
Reporter: Jayesh
Assignee: Jayesh
Priority: Minor
following piece of code is the root cause of it.
{code}
line = ''
for line in iter(sp.stdout.readline, b''):
line = line.decode(self.output_encoding).strip()
logging.info(line)
{code}
I plan to fix it using string buffer instead of just 1 line string variable
here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AIRFLOW-751) SFTP file transfer functionality
[ https://issues.apache.org/jira/browse/AIRFLOW-751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824443#comment-15824443 ] Jayesh commented on AIRFLOW-751: [~genomics-geek], I looked at your PR about sftp, while it has the required get/put functionality, I will have to refactor it based on the new ssh_hook from AIRFLOW-756 > SFTP file transfer functionality > > > Key: AIRFLOW-751 > URL: https://issues.apache.org/jira/browse/AIRFLOW-751 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Michael Gonzalez >Assignee: Jayesh >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AIRFLOW-756) Refactor ssh_hook and ssh_operator
[ https://issues.apache.org/jira/browse/AIRFLOW-756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824441#comment-15824441 ] Jayesh commented on AIRFLOW-756: I can't update this ticket, i also assigned this to me, but no luck. anyway, summary is " Refactor ssh_hook and ssh_operator to use Paramiko instead of current subprocess" > Refactor ssh_hook and ssh_operator > -- > > Key: AIRFLOW-756 > URL: https://issues.apache.org/jira/browse/AIRFLOW-756 > Project: Apache Airflow > Issue Type: Improvement > Components: hooks, operators >Reporter: Michael Gonzalez >Assignee: Jayesh >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (AIRFLOW-751) SFTP file transfer functionality
[ https://issues.apache.org/jira/browse/AIRFLOW-751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jayesh reassigned AIRFLOW-751: -- Assignee: Jayesh > SFTP file transfer functionality > > > Key: AIRFLOW-751 > URL: https://issues.apache.org/jira/browse/AIRFLOW-751 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Michael Gonzalez >Assignee: Jayesh >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (AIRFLOW-756) Refactor ssh_hook and ssh_operator
[ https://issues.apache.org/jira/browse/AIRFLOW-756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jayesh reassigned AIRFLOW-756: -- Assignee: Jayesh > Refactor ssh_hook and ssh_operator > -- > > Key: AIRFLOW-756 > URL: https://issues.apache.org/jira/browse/AIRFLOW-756 > Project: Apache Airflow > Issue Type: Improvement > Components: hooks, operators >Reporter: Michael Gonzalez >Assignee: Jayesh >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AIRFLOW-744) incompatibility notice for 1.7.3 and note on how to fix it
Jayesh created AIRFLOW-744: -- Summary: incompatibility notice for 1.7.3 and note on how to fix it Key: AIRFLOW-744 URL: https://issues.apache.org/jira/browse/AIRFLOW-744 Project: Apache Airflow Issue Type: Bug Components: docs, webserver Reporter: Jayesh Priority: Minor since airflow 1.7.3 has some compatibility issues regarding Jinja2 (2.9.2), its been breaking for lot of people around the world. so we came up with suggestion to put up the notice on website about this version and also note on how to fix it ( most likely only downgrading to Jinja2 2.8.1 is required ) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (AIRFLOW-499) Add in capabilities to use atlassian tools with Airflow
[ https://issues.apache.org/jira/browse/AIRFLOW-499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15783749#comment-15783749 ] Jayesh edited comment on AIRFLOW-499 at 12/29/16 8:50 PM: -- "Atlassian tools" is a very broad term, so I created AIRFLOW-683 for specifically jira connector was (Author: jhsenjaliya): "Atlassian tools" is a very broad term, there is a AIRFLOW-683 for specifically jira capabilities > Add in capabilities to use atlassian tools with Airflow > --- > > Key: AIRFLOW-499 > URL: https://issues.apache.org/jira/browse/AIRFLOW-499 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Tony Doran >Assignee: Jayesh > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AIRFLOW-499) Add in capabilities to use atlassian tools with Airflow
[ https://issues.apache.org/jira/browse/AIRFLOW-499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15783749#comment-15783749 ] Jayesh commented on AIRFLOW-499: "Atlassian tools" is a very broad term, there is a AIRFLOW-683 for specifically jira capabilities > Add in capabilities to use atlassian tools with Airflow > --- > > Key: AIRFLOW-499 > URL: https://issues.apache.org/jira/browse/AIRFLOW-499 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Tony Doran >Assignee: Jayesh > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AIRFLOW-711) stopping scheduler makes running sensor fail
Jayesh created AIRFLOW-711:
--
Summary: stopping scheduler makes running sensor fail
Key: AIRFLOW-711
URL: https://issues.apache.org/jira/browse/AIRFLOW-711
Project: Apache Airflow
Issue Type: Bug
Reporter: Jayesh
when i stop the scheduler ( by stopping it via my IDE ) it causes the sensor
which is in running state and has run fine for couple of time already, will
instantly fail with following error
{code}
[2016-12-21 19:25:08,588] {models.py:1347} ERROR -
Traceback (most recent call last):
File "/src/apache/airflow/airflow/models.py", line 1304, in run
result = task_copy.execute(context=context)
File "/src/apache/airflow/airflow/operators/sensors.py", line 79, in execute
sleep(self.poke_interval)
KeyboardInterrupt
{code}
on IDE side i can see the following traceback generated
{code}
Traceback (most recent call last):
File "./airflow", line 28, in
args.func(args)
File "/src/apache/airflow/airflow/bin/cli.py", line 370, in run
run_job.run()
File "/src/apache/airflow/airflow/jobs.py", line 202, in run
self._execute()
File "/src/apache/airflow/airflow/jobs.py", line 2028, in _execute
self.heartbeat()
File "/src/apache/airflow/airflow/jobs.py", line 177, in heartbeat
sleep(sleep_for)
KeyboardInterrupt
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (AIRFLOW-711) stopping scheduler makes running sensor fail
[
https://issues.apache.org/jira/browse/AIRFLOW-711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jayesh updated AIRFLOW-711:
---
Description:
on master brach, when i stop the scheduler ( by stopping it via my IDE ) it
causes the sensor which is in running state and has run fine for couple of time
already, will instantly fail with following error
{code}
[2016-12-21 19:25:08,588] {models.py:1347} ERROR -
Traceback (most recent call last):
File "/src/apache/airflow/airflow/models.py", line 1304, in run
result = task_copy.execute(context=context)
File "/src/apache/airflow/airflow/operators/sensors.py", line 79, in execute
sleep(self.poke_interval)
KeyboardInterrupt
{code}
on IDE side i can see the following traceback generated
{code}
Traceback (most recent call last):
File "./airflow", line 28, in
args.func(args)
File "/src/apache/airflow/airflow/bin/cli.py", line 370, in run
run_job.run()
File "/src/apache/airflow/airflow/jobs.py", line 202, in run
self._execute()
File "/src/apache/airflow/airflow/jobs.py", line 2028, in _execute
self.heartbeat()
File "/src/apache/airflow/airflow/jobs.py", line 177, in heartbeat
sleep(sleep_for)
KeyboardInterrupt
{code}
was:
when i stop the scheduler ( by stopping it via my IDE ) it causes the sensor
which is in running state and has run fine for couple of time already, will
instantly fail with following error
{code}
[2016-12-21 19:25:08,588] {models.py:1347} ERROR -
Traceback (most recent call last):
File "/src/apache/airflow/airflow/models.py", line 1304, in run
result = task_copy.execute(context=context)
File "/src/apache/airflow/airflow/operators/sensors.py", line 79, in execute
sleep(self.poke_interval)
KeyboardInterrupt
{code}
on IDE side i can see the following traceback generated
{code}
Traceback (most recent call last):
File "./airflow", line 28, in
args.func(args)
File "/src/apache/airflow/airflow/bin/cli.py", line 370, in run
run_job.run()
File "/src/apache/airflow/airflow/jobs.py", line 202, in run
self._execute()
File "/src/apache/airflow/airflow/jobs.py", line 2028, in _execute
self.heartbeat()
File "/src/apache/airflow/airflow/jobs.py", line 177, in heartbeat
sleep(sleep_for)
KeyboardInterrupt
{code}
> stopping scheduler makes running sensor fail
>
>
> Key: AIRFLOW-711
> URL: https://issues.apache.org/jira/browse/AIRFLOW-711
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Jayesh
>
> on master brach, when i stop the scheduler ( by stopping it via my IDE ) it
> causes the sensor which is in running state and has run fine for couple of
> time already, will instantly fail with following error
> {code}
> [2016-12-21 19:25:08,588] {models.py:1347} ERROR -
> Traceback (most recent call last):
> File "/src/apache/airflow/airflow/models.py", line 1304, in run
> result = task_copy.execute(context=context)
> File "/src/apache/airflow/airflow/operators/sensors.py", line 79, in execute
> sleep(self.poke_interval)
> KeyboardInterrupt
> {code}
> on IDE side i can see the following traceback generated
> {code}
> Traceback (most recent call last):
> File "./airflow", line 28, in
> args.func(args)
> File "/src/apache/airflow/airflow/bin/cli.py", line 370, in run
> run_job.run()
> File "/src/apache/airflow/airflow/jobs.py", line 202, in run
> self._execute()
> File "/src/apache/airflow/airflow/jobs.py", line 2028, in _execute
> self.heartbeat()
> File "/src/apache/airflow/airflow/jobs.py", line 177, in heartbeat
> sleep(sleep_for)
> KeyboardInterrupt
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (AIRFLOW-708) SSHExecuteOperator dont respect multiline output from the command
[
https://issues.apache.org/jira/browse/AIRFLOW-708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jayesh reassigned AIRFLOW-708:
--
Assignee: Jayesh
> SSHExecuteOperator dont respect multiline output from the command
> --
>
> Key: AIRFLOW-708
> URL: https://issues.apache.org/jira/browse/AIRFLOW-708
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Jayesh
>Assignee: Jayesh
>
> I find following piece of code works when you have 1 liner output but simply
> cant work for multiline output from the given bash_command ( it will print
> proper multiline output in log though )
> {code}
> line = ''
> for line in iter(sp.stdout.readline, b''):
> line = line.decode().strip()
> logging.info(line)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (AIRFLOW-688) admin->connection extra value dont show up
[
https://issues.apache.org/jira/browse/AIRFLOW-688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jayesh updated AIRFLOW-688:
---
Assignee: (was: Jayesh)
> admin->connection extra value dont show up
> --
>
> Key: AIRFLOW-688
> URL: https://issues.apache.org/jira/browse/AIRFLOW-688
> Project: Apache Airflow
> Issue Type: Bug
> Components: ui
>Reporter: Jayesh
>Priority: Minor
>
> steps to recreate
> # go to http://localhost:8080/admin/connection/
> # create new connection and add this into extra field {code} {"verify":
> False} {code}
> # save the connection, and come back to UI on list of connection
> # click on + to see the extra field
> It will not show up, but this json string will work: {code} {"verify":
> "False"} {code}
> basically it does not appear when the value field of the json string is not
> quoted.
> looks like it throws an error {code} {models.py:594} ERROR - No JSON object
> could be decoded {code}
> in anycase, extra string gets stored into database, but it creates problems
> when passing the extra params to the external library
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (AIRFLOW-688) admin->connection extra value dont show up
[
https://issues.apache.org/jira/browse/AIRFLOW-688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jayesh reassigned AIRFLOW-688:
--
Assignee: Jayesh
> admin->connection extra value dont show up
> --
>
> Key: AIRFLOW-688
> URL: https://issues.apache.org/jira/browse/AIRFLOW-688
> Project: Apache Airflow
> Issue Type: Bug
> Components: ui
>Reporter: Jayesh
>Assignee: Jayesh
>Priority: Minor
>
> steps to recreate
> # go to http://localhost:8080/admin/connection/
> # create new connection and add this into extra field {code} {"verify":
> False} {code}
> # save the connection, and come back to UI on list of connection
> # click on + to see the extra field
> It will not show up, but this json string will work: {code} {"verify":
> "False"} {code}
> basically it does not appear when the value field of the json string is not
> quoted.
> looks like it throws an error {code} {models.py:594} ERROR - No JSON object
> could be decoded {code}
> in anycase, extra string gets stored into database, but it creates problems
> when passing the extra params to the external library
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (AIRFLOW-688) admin->connection extra value dont show up
[
https://issues.apache.org/jira/browse/AIRFLOW-688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jayesh updated AIRFLOW-688:
---
Description:
steps to recreate
# go to http://localhost:8080/admin/connection/
# create new connection and add this into extra field {code} {"verify": False}
{code}
# save the connection, and come back to UI on list of connection
# click on + to see the extra field
It will not show up, but this json string will work: {code} {"verify": "False"}
{code}
basically it does not appear when the value field of the json string is not
quoted.
looks like it throws an error {code} {models.py:594} ERROR - No JSON object
could be decoded {code}
in anycase, extra string gets stored into database, but it creates problems
when passing the extra params to the external library
was:
steps to recreate
# go to http://localhost:8080/admin/connection/
# create new connection and add this into extra field {code} {"verify": False}
{code}
# save the connection, and come back to UI on list of connection
# click on + to see the extra field
It will not show up, but this json string will work: {code} {"verify": "False"}
{code}
basically it does not appear when the value field of the json string is not
quoted.
looks like it throws an error {code} {models.py:594} ERROR - No JSON object
could be decoded {code}
in anycase, extra string gets stored into database, so no issue there.
> admin->connection extra value dont show up
> --
>
> Key: AIRFLOW-688
> URL: https://issues.apache.org/jira/browse/AIRFLOW-688
> Project: Apache Airflow
> Issue Type: Bug
> Components: ui
>Reporter: Jayesh
>Priority: Minor
>
> steps to recreate
> # go to http://localhost:8080/admin/connection/
> # create new connection and add this into extra field {code} {"verify":
> False} {code}
> # save the connection, and come back to UI on list of connection
> # click on + to see the extra field
> It will not show up, but this json string will work: {code} {"verify":
> "False"} {code}
> basically it does not appear when the value field of the json string is not
> quoted.
> looks like it throws an error {code} {models.py:594} ERROR - No JSON object
> could be decoded {code}
> in anycase, extra string gets stored into database, but it creates problems
> when passing the extra params to the external library
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (AIRFLOW-688) admin->connection extra value dont show up
[
https://issues.apache.org/jira/browse/AIRFLOW-688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jayesh updated AIRFLOW-688:
---
Description:
steps to recreate
# go to http://localhost:8080/admin/connection/
# create new connection and add this into extra field {code} {"verify": False}
{code}
# save the connection, and come back to UI on list of connection
# click on + to see the extra field
It will not show up, but this json string will work: {code} {"verify": "False"}
{code}
basically it does not appear when the value field of the json string is not
quoted.
looks like it throws an error {code} {models.py:594} ERROR - No JSON object
could be decoded {code}
in anycase, extra string gets stored into database, so no issue there.
was:
steps to recreate
# go to http://localhost:8080/admin/connection/
# create new connection and add this into extra field {code} {"verify": False}
{code}
# save the connection, and come back to UI on list of connection
# click on + to see the extra field
It will not show up, but this json string will show up: {code} {"verify":
"False"} {code}
basically it does not appear when the value field of the json string is not
quoted.
in anycase, extra string gets stored into database, so no issue there.
> admin->connection extra value dont show up
> --
>
> Key: AIRFLOW-688
> URL: https://issues.apache.org/jira/browse/AIRFLOW-688
> Project: Apache Airflow
> Issue Type: Bug
> Components: ui
>Reporter: Jayesh
>Priority: Minor
>
> steps to recreate
> # go to http://localhost:8080/admin/connection/
> # create new connection and add this into extra field {code} {"verify":
> False} {code}
> # save the connection, and come back to UI on list of connection
> # click on + to see the extra field
> It will not show up, but this json string will work: {code} {"verify":
> "False"} {code}
> basically it does not appear when the value field of the json string is not
> quoted.
> looks like it throws an error {code} {models.py:594} ERROR - No JSON object
> could be decoded {code}
> in anycase, extra string gets stored into database, so no issue there.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (AIRFLOW-688) admin->connection extra value dont show up
Jayesh created AIRFLOW-688:
--
Summary: admin->connection extra value dont show up
Key: AIRFLOW-688
URL: https://issues.apache.org/jira/browse/AIRFLOW-688
Project: Apache Airflow
Issue Type: Bug
Components: ui
Reporter: Jayesh
Priority: Minor
steps to recreate
# go to http://localhost:8080/admin/connection/
# create new connection and add this into extra field {code} {"verify": False}
{code}
# save the connection, and come back to UI on list of connection
# click on + to see the extra field
It will not show up, but this json string will show up: {code} {"verify":
"False"} {code}
basically it does not appear when the value field of the json string is not
quoted.
in anycase, extra string gets stored into database, so no issue there.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (AIRFLOW-499) Add in capabilities to use atlassian tools with Airflow
[ https://issues.apache.org/jira/browse/AIRFLOW-499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jayesh reassigned AIRFLOW-499: -- Assignee: Jayesh > Add in capabilities to use atlassian tools with Airflow > --- > > Key: AIRFLOW-499 > URL: https://issues.apache.org/jira/browse/AIRFLOW-499 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Tony Doran >Assignee: Jayesh > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AIRFLOW-683) provide Jira Hook, Operator & Sensor
Jayesh created AIRFLOW-683: -- Summary: provide Jira Hook, Operator & Sensor Key: AIRFLOW-683 URL: https://issues.apache.org/jira/browse/AIRFLOW-683 Project: Apache Airflow Issue Type: New Feature Components: hooks, operators Reporter: Jayesh Assignee: Jayesh Task to add Jira Hook, Operator & Sensor so that user can interact with Jira system. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
