[ 
https://issues.apache.org/jira/browse/AIRFLOW-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Soundar updated AIRFLOW-2141:
-----------------------------
    Description: 
I'm trying to create Airflow variables using a json file. I am trying to import 
airflow variables using UI(webserver) when I upload the json file I get this 
error "Missing file or syntax error" and when I try to upload using airflow cli 
not all the variables gets uploaded properly. The catch is that I have a list 
of dictionary in my json file, say
 ex:
 {
 "demo_archivedir": "/home/ubuntu/folders/archive",
 "demo_filepattern": [

{ "id": "reference", "pattern": "Sample Data.xlsx" }

,

{ "id": "sale", "pattern": "Sales.xlsx" }

],
 "demo_sourcepath": "/home/ubuntu/folders/input",
 "demo_workdir": "/home/ubuntu/folders/working"
 }

I've attached two images

img1. Using airflow variables cli command I was able to create partial 
variables from my json file(airflow_cli.png)img2. After inserting logs in the 
"airflow/bin/cli.py" file, I got this error. (airflow_cli2_crop.png)

The thing is I gave this value through the Admin UI one by one and it worked. 
Then I exported those same variable using "airflow variables" cli command and 
tried importing them, still it failed and the above mentioned error still 
occurs.

Note:
   I am using Python 3.5 with Airflow version 1.8

The stack trace is as follows

.compute-1.amazonaws.com:22] out: 0 of 4 variables successfully updated.
.compute-1.amazonaws.com:22] out: Traceback (most recent call last):
.compute-1.amazonaws.com:22] out:   File "/home/ubuntu/Env/bin/airflow", line 
28, in <module>
.compute-1.amazonaws.com:22] out:     args.func(args)
.compute-1.amazonaws.com:22] out:   File 
"/home/ubuntu/Env/lib/python3.5/site-packages/airflow/bin/cli.py", line 242, in 
variables
.compute-1.amazonaws.com:22] out:     import_helper(imp)
.compute-1.amazonaws.com:22] out:   File 
"/home/ubuntu/Env/lib/python3.5/site-packages/airflow/bin/cli.py", line 273, in 
import_helper
.compute-1.amazonaws.com:22] out:     Variable.set(k, v)
.compute-1.amazonaws.com:22] out:   File 
"/home/ubuntu/Env/lib/python3.5/site-packages/airflow/utils/db.py", line 53, in 
wrapper
.compute-1.amazonaws.com:22] out:     result = func(*args, **kwargs)
.compute-1.amazonaws.com:22] out:   File 
"/home/ubuntu/Env/lib/python3.5/site-packages/airflow/models.py", line 3615, in 
set
.compute-1.amazonaws.com:22] out:     session.add(Variable(key=key, 
val=stored_value))
.compute-1.amazonaws.com:22] out:   File "<string>", line 4, in __init__
.compute-1.amazonaws.com:22] out:   File 
"/home/ubuntu/Env/lib/python3.5/site-packages/sqlalchemy/orm/state.py", line 
417, in _initialize_instance
.compute-1.amazonaws.com:22] out:     manager.dispatch.init_failure(self, args, 
kwargs)
.compute-1.amazonaws.com:22] out:   File 
"/home/ubuntu/Env/lib/python3.5/site-packages/sqlalchemy/util/langhelpers.py", 
line 66, in __exit__
.compute-1.amazonaws.com:22] out:     compat.reraise(exc_type, exc_value, 
exc_tb)
.compute-1.amazonaws.com:22] out:   File 
"/home/ubuntu/Env/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line 
187, in reraise
.compute-1.amazonaws.com:22] out:     raise value
.compute-1.amazonaws.com:22] out:   File 
"/home/ubuntu/Env/lib/python3.5/site-packages/sqlalchemy/orm/state.py", line 
414, in _initialize_instance
.compute-1.amazonaws.com:22] out:     return manager.original_init(*mixed[1:], 
**kwargs)
.compute-1.amazonaws.com:22] out:   File 
"/home/ubuntu/Env/lib/python3.5/site-packages/sqlalchemy/ext/declarative/base.py",
 line 700, in _declarative_constructor
.compute-1.amazonaws.com:22] out:     setattr(self, k, kwargs[k])
compute-1.amazonaws.com:22] out:   File "<string>", line 1, in __set__
.compute-1.amazonaws.com:22] out:   File 
"/home/ubuntu/Env/lib/python3.5/site-packages/airflow/models.py", line 3550, in 
set_val
.compute-1.amazonaws.com:22] out:     self._val = FERNET.encrypt(bytes(value, 
'utf-8')).decode()
.compute-1.amazonaws.com:22] out: TypeError: encoding without a string argument
.compute-1.amazonaws.com:22] out:

  was:
I'm trying to create Airflow variables using a json file. I am trying to import 
airflow variables using UI(webserver) when I upload the json file I get this 
error "Missing file or syntax error" and when I try to upload using airflow cli 
not all the variables gets uploaded properly. The catch is that I have a list 
of dictionary in my json file, say
 ex:
 {
 "demo_archivedir": "/home/ubuntu/folders/archive",
 "demo_filepattern": [

{ "id": "reference", "pattern": "Sample Data.xlsx" }

,

{ "id": "sale", "pattern": "Sales.xlsx" }

],
 "demo_sourcepath": "/home/ubuntu/folders/input",
 "demo_workdir": "/home/ubuntu/folders/working"
 }

I've attached two images 

img1. Using airflow variables cli command I was able to create partial 
variables from my json file(airflow_cli.png)img2. After inserting logs in the 
"airflow/bin/cli.py" file, I got this error. (airflow_cli2_crop.png)

The thing is I gave this value through the Admin UI one by one and it worked. 
Then I exported those same variable using "airflow variables" cli command and 
tried importing them, still it failed and the above mentioned error still 
occurs.

Note:
  I am using Python 3.5 with Airflow version 1.8


> Cannot create airflow variables when there is a list of dictionary as a value
> -----------------------------------------------------------------------------
>
>                 Key: AIRFLOW-2141
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2141
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: aws
>    Affects Versions: Airflow 1.8
>            Reporter: Soundar
>            Priority: Major
>              Labels: beginner, newbie
>         Attachments: airflow_cli.png, airflow_cli2_crop.png
>
>
> I'm trying to create Airflow variables using a json file. I am trying to 
> import airflow variables using UI(webserver) when I upload the json file I 
> get this error "Missing file or syntax error" and when I try to upload using 
> airflow cli not all the variables gets uploaded properly. The catch is that I 
> have a list of dictionary in my json file, say
>  ex:
>  {
>  "demo_archivedir": "/home/ubuntu/folders/archive",
>  "demo_filepattern": [
> { "id": "reference", "pattern": "Sample Data.xlsx" }
> ,
> { "id": "sale", "pattern": "Sales.xlsx" }
> ],
>  "demo_sourcepath": "/home/ubuntu/folders/input",
>  "demo_workdir": "/home/ubuntu/folders/working"
>  }
> I've attached two images
> img1. Using airflow variables cli command I was able to create partial 
> variables from my json file(airflow_cli.png)img2. After inserting logs in the 
> "airflow/bin/cli.py" file, I got this error. (airflow_cli2_crop.png)
> The thing is I gave this value through the Admin UI one by one and it worked. 
> Then I exported those same variable using "airflow variables" cli command and 
> tried importing them, still it failed and the above mentioned error still 
> occurs.
> Note:
>    I am using Python 3.5 with Airflow version 1.8
> The stack trace is as follows
> .compute-1.amazonaws.com:22] out: 0 of 4 variables successfully updated.
> .compute-1.amazonaws.com:22] out: Traceback (most recent call last):
> .compute-1.amazonaws.com:22] out:   File "/home/ubuntu/Env/bin/airflow", line 
> 28, in <module>
> .compute-1.amazonaws.com:22] out:     args.func(args)
> .compute-1.amazonaws.com:22] out:   File 
> "/home/ubuntu/Env/lib/python3.5/site-packages/airflow/bin/cli.py", line 242, 
> in variables
> .compute-1.amazonaws.com:22] out:     import_helper(imp)
> .compute-1.amazonaws.com:22] out:   File 
> "/home/ubuntu/Env/lib/python3.5/site-packages/airflow/bin/cli.py", line 273, 
> in import_helper
> .compute-1.amazonaws.com:22] out:     Variable.set(k, v)
> .compute-1.amazonaws.com:22] out:   File 
> "/home/ubuntu/Env/lib/python3.5/site-packages/airflow/utils/db.py", line 53, 
> in wrapper
> .compute-1.amazonaws.com:22] out:     result = func(*args, **kwargs)
> .compute-1.amazonaws.com:22] out:   File 
> "/home/ubuntu/Env/lib/python3.5/site-packages/airflow/models.py", line 3615, 
> in set
> .compute-1.amazonaws.com:22] out:     session.add(Variable(key=key, 
> val=stored_value))
> .compute-1.amazonaws.com:22] out:   File "<string>", line 4, in __init__
> .compute-1.amazonaws.com:22] out:   File 
> "/home/ubuntu/Env/lib/python3.5/site-packages/sqlalchemy/orm/state.py", line 
> 417, in _initialize_instance
> .compute-1.amazonaws.com:22] out:     manager.dispatch.init_failure(self, 
> args, kwargs)
> .compute-1.amazonaws.com:22] out:   File 
> "/home/ubuntu/Env/lib/python3.5/site-packages/sqlalchemy/util/langhelpers.py",
>  line 66, in __exit__
> .compute-1.amazonaws.com:22] out:     compat.reraise(exc_type, exc_value, 
> exc_tb)
> .compute-1.amazonaws.com:22] out:   File 
> "/home/ubuntu/Env/lib/python3.5/site-packages/sqlalchemy/util/compat.py", 
> line 187, in reraise
> .compute-1.amazonaws.com:22] out:     raise value
> .compute-1.amazonaws.com:22] out:   File 
> "/home/ubuntu/Env/lib/python3.5/site-packages/sqlalchemy/orm/state.py", line 
> 414, in _initialize_instance
> .compute-1.amazonaws.com:22] out:     return 
> manager.original_init(*mixed[1:], **kwargs)
> .compute-1.amazonaws.com:22] out:   File 
> "/home/ubuntu/Env/lib/python3.5/site-packages/sqlalchemy/ext/declarative/base.py",
>  line 700, in _declarative_constructor
> .compute-1.amazonaws.com:22] out:     setattr(self, k, kwargs[k])
> compute-1.amazonaws.com:22] out:   File "<string>", line 1, in __set__
> .compute-1.amazonaws.com:22] out:   File 
> "/home/ubuntu/Env/lib/python3.5/site-packages/airflow/models.py", line 3550, 
> in set_val
> .compute-1.amazonaws.com:22] out:     self._val = FERNET.encrypt(bytes(value, 
> 'utf-8')).decode()
> .compute-1.amazonaws.com:22] out: TypeError: encoding without a string 
> argument
> .compute-1.amazonaws.com:22] out:



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to