[GitHub] Fokko commented on a change in pull request #3730: [AIRFLOW-2882] Add import and export for pool cli using JSON
Fokko commented on a change in pull request #3730: [AIRFLOW-2882] Add import
and export for pool cli using JSON
URL: https://github.com/apache/incubator-airflow/pull/3730#discussion_r211105245
##
File path: airflow/bin/cli.py
##
@@ -283,6 +292,43 @@ def _tabulate(pools):
log.info(_tabulate(pools=pools))
+def pool_import_helper(filepath):
+with open(filepath, 'r') as poolfile:
+pl = poolfile.read()
+try:
+d = json.loads(pl)
+except Exception:
+print("Invalid pool file.")
+else:
+try:
+pools = []
+n = 0
+for k, v in d.items():
+if isinstance(v, dict) and len(v) == 2:
+pools.append(api_client.create_pool(name=k,
+slots=v["slots"],
+
description=v["description"]))
+n += 1
+else:
+pass
+except Exception:
+pass
+finally:
+print("{} of {} pool(s) successfully updated.".format(n, len(d)))
+return pools
+
+
+def pool_export_helper(filepath):
+pool_dict = {}
+pools = api_client.get_pools()
+for pool in pools:
+pool_dict[pool[0]] = {"slots": pool[1], "description": pool[2]}
Review comment:
Yes, that would be my preference.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] Fokko commented on a change in pull request #3730: [AIRFLOW-2882] Add import and export for pool cli using JSON
Fokko commented on a change in pull request #3730: [AIRFLOW-2882] Add import
and export for pool cli using JSON
URL: https://github.com/apache/incubator-airflow/pull/3730#discussion_r210185860
##
File path: airflow/bin/cli.py
##
@@ -283,6 +292,43 @@ def _tabulate(pools):
log.info(_tabulate(pools=pools))
+def pool_import_helper(filepath):
+with open(filepath, 'r') as poolfile:
+pl = poolfile.read()
+try:
+d = json.loads(pl)
+except Exception:
+print("Invalid pool file.")
Review comment:
Maybe help the user a bit here. Tell them to check the validity of the json.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] Fokko commented on a change in pull request #3730: [AIRFLOW-2882] Add import and export for pool cli using JSON
Fokko commented on a change in pull request #3730: [AIRFLOW-2882] Add import
and export for pool cli using JSON
URL: https://github.com/apache/incubator-airflow/pull/3730#discussion_r210185691
##
File path: airflow/bin/cli.py
##
@@ -283,6 +292,43 @@ def _tabulate(pools):
log.info(_tabulate(pools=pools))
+def pool_import_helper(filepath):
+with open(filepath, 'r') as poolfile:
+pl = poolfile.read()
+try:
+d = json.loads(pl)
+except Exception:
+print("Invalid pool file.")
+else:
+try:
+pools = []
+n = 0
Review comment:
We don't need this `n` counter, we can also do `len(pools)`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] Fokko commented on a change in pull request #3730: [AIRFLOW-2882] Add import and export for pool cli using JSON
Fokko commented on a change in pull request #3730: [AIRFLOW-2882] Add import
and export for pool cli using JSON
URL: https://github.com/apache/incubator-airflow/pull/3730#discussion_r210185424
##
File path: airflow/bin/cli.py
##
@@ -283,6 +292,43 @@ def _tabulate(pools):
log.info(_tabulate(pools=pools))
+def pool_import_helper(filepath):
+with open(filepath, 'r') as poolfile:
+pl = poolfile.read()
+try:
+d = json.loads(pl)
+except Exception:
+print("Invalid pool file.")
+else:
+try:
+pools = []
+n = 0
+for k, v in d.items():
+if isinstance(v, dict) and len(v) == 2:
+pools.append(api_client.create_pool(name=k,
+slots=v["slots"],
+
description=v["description"]))
+n += 1
+else:
+pass
+except Exception:
+pass
+finally:
+print("{} of {} pool(s) successfully updated.".format(n, len(d)))
+return pools
+
+
+def pool_export_helper(filepath):
+pool_dict = {}
+pools = api_client.get_pools()
Review comment:
https://github.com/apache/incubator-airflow/blob/0ba510e238ebb197bce9afccdb06b88871c4a7e6/airflow/api/client/api_client.py#L53-L55
樂 Did you test this?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] Fokko commented on a change in pull request #3730: [AIRFLOW-2882] Add import and export for pool cli using JSON
Fokko commented on a change in pull request #3730: [AIRFLOW-2882] Add import
and export for pool cli using JSON
URL: https://github.com/apache/incubator-airflow/pull/3730#discussion_r210185544
##
File path: airflow/bin/cli.py
##
@@ -283,6 +292,43 @@ def _tabulate(pools):
log.info(_tabulate(pools=pools))
+def pool_import_helper(filepath):
+with open(filepath, 'r') as poolfile:
+pl = poolfile.read()
+try:
+d = json.loads(pl)
+except Exception:
+print("Invalid pool file.")
+else:
+try:
+pools = []
+n = 0
+for k, v in d.items():
+if isinstance(v, dict) and len(v) == 2:
+pools.append(api_client.create_pool(name=k,
+slots=v["slots"],
+
description=v["description"]))
+n += 1
+else:
+pass
+except Exception:
+pass
+finally:
+print("{} of {} pool(s) successfully updated.".format(n, len(d)))
+return pools
+
+
+def pool_export_helper(filepath):
+pool_dict = {}
+pools = api_client.get_pools()
+for pool in pools:
+pool_dict[pool[0]] = {"slots": pool[1], "description": pool[2]}
Review comment:
This code makes me a bit sad, I would expect a more descriptive response
from the API.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] Fokko commented on a change in pull request #3730: [AIRFLOW-2882] Add import and export for pool cli using JSON
Fokko commented on a change in pull request #3730: [AIRFLOW-2882] Add import
and export for pool cli using JSON
URL: https://github.com/apache/incubator-airflow/pull/3730#discussion_r209166353
##
File path: tests/cli/test_cli.py
##
@@ -165,3 +166,38 @@ def test_local_run(self):
ti.refresh_from_db()
state = ti.current_state()
self.assertEqual(state, State.SUCCESS)
+
+def test_cli_pool_import_export(self):
+pool_config_input = {
+"s3_pool": {
+"description": "This is my test s3_pool",
+"slots": 5
+},
+"s3_pool2": {
+"description": "This is my test s3_pool",
+"slots": 8
+}
+}
+with open('pool_import.json', mode='w', encoding='utf-8') as f:
+json.dump(pool_config_input, f)
+process_import = psutil.Popen(["airflow", "pool", "-i",
"pool_import.json"])
+sleep(3) # wait for webserver to start
Review comment:
Please poll if the process is available. This makes the test a bit flaky.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
