pankajastro commented on code in PR #38423:
URL: https://github.com/apache/airflow/pull/38423#discussion_r1552034125
##########
airflow/providers/elasticsearch/log/es_task_handler.py:
##########
@@ -213,6 +218,21 @@ def format_url(host: str) -> str:
return host
+ def _get_index_patterns(self, ti: TaskInstance | None) -> str | None:
+ """
+ Get index patterns by calling index_patterns_callable or None.
+
+ :param ti: A TaskInstance object or None.
+ """
+ if self.index_patterns_callable:
+ self.log.debug("Using %s index_patterns_callable",
self.index_patterns_callable)
+ module_path, index_pattern_function =
self.index_patterns_callable.rsplit(".", 1)
+ module = importlib.import_module(module_path)
+ index_pattern_callable_obj = getattr(module,
index_pattern_function)
+ return index_pattern_callable_obj(ti)
Review Comment:
I think if `index_patterns_callable` is configured incorrectly then worth
failing for better visibility or do want to just catch and log it instead of
crashing?
##########
airflow/providers/elasticsearch/log/es_task_handler.py:
##########
@@ -213,6 +218,21 @@ def format_url(host: str) -> str:
return host
+ def _get_index_patterns(self, ti: TaskInstance | None) -> str | None:
Review Comment:
depend on what callable return. I'm assuming that the callable will return a
string so this function will return a string either by calling
index_patterns_callable or configured index_patterns. or do you think callable
can return None too?
##########
airflow/providers/elasticsearch/log/es_task_handler.py:
##########
@@ -153,6 +154,9 @@ def __init__(
host: str = "http://localhost:9200";,
frontend: str = "localhost:5601",
index_patterns: str | None = conf.get("elasticsearch",
"index_patterns", fallback="_all"),
Review Comment:
do you mean to suggest for `index_patterns_callable` param? if not then if
we remove fallback then it would be breaking change right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
