phanikumv commented on code in PR #28874:
URL: https://github.com/apache/airflow/pull/28874#discussion_r1067780016
##########
airflow/providers/apache/hive/provider.yaml:
##########
@@ -56,6 +56,7 @@ dependencies:
# the sasl library anyway (and there sasl library version is not relevant)
- sasl>=0.3.1; python_version>="3.9"
- thrift>=0.9.2
+ - impyla
Review Comment:
We have written our own asyncio method as shown below, because impyla give
us a handle immediately after submitting the query
```
async def partition_exists(self, table: str, schema: str, partition: str,
polling_interval: float) -> str:
"""
Checks for the existence of a partition in the given hive table.
:param table: table in hive where the partition exists.
:param schema: database where the hive table exists
:param partition: partition to check for in given hive database and
hive table.
:param polling_interval: polling interval in seconds to sleep
between checks
"""
client = self.get_hive_client()
cursor = client.cursor()
query = f"show partitions {schema}.{table} partition({partition})"
cursor.execute_async(query)
while cursor.is_executing():
await asyncio.sleep(polling_interval)
results = cursor.fetchall()
if len(results) == 0:
return "failure"
return "success"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
