BhuviTheDataGuy opened a new issue #11905:
URL: https://github.com/apache/airflow/issues/11905
In the latest BigQueryCreateEmptyTableOperator(from providers) I tried to
create an empty table. And I wanted to use table resources (like creating a
partition), but its not considering that.
**Code to reproduce:**
```py
from airflow import DAG
import json
from datetime import datetime, timedelta
from airflow.utils.dates import days_ago
from airflow.operators.python_operator import PythonOperator
from airflow.providers.google.cloud.operators.bigquery import
BigQueryCreateEmptyTableOperator
# Add args and Dag
default_args = {
'owner': 'DBteam',
'depends_on_past': False,
'start_date': days_ago(1),
'retries': 0
}
dag = DAG(
'bqtest',
default_args=default_args,
description='create the table structure',
schedule_interval=None,
catchup=False,
max_active_runs=1
)
resource="""{
"TimePartitioning": {
"type": "DAY",
"field": "created_at"
}
}"""
schema="""[{"mode": "REQUIRED", "name": "code", "type": "STRING"},
{"mode": "REQUIRED", "name": "created_at", "type": "DATETIME"},
{"mode": "REQUIRED", "name": "service_name", "type": "STRING"}]"""
def bq_create(**kwargs):
table_schema = 'bhuvi'
table_name = 'sampletable'
create = BigQueryCreateEmptyTableOperator (
task_id='create_bq_{}'.format(table_name),
project_id = 'myproject',
dataset_id=table_schema,
table_id=table_name,
schema_fields=json.loads(schema),
bigquery_conn_id='bigquery_default',
table_resource =json.loads(resource)
)
create.execute(context=kwargs)
bqcreate = PythonOperator(
task_id="bqcreate",
python_callable=bq_create,
provide_context=True,
dag=dag
)
bqcreate
```
The table got created, there is no partition. (I think we have a sperate
parameter for time partition, but I want to use this table resource option)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
