ranjanankur commented on issue #9355:
URL: https://github.com/apache/hudi/issues/9355#issuecomment-1696764242
Hi @the-other-tim-brown , @emkornfield
In the above article it is mentioned to use the `use-bq-manifest-file` flag
while running `BigQuerySyncTool` function to sync Hudi table with BigQuery
tables.
It is mentioned to use the `hudi-gcp-bundle-0.14.0-SNAPSHOT.jar`. We have
upgraded this JAR too but we are finding difficulty to add this
`use-bq-manifest-file` in the code.
Instead of using spark-summit to run this code, we just call
`BigQuerySyncTool` function from our code itself & we are confused where and
how to pass this flag.
Please let us know where to pass this flag.
```
def getBigQueryProps: Properties = {
val props = new Properties()
props.setProperty(BIGQUERY_SYNC_PROJECT_ID.key, projectId)
props.setProperty(BIGQUERY_SYNC_DATASET_NAME.key, datasetName)
props.setProperty(BIGQUERY_SYNC_DATASET_LOCATION.key, datasetLocation)
props.setProperty(BIGQUERY_SYNC_TABLE_NAME.key, tableName)
props.setProperty(BIGQUERY_SYNC_SOURCE_URI.key,
s"$tablePath/$firstLevelPartition=*")
props.setProperty(BIGQUERY_SYNC_SOURCE_URI_PREFIX.key, s"$tablePath/")
props.setProperty(BIGQUERY_SYNC_SYNC_BASE_PATH.key, tablePath)
props.setProperty(BIGQUERY_SYNC_PARTITION_FIELDS.key, partitionKey)
props.setProperty(META_SYNC_BASE_PATH.key(), tablePath)
props.setProperty(BIGQUERY_SYNC_USE_FILE_LISTING_FROM_METADATA.key,
"true")
props.setProperty(BIGQUERY_SYNC_ASSUME_DATE_PARTITIONING.key, "false")
props
}
new BigQuerySyncTool(getBigQueryProps).syncHoodieTable()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
