vingov opened a new pull request, #5659:
URL: https://github.com/apache/hudi/pull/5659
## What is the purpose of the pull request
This pull request adds the Snowflake Sync feature, this is a requirement to
read Hudi tables on the Snowflake warehouse.
## Brief change log
- *Added SnowflakeSyncTool to sync Hudi dataset to Snowflake cloud data
warehouse.*
## Verify this pull request
This change added tests and can be verified as follows:
*(example:)*
- *Manually verified the change by running a job locally.*
1. Use the docker demo steps to set up the stock_ticks_cow table on google
cloud storage.
2. Create showflake_profile.properties file with the following configs:
`# snowflake account details
URL = https://el36393.us-central1.gcp.snowflakecomputing.com:443
USER = hudidemo
PRIVATE_KEY_FILE = /Users/vinothg/.ssh/rsa_key.p8
ROLE = ACCOUNTADMIN
WAREHOUSE = COMPUTE_WH
DB = hudi
SCHEMA = dwh`
3. Run the following command to sync the table to snowflake
`java -cp
guava-25.1-jre.jar:hadoop-auth-2.4.1.jar:commons-configuration-1.9.jar:commons-collections-3.2.1.jar:guava-r05.jar:woodstox-core-5.3.0.jar:stax2-api-4.2.jar:hadoop-common-2.7.3.jar:hudi-spark-bundle_2.12-0.12.0-SNAPSHOT.jar:hudi-snowflake-bundle-0.12.0-SNAPSHOT-jar-with-dependencies.jar:gcs-connector-hadoop2-latest.jar
org.apache.hudi.snowflake.sync.SnowflakeSyncTool --properties-file
~/snowflake_profile.properties --base-path gs://hudi-demo/stock_ticks_cow
--table-name stock_ticks_cow --storage-integration hudi_demo_int
--partitioned-by "date" --partition-extract-expr "\"date\" date as
to_date(substr(metadata\$filename, 22, 10), 'YYYY-MM-DD')"`
## Committer checklist
- [x] Has a corresponding JIRA in PR title & commit
- [x] Commit message is descriptive of the change
- [x] CI is green
- [ ] Necessary doc changes done or have another open PR
- [x] For large changes, please consider breaking it into sub-tasks under
an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
