manishmalhotrawork opened a new issue #116: Spark based functional test-cases URL: https://github.com/apache/incubator-iceberg/issues/116 On the google groups, saw many users ask about the usage of Iceberg. ( Datasource , Table API etc.) So think this would be useful for us to test and new users to understand how to use Iceberg for different scenarios. So, wrote few functional test-cases to test using spark and iceberg data-source and Table API's. In general flow is like this: 1. read local resources/<data> file. 2. convert it to RDD by running mapper function, and converting epoch to day time. 3. write RDD/DF to local folder as parquet data partitioned by event_date. 4. create or get local iceberg table. 5. add partitions/files/RDD to iceberg table using Iceberg's Table APIs / Datasource. 6. read data from Parquet. 7. read data from iceberg table. 8. compare results.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
