yihua commented on a change in pull request #1246: [HUDI-552] Fix the schema
mismatch in Row-to-Avro conversion
URL: https://github.com/apache/incubator-hudi/pull/1246#discussion_r368785363
##########
File path:
hudi-utilities/src/test/java/org/apache/hudi/utilities/TestHoodieDeltaStreamer.java
##########
@@ -620,6 +636,62 @@ public void testDistributedTestDataSource() {
Assert.assertEquals(1000, c);
}
+ private static void prepareParquetDFSFiles(int numRecords) throws
IOException {
+ String path = PARQUET_SOURCE_ROOT + "/1.parquet";
+ HoodieTestDataGenerator dataGenerator = new HoodieTestDataGenerator();
+ Helpers.saveParquetToDFS(Helpers.toGenericRecords(
+ dataGenerator.generateInserts("000", numRecords), dataGenerator), new
Path(path));
+ }
+
+ private void prepareParquetDFSSource(boolean useSchemaProvider, boolean
hasTransformer) throws IOException {
+ // Properties used for testing delta-streamer with Parquet source
+ TypedProperties parquetProps = new TypedProperties();
+ parquetProps.setProperty("include", "base.properties");
+ parquetProps.setProperty("hoodie.datasource.write.recordkey.field",
"_row_key");
+ parquetProps.setProperty("hoodie.datasource.write.partitionpath.field",
"not_there");
+ if (useSchemaProvider) {
+
parquetProps.setProperty("hoodie.deltastreamer.schemaprovider.source.schema.file",
dfsBasePath + "/source.avsc");
+ if (hasTransformer) {
+
parquetProps.setProperty("hoodie.deltastreamer.schemaprovider.source.schema.file",
dfsBasePath + "/target.avsc");
Review comment:
Good catch. I've found and fixed that in #1165.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
