yihua commented on a change in pull request #1165: [HUDI-76] Add CSV Source
support for Hudi Delta Streamer
URL: https://github.com/apache/incubator-hudi/pull/1165#discussion_r384313736
##########
File path:
hudi-client/src/test/java/org/apache/hudi/common/HoodieTestDataGenerator.java
##########
@@ -74,20 +74,30 @@
public static final String[] DEFAULT_PARTITION_PATHS =
{DEFAULT_FIRST_PARTITION_PATH, DEFAULT_SECOND_PARTITION_PATH,
DEFAULT_THIRD_PARTITION_PATH};
public static final int DEFAULT_PARTITION_DEPTH = 3;
- public static final String TRIP_EXAMPLE_SCHEMA = "{\"type\": \"record\"," +
"\"name\": \"triprec\"," + "\"fields\": [ "
+ public static final String TRIP_SCHEMA_PREFIX = "{\"type\": \"record\"," +
"\"name\": \"triprec\"," + "\"fields\": [ "
+ "{\"name\": \"timestamp\",\"type\": \"double\"}," + "{\"name\":
\"_row_key\", \"type\": \"string\"},"
+ "{\"name\": \"rider\", \"type\": \"string\"}," + "{\"name\":
\"driver\", \"type\": \"string\"},"
+ "{\"name\": \"begin_lat\", \"type\": \"double\"}," + "{\"name\":
\"begin_lon\", \"type\": \"double\"},"
- + "{\"name\": \"end_lat\", \"type\": \"double\"}," + "{\"name\":
\"end_lon\", \"type\": \"double\"},"
- + "{\"name\": \"fare\",\"type\": {\"type\":\"record\",
\"name\":\"fare\",\"fields\": ["
- + "{\"name\": \"amount\",\"type\": \"double\"},{\"name\": \"currency\",
\"type\": \"string\"}]}},"
- + "{\"name\": \"_hoodie_is_deleted\", \"type\": \"boolean\",
\"default\": false} ]}";
+ + "{\"name\": \"end_lat\", \"type\": \"double\"}," + "{\"name\":
\"end_lon\", \"type\": \"double\"},";
+ public static final String TRIP_SCHEMA_SUFFIX = "{\"name\":
\"_hoodie_is_deleted\", \"type\": \"boolean\", \"default\": false} ]}";
+ public static final String FARE_NESTED_SCHEMA = "{\"name\":
\"fare\",\"type\": {\"type\":\"record\", \"name\":\"fare\",\"fields\": ["
+ + "{\"name\": \"amount\",\"type\": \"double\"},{\"name\": \"currency\",
\"type\": \"string\"}]}},";
+ public static final String FARE_FLATTENED_SCHEMA = "{\"name\": \"fare\",
\"type\": \"double\"},"
+ + "{\"name\": \"currency\", \"type\": \"string\"},";
+
+ public static final String TRIP_EXAMPLE_SCHEMA =
+ TRIP_SCHEMA_PREFIX + FARE_NESTED_SCHEMA + TRIP_SCHEMA_SUFFIX;
+ public static final String TRIP_FLATTENED_SCHEMA =
Review comment:
Yes, for CSV format, the nested schema is not well supported. So to test
CSV source, we need to generate the test CSV data with a flattened schema.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
