nsivabalan commented on a change in pull request #4818:
URL: https://github.com/apache/hudi/pull/4818#discussion_r820280799
##########
File path:
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/testutils/SparkClientFunctionalTestHarness.java
##########
@@ -348,6 +356,21 @@ protected HoodieWriteConfig getConfig(Boolean autoCommit,
Boolean rollbackUsingM
.withRollbackUsingMarkers(rollbackUsingMarkers);
}
+ protected Dataset<Row> toDataset(List<HoodieRecord> records, Schema schema) {
+ List<GenericRecord> avroRecords = records.stream()
+ .map(r -> {
+ HoodieRecordPayload payload = (HoodieRecordPayload) r.getData();
+ try {
+ return (GenericRecord) payload.getInsertValue(schema).get();
+ } catch (IOException e) {
+ throw new HoodieIOException("Failed to extract Avro payload", e);
+ }
+ })
+ .collect(Collectors.toList());
+ JavaRDD<GenericRecord> jrdd = jsc.parallelize(avroRecords, 2);
+ return AvroConversionUtils.createDataFrame(jrdd.rdd(),
HoodieTestDataGenerator.AVRO_SCHEMA.toString(), spark);
Review comment:
shouldn't this be "schema", the 2nd argument in the method ?
##########
File path:
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/testutils/SparkClientFunctionalTestHarness.java
##########
@@ -348,6 +355,21 @@ protected HoodieWriteConfig getConfig(Boolean autoCommit,
Boolean rollbackUsingM
.withRollbackUsingMarkers(rollbackUsingMarkers);
}
+ protected Dataset<Row> toDataset(List<HoodieRecord> records) {
Review comment:
is this addressed ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
