szknb commented on issue #6530:
URL: https://github.com/apache/hudi/issues/6530#issuecomment-1231068629
@nsivabalan
`public class HudiExample {
private static final Logger LOG =
LogManager.getLogger(HudiExample.class);
private static String tableType = HoodieTableType.COPY_ON_WRITE.name();
public static void main(String[] args) throws Exception {
String tablePath = "hdfs://haruna/home/xxx/xxx/hudi";
String tableName = "hudi-test";
SparkConf sparkConf =
HoodieExampleSparkUtils.defaultSparkConf("hoodie-client-example");
try (JavaSparkContext jsc = new JavaSparkContext(sparkConf)) {
// Generator of some records to be loaded in.
HoodieExampleDataGenerator<HoodieAvroPayload> dataGen = new
HoodieExampleDataGenerator<>();
// initialize the table, if not done already
Path path = new Path(tablePath);
FileSystem fs = FSUtils.getFs(tablePath,
jsc.hadoopConfiguration());
if (!fs.exists(path)) {
HoodieTableMetaClient.initTableType(jsc.hadoopConfiguration(), tablePath,
new HoodieTableConfig.Builder()
.withTableType(HoodieTableType.valueOf(tableType))
.withTableName(tableName)
.withPayloadClassName(HoodieTableType.valueOf(tableType),
HoodieAvroPayload.class.getName()).build());
}
// Create the write client to write some records in
HoodieWriteConfig cfg = HoodieWriteConfig
.newBuilder()
.withPath(tablePath)
.withSchema(HoodieExampleDataGenerator.TRIP_EXAMPLE_SCHEMA)
.withParallelism(2, 2)
.withDeleteParallelism(2)
.forTable(tableName)
.withIndexConfig(HoodieIndexConfig.newBuilder().withIndexType(HoodieIndex.IndexType.BLOOM).build())
.withCompactionConfig(HoodieCompactionConfig.newBuilder().archiveCommitsWith(20,
30).build()).build();
SparkRDDWriteClient<HoodieAvroPayload> client = new
SparkRDDWriteClient<>(new HoodieSparkEngineContext(jsc), cfg);
// inserts
String newCommitTime = client.startCommit();
LOG.info("Starting commit " + newCommitTime);
List<HoodieRecord<HoodieAvroPayload>> records =
dataGen.generateInserts(newCommitTime, 10);
List<HoodieRecord<HoodieAvroPayload>> recordsSoFar = new
ArrayList<>(records);
JavaRDD<HoodieRecord<HoodieAvroPayload>> writeRecords =
jsc.parallelize(records, 1);
client.upsert(writeRecords, newCommitTime);
LOG.info("insert finished");
}
}
}`
the HoodieExampleDataGenerator is:
org.apache.hudi.examples.common.HoodieExampleDataGenerator;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
