James601232 edited a comment on issue #3963:
URL: https://github.com/apache/hudi/issues/3963#issuecomment-966082120
> @James601232 I'm not sure if i get it. In the code `client` it does not do
any upsert/insert/bulkinsert operation, it's just to init a hudi table, which
it did by creating `.hoodie/`. You need to use the client to write data to the
table as well if you intend to do so.
yes. you are right. so i add insert method in my codes. now create table
is ok. but no data insert into table. my codes as following:
```
package com.tx.bigdata;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hudi.client.HoodieJavaWriteClient;
import org.apache.hudi.client.common.HoodieJavaEngineContext;
import org.apache.hudi.common.fs.FSUtils;
import org.apache.hudi.common.model.HoodieAvroPayload;
import org.apache.hudi.common.model.HoodieKey;
import org.apache.hudi.common.model.HoodieRecord;
import org.apache.hudi.common.model.HoodieTableType;
import org.apache.hudi.common.table.HoodieTableMetaClient;
import org.apache.hudi.config.HoodieCompactionConfig;
import org.apache.hudi.config.HoodieIndexConfig;
import org.apache.hudi.config.HoodieWriteConfig;
import org.apache.hudi.examples.common.HoodieExampleDataGenerator;
import org.apache.hudi.index.HoodieIndex;
import org.apache.log4j.LogManager;
import org.apache.log4j.Logger;
import java.util.ArrayList;
import java.util.List;
import java.util.stream.Collectors;
public class HoodieJavaWriteClientExample {
private static final Logger LOG =
LogManager.getLogger(HoodieJavaWriteClientExample.class);
private static String tableType = HoodieTableType.COPY_ON_WRITE.name();
public static void main(String[] args) throws Exception {
// if (args.length < 2) {
// System.err.println("Usage: HoodieJavaWriteClientExample <tablePath>
<tableName>");
// System.exit(1);
// }
// String tablePath = args[0];
// String tableName = args[1];
String tablePath = "/home/work/hudi_catalog4";
String tableName = "hudi_clyang_table";
// Generator of some records to be loaded in.
HoodieExampleDataGenerator<HoodieAvroPayload> dataGen = new
HoodieExampleDataGenerator<>();
Configuration hadoopConf = new Configuration();
hadoopConf.addResource(new Path("core-site.xml"));
hadoopConf.addResource(new Path("hdfs-site.xml"));
System.setProperty("HADOOP_USER_NAME", "work");
// initialize the table, if not done already
Path path = new Path(tablePath);
FileSystem fs = FSUtils.getFs(tablePath, hadoopConf);
if (!fs.exists(path)) {
HoodieTableMetaClient.withPropertyBuilder()
.setTableType(tableType)
.setTableName(tableName)
.setPayloadClassName(HoodieAvroPayload.class.getName())
.initTable(hadoopConf, tablePath);
}
// Create the write client to write some records in
HoodieWriteConfig cfg =
HoodieWriteConfig.newBuilder().withPath(tablePath)
.withSchema(HoodieExampleDataGenerator.TRIP_EXAMPLE_SCHEMA).withParallelism(2,
2)
.withDeleteParallelism(2).forTable(tableName)
.withIndexConfig(HoodieIndexConfig.newBuilder().withIndexType(HoodieIndex.IndexType.INMEMORY).build())
.withCompactionConfig(HoodieCompactionConfig.newBuilder().archiveCommitsWith(20,
30).build()).build();
HoodieJavaWriteClient<HoodieAvroPayload> client =
new HoodieJavaWriteClient<>(new HoodieJavaEngineContext(hadoopConf),
cfg);
// inserts
String newCommitTime = client.startCommit();
LOG.info("Starting commit " + newCommitTime);
List<HoodieRecord<HoodieAvroPayload>> records =
dataGen.generateInserts(newCommitTime, 10);
List<HoodieRecord<HoodieAvroPayload>> recordsSoFar = new
ArrayList<>(records);
List<HoodieRecord<HoodieAvroPayload>> writeRecords =
recordsSoFar.stream().map(r -> new
HoodieRecord<HoodieAvroPayload>(r)).collect(Collectors.toList());
client.insert(writeRecords, newCommitTime);
client.close();
}
}
```
these codes from hudi githup test project
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
