lokeshj1703 commented on code in PR #13436: URL: https://github.com/apache/hudi/pull/13436#discussion_r2149878911
########## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/SecondaryIndexStats.java: ########## @@ -0,0 +1,48 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hudi.client; + +/** + * Class is used to hold secondary index metadata updates. These updates are generated from + * various write handles during write. + */ +public class SecondaryIndexStats { + + private String recordKey; Review Comment: Addressed ########## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieCreateHandle.java: ########## @@ -62,7 +65,7 @@ public class HoodieCreateHandle<T, I, K, O> extends HoodieWriteHandle<T, I, K, O protected long recordsDeleted = 0; private Map<String, HoodieRecord<T>> recordMap; private boolean useWriterSchema = false; - private final boolean preserveMetadata; + private boolean preserveMetadata; Review Comment: Removed this change. I had addded it for fixing one of the tests. Not required anymore. ########## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieCreateHandle.java: ########## @@ -173,6 +178,23 @@ record = record.prependMetaFields(schema, writeSchemaWithMetaFields, new Metadat } } + private void trackMetadataIndexStats(HoodieRecord record) { + if (!config.isSecondaryIndexEnabled() || secondaryIndexDefns.isEmpty() || !config.isMetadataStreamingWritesEnabled(hoodieTable.getMetaClient().getTableConfig().getTableVersion())) { Review Comment: Moved it ########## hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/io/BaseTestHandle.java: ########## @@ -55,13 +55,15 @@ public void tearDown() throws Exception { } Pair<WriteStatus, List<HoodieRecord>> createParquetFile(HoodieWriteConfig config, HoodieTable table, String partitionPath, - String fileId, String instantTime, HoodieTestDataGenerator dataGenerator) { + String fileId, String instantTime, HoodieTestDataGenerator dataGenerator, + boolean preserveMetadata) { List<HoodieRecord> records = dataGenerator.generateInserts(instantTime, 100); Map<String, HoodieRecord> recordMap = new HashMap<>(); for (int i = 0; i < records.size(); i++) { recordMap.put(String.valueOf(i), records.get(i)); } HoodieCreateHandle handle = new HoodieCreateHandle(config, instantTime, table, partitionPath, fileId, recordMap, new LocalTaskContextSupplier()); + handle.setPreserveMetadata(preserveMetadata); Review Comment: Removed this change. I had added it for fixing one of the tests. Not required anymore. ########## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieAppendHandle.java: ########## @@ -166,6 +175,13 @@ public HoodieAppendHandle(HoodieWriteConfig config, String instantTime, HoodieTa this.baseFileInstantTimeOfPositions = shouldWriteRecordPositions ? getBaseFileInstantTimeOfPositions() : Option.empty(); + if (!hoodieTable.isMetadataTable() && config.isSecondaryIndexEnabled() && isStreamingWriteToMetadataEnabled) { Review Comment: While fetching the record key from reader context, we need the record metadata field. Since when `populateMetaFields` is false, we do not have the record key metadata field and read fails with NPE. ``` Caused by: java.lang.NullPointerException at org.apache.hudi.common.engine.HoodieReaderContext.lambda$metadataKeyExtractor$2(HoodieReaderContext.java:340) at org.apache.hudi.common.engine.HoodieReaderContext.getRecordKey(HoodieReaderContext.java:336) at org.apache.hudi.common.table.read.BufferedRecord.forRecordWithContext(BufferedRecord.java:71) at org.apache.hudi.common.table.read.KeyBasedFileGroupRecordBuffer.processDataBlock(KeyBasedFileGroupRecordBuffer.java:82) at org.apache.hudi.common.table.log.BaseHoodieLogRecordReader.processQueuedBlocksForInstant(BaseHoodieLogRecordReader.java:728) at org.apache.hudi.common.table.log.BaseHoodieLogRecordReader.scanInternalV1(BaseHoodieLogRecordReader.java:378) ... 136 more ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
