ann created HUDI-1446:
-------------------------
Summary: Support skip bootstrapIndex's init in abstract fs view
init
Key: HUDI-1446
URL: https://issues.apache.org/jira/browse/HUDI-1446
Project: Apache Hudi
Issue Type: Improvement
Reporter: ann
the bootstrapIndex default is HFileBootstrapIndex. If program doesn't include
hbase pom, it would throw a exception as follows:
Caused by: java.lang.NoClassDefFoundError:
org/apache/hadoop/hbase/util/BytesCaused by: java.lang.NoClassDefFoundError:
org/apache/hadoop/hbase/util/Bytes at
org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex.<clinit>(HFileBootstrapIndex.java:92)
at java.lang.Class.forName0(Native Method) at
java.lang.Class.forName(Class.java:264) at
org.apache.hudi.common.util.ReflectionUtils.getClass(ReflectionUtils.java:53)
at
org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:87)
at
org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:98)
at
org.apache.hudi.common.bootstrap.index.BootstrapIndex.getBootstrapIndex(BootstrapIndex.java:159)
at
org.apache.hudi.common.table.view.AbstractTableFileSystemView.init(AbstractTableFileSystemView.java:106)
at
org.apache.hudi.common.table.view.HoodieTableFileSystemView.init(HoodieTableFileSystemView.java:106)
at
org.apache.hudi.common.table.view.HoodieTableFileSystemView.<init>(HoodieTableFileSystemView.java:100)
at
org.apache.hudi.common.table.view.FileSystemViewManager.createInMemoryFileSystemView(FileSystemViewManager.java:158)
at
org.apache.hudi.common.table.view.FileSystemViewManager.lambda$createViewManager$ab86fd6b$5(FileSystemViewManager.java:213)
at
org.apache.hudi.common.table.view.FileSystemViewManager.lambda$getFileSystemView$1(FileSystemViewManager.java:104)
at
java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
at
org.apache.hudi.common.table.view.FileSystemViewManager.getFileSystemView(FileSystemViewManager.java:103)
at org.apache.hudi.table.HoodieTable.getBaseFileOnlyView(HoodieTable.java:242)
at
org.apache.hudi.table.action.commit.UpsertPartitioner.getSmallFiles(UpsertPartitioner.java:230)
at
org.apache.hudi.table.action.commit.UpsertPartitioner.lambda$getSmallFilesForPartitions$681ff4ab$1(UpsertPartitioner.java:213)
at
org.apache.hudi.client.common.function.FunctionWrapper.lambda$throwingMapToPairWrapper$3(FunctionWrapper.java:68)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at
java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) at
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) at
java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) at
java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at
java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) at
org.apache.hudi.client.common.HoodieFlinkEngineContext.mapToPair(HoodieFlinkEngineContext.java:79)
at
org.apache.hudi.table.action.commit.UpsertPartitioner.getSmallFilesForPartitions(UpsertPartitioner.java:213)
at
org.apache.hudi.table.action.commit.UpsertPartitioner.assignInserts(UpsertPartitioner.java:136)
at
org.apache.hudi.table.action.commit.UpsertPartitioner.<init>(UpsertPartitioner.java:97)
at
org.apache.hudi.table.action.commit.BaseFlinkCommitActionExecutor.getUpsertPartitioner(BaseFlinkCommitActionExecutor.java:321)
at
org.apache.hudi.table.action.commit.BaseFlinkCommitActionExecutor.getPartitioner(BaseFlinkCommitActionExecutor.java:144)
at
org.apache.hudi.table.action.commit.BaseFlinkCommitActionExecutor.execute(BaseFlinkCommitActionExecutor.java:114)
at
org.apache.hudi.table.action.commit.BaseFlinkCommitActionExecutor.execute(BaseFlinkCommitActionExecutor.java:68)
at
org.apache.hudi.table.action.commit.AbstractWriteHelper.write(AbstractWriteHelper.java:55)
When write data into hoodie table using operation api, shouldn't init a
bootstrap index.
The patch change Bootstrap api , it would return a null if execution config
does not set bootstrap path, and skip some process when index is null in
abstract fs view.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
