[jira] [Updated] (HUDI-4409) Improve LockManager wait logic when catch exception
[ https://issues.apache.org/jira/browse/HUDI-4409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang updated HUDI-4409: --- Summary: Improve LockManager wait logic when catch exception (was: LockManager improve wait time logic) > Improve LockManager wait logic when catch exception > --- > > Key: HUDI-4409 > URL: https://issues.apache.org/jira/browse/HUDI-4409 > Project: Apache Hudi > Issue Type: Improvement >Reporter: liujinhui >Priority: Major > Labels: pull-request-available > > {code:java} > //public void lock() { > if > (writeConfig.getWriteConcurrencyMode().supportsOptimisticConcurrencyControl()) > { > LockProvider lockProvider = getLockProvider(); > int retryCount = 0; > boolean acquired = false; > while (retryCount <= maxRetries) { > try { > acquired = > lockProvider.tryLock(writeConfig.getLockAcquireWaitTimeoutInMs(), > TimeUnit.MILLISECONDS); > if (acquired) { > break; > } > LOG.info("Retrying to acquire lock..."); > Thread.sleep(maxWaitTimeInMs); > } catch (HoodieLockException | InterruptedException e) { > if (retryCount >= maxRetries) { > throw new HoodieLockException("Unable to acquire lock, lock object > ", e); > } > } finally { > retryCount++; > } > } > if (!acquired) { > throw new HoodieLockException("Unable to acquire lock, lock object " + > lockProvider.getLock()); > } > } > } {code} > We should put sleep in catch -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HUDI-4409) Improve LockManager wait logic when catch exception
[ https://issues.apache.org/jira/browse/HUDI-4409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang reassigned HUDI-4409: -- Assignee: liujinhui > Improve LockManager wait logic when catch exception > --- > > Key: HUDI-4409 > URL: https://issues.apache.org/jira/browse/HUDI-4409 > Project: Apache Hudi > Issue Type: Improvement >Reporter: liujinhui >Assignee: liujinhui >Priority: Major > Labels: pull-request-available > > {code:java} > //public void lock() { > if > (writeConfig.getWriteConcurrencyMode().supportsOptimisticConcurrencyControl()) > { > LockProvider lockProvider = getLockProvider(); > int retryCount = 0; > boolean acquired = false; > while (retryCount <= maxRetries) { > try { > acquired = > lockProvider.tryLock(writeConfig.getLockAcquireWaitTimeoutInMs(), > TimeUnit.MILLISECONDS); > if (acquired) { > break; > } > LOG.info("Retrying to acquire lock..."); > Thread.sleep(maxWaitTimeInMs); > } catch (HoodieLockException | InterruptedException e) { > if (retryCount >= maxRetries) { > throw new HoodieLockException("Unable to acquire lock, lock object > ", e); > } > } finally { > retryCount++; > } > } > if (!acquired) { > throw new HoodieLockException("Unable to acquire lock, lock object " + > lockProvider.getLock()); > } > } > } {code} > We should put sleep in catch -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (HUDI-4409) Improve LockManager wait logic when catch exception
[ https://issues.apache.org/jira/browse/HUDI-4409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-4409. -- Resolution: Done > Improve LockManager wait logic when catch exception > --- > > Key: HUDI-4409 > URL: https://issues.apache.org/jira/browse/HUDI-4409 > Project: Apache Hudi > Issue Type: Improvement >Reporter: liujinhui >Assignee: liujinhui >Priority: Major > Labels: pull-request-available > Fix For: 0.12.0 > > > {code:java} > //public void lock() { > if > (writeConfig.getWriteConcurrencyMode().supportsOptimisticConcurrencyControl()) > { > LockProvider lockProvider = getLockProvider(); > int retryCount = 0; > boolean acquired = false; > while (retryCount <= maxRetries) { > try { > acquired = > lockProvider.tryLock(writeConfig.getLockAcquireWaitTimeoutInMs(), > TimeUnit.MILLISECONDS); > if (acquired) { > break; > } > LOG.info("Retrying to acquire lock..."); > Thread.sleep(maxWaitTimeInMs); > } catch (HoodieLockException | InterruptedException e) { > if (retryCount >= maxRetries) { > throw new HoodieLockException("Unable to acquire lock, lock object > ", e); > } > } finally { > retryCount++; > } > } > if (!acquired) { > throw new HoodieLockException("Unable to acquire lock, lock object " + > lockProvider.getLock()); > } > } > } {code} > We should put sleep in catch -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HUDI-4409) Improve LockManager wait logic when catch exception
[ https://issues.apache.org/jira/browse/HUDI-4409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang updated HUDI-4409: --- Fix Version/s: 0.12.0 > Improve LockManager wait logic when catch exception > --- > > Key: HUDI-4409 > URL: https://issues.apache.org/jira/browse/HUDI-4409 > Project: Apache Hudi > Issue Type: Improvement >Reporter: liujinhui >Assignee: liujinhui >Priority: Major > Labels: pull-request-available > Fix For: 0.12.0 > > > {code:java} > //public void lock() { > if > (writeConfig.getWriteConcurrencyMode().supportsOptimisticConcurrencyControl()) > { > LockProvider lockProvider = getLockProvider(); > int retryCount = 0; > boolean acquired = false; > while (retryCount <= maxRetries) { > try { > acquired = > lockProvider.tryLock(writeConfig.getLockAcquireWaitTimeoutInMs(), > TimeUnit.MILLISECONDS); > if (acquired) { > break; > } > LOG.info("Retrying to acquire lock..."); > Thread.sleep(maxWaitTimeInMs); > } catch (HoodieLockException | InterruptedException e) { > if (retryCount >= maxRetries) { > throw new HoodieLockException("Unable to acquire lock, lock object > ", e); > } > } finally { > retryCount++; > } > } > if (!acquired) { > throw new HoodieLockException("Unable to acquire lock, lock object " + > lockProvider.getLock()); > } > } > } {code} > We should put sleep in catch -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (HUDI-184) Integrate Hudi with Apache Flink
[ https://issues.apache.org/jira/browse/HUDI-184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-184. - Resolution: Implemented This feature has been tracked via https://issues.apache.org/jira/browse/HUDI-1521 > Integrate Hudi with Apache Flink > > > Key: HUDI-184 > URL: https://issues.apache.org/jira/browse/HUDI-184 > Project: Apache Hudi > Issue Type: New Feature > Components: writer-core >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > > Apache Flink is a popular streaming processing engine. > Integrating Hudi with Flink is a valuable work. > The discussion mailing thread is here: > [https://lists.apache.org/api/source.lua/1533de2d4cd4243fa9e8f8bf057ffd02f2ac0bec7c7539d8f72166ea@%3Cdev.hudi.apache.org%3E] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Reopened] (HUDI-184) Integrate Hudi with Apache Flink
[ https://issues.apache.org/jira/browse/HUDI-184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang reopened HUDI-184: --- > Integrate Hudi with Apache Flink > > > Key: HUDI-184 > URL: https://issues.apache.org/jira/browse/HUDI-184 > Project: Apache Hudi > Issue Type: New Feature > Components: writer-core >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > > Apache Flink is a popular streaming processing engine. > Integrating Hudi with Flink is a valuable work. > The discussion mailing thread is here: > [https://lists.apache.org/api/source.lua/1533de2d4cd4243fa9e8f8bf057ffd02f2ac0bec7c7539d8f72166ea@%3Cdev.hudi.apache.org%3E] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Closed] (HUDI-609) Implement a Flink specific HoodieIndex
[ https://issues.apache.org/jira/browse/HUDI-609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-609. - Resolution: Won't Do > Implement a Flink specific HoodieIndex > -- > > Key: HUDI-609 > URL: https://issues.apache.org/jira/browse/HUDI-609 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > > Indexing is a key step in hudi's write flow. {{HoodieIndex}} is the super > abstract class of all the implement of the index. Currently, {{HoodieIndex}} > couples with Spark in the design. However, HUDI-538 is doing the restructure > for hudi-client so that hudi can be decoupled with Spark. After that, we > would get an engine-irrelevant implementation of {{HoodieIndex}}. And > extending that class, we could implement a Flink specific index. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Closed] (HUDI-184) Integrate Hudi with Apache Flink
[ https://issues.apache.org/jira/browse/HUDI-184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-184. - Resolution: Won't Do > Integrate Hudi with Apache Flink > > > Key: HUDI-184 > URL: https://issues.apache.org/jira/browse/HUDI-184 > Project: Apache Hudi > Issue Type: New Feature > Components: writer-core >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > > Apache Flink is a popular streaming processing engine. > Integrating Hudi with Flink is a valuable work. > The discussion mailing thread is here: > [https://lists.apache.org/api/source.lua/1533de2d4cd4243fa9e8f8bf057ffd02f2ac0bec7c7539d8f72166ea@%3Cdev.hudi.apache.org%3E] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Closed] (HUDI-608) Implement a flink datastream execution context
[ https://issues.apache.org/jira/browse/HUDI-608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-608. - Resolution: Won't Do > Implement a flink datastream execution context > -- > > Key: HUDI-608 > URL: https://issues.apache.org/jira/browse/HUDI-608 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > > Currently {{HoodieWriteClient}} does something like > `hoodieRecordRDD.map().sort()` internally.. if we want to support Flink > DataStream as the object, then we need to somehow define an abstraction like > {{HoodieExecutionContext}} which will have a common set of map(T) -> T, > filter(), repartition() methods. There will be subclass like > {{HoodieFlinkDataStreamExecutionContext}} which will implement it > in Flink specific ways and hand back the transformed T object. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-2418) Support HiveSchemaProvider
[ https://issues.apache.org/jira/browse/HUDI-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang updated HUDI-2418: --- Summary: Support HiveSchemaProvider (was: add HiveSchemaProvider ) > Support HiveSchemaProvider > --- > > Key: HUDI-2418 > URL: https://issues.apache.org/jira/browse/HUDI-2418 > Project: Apache Hudi > Issue Type: Improvement > Components: DeltaStreamer >Reporter: Jian Feng >Assignee: Jian Feng >Priority: Major > Labels: pull-request-available > Fix For: 0.11.0 > > > when using DeltaStreamer to migrate exist Hive table, it better to have a > HiveSchemaProvider instead of avro schema file. > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HUDI-2699) Remove duplicated zookeeper with tests classifier exists in bundles
vinoyang created HUDI-2699: -- Summary: Remove duplicated zookeeper with tests classifier exists in bundles Key: HUDI-2699 URL: https://issues.apache.org/jira/browse/HUDI-2699 Project: Apache Hudi Issue Type: Sub-task Reporter: vinoyang Assignee: vinoyang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (HUDI-2643) Remove duplicated hbase-common with tests classifier exists in bundles
[ https://issues.apache.org/jira/browse/HUDI-2643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-2643. -- Resolution: Done 13b637ddc3ab9fba51e303cfa0343a496e476d26 > Remove duplicated hbase-common with tests classifier exists in bundles > -- > > Key: HUDI-2643 > URL: https://issues.apache.org/jira/browse/HUDI-2643 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > Labels: pull-request-available > Fix For: 0.10.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-2643) Remove duplicated hbase-common with tests classifier exists in bundles
[ https://issues.apache.org/jira/browse/HUDI-2643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang updated HUDI-2643: --- Fix Version/s: 0.10.0 > Remove duplicated hbase-common with tests classifier exists in bundles > -- > > Key: HUDI-2643 > URL: https://issues.apache.org/jira/browse/HUDI-2643 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > Labels: pull-request-available > Fix For: 0.10.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-2643) Remove duplicated hbase-common with tests classifier exists in bundles
vinoyang created HUDI-2643: -- Summary: Remove duplicated hbase-common with tests classifier exists in bundles Key: HUDI-2643 URL: https://issues.apache.org/jira/browse/HUDI-2643 Project: Apache Hudi Issue Type: Sub-task Reporter: vinoyang Assignee: vinoyang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (HUDI-2614) Remove duplicated hadoop-hdfs with tests classifier exists in bundles
[ https://issues.apache.org/jira/browse/HUDI-2614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-2614. -- Resolution: Done b1c4acf0aeb0f3d650c8e704828b1c2b0d2b5b40 > Remove duplicated hadoop-hdfs with tests classifier exists in bundles > - > > Key: HUDI-2614 > URL: https://issues.apache.org/jira/browse/HUDI-2614 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > Labels: pull-request-available > Fix For: 0.10.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-2614) Remove duplicated hadoop-hdfs with tests classifier exists in bundles
[ https://issues.apache.org/jira/browse/HUDI-2614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang updated HUDI-2614: --- Fix Version/s: 0.10.0 > Remove duplicated hadoop-hdfs with tests classifier exists in bundles > - > > Key: HUDI-2614 > URL: https://issues.apache.org/jira/browse/HUDI-2614 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > Labels: pull-request-available > Fix For: 0.10.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-2614) Remove duplicated hadoop-hdfs with tests classifier exists in bundles
vinoyang created HUDI-2614: -- Summary: Remove duplicated hadoop-hdfs with tests classifier exists in bundles Key: HUDI-2614 URL: https://issues.apache.org/jira/browse/HUDI-2614 Project: Apache Hudi Issue Type: Sub-task Reporter: vinoyang Assignee: vinoyang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-2600) Remove duplicated hadoop-common with tests classifier exists in bundles
[ https://issues.apache.org/jira/browse/HUDI-2600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang updated HUDI-2600: --- Fix Version/s: 0.10.0 > Remove duplicated hadoop-common with tests classifier exists in bundles > --- > > Key: HUDI-2600 > URL: https://issues.apache.org/jira/browse/HUDI-2600 > Project: Apache Hudi > Issue Type: Sub-task > Components: Release & Administrative >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > Labels: pull-request-available > Fix For: 0.10.0 > > > We found many duplicated dependencies in the generated dependency list, > `hadoop-common` is one of them: > {code:java} > hadoop-common/org.apache.hadoop/2.7.3//hadoop-common-2.7.3.jar > hadoop-common/org.apache.hadoop/2.7.3/tests/hadoop-common-2.7.3-tests.jar > {code} > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (HUDI-2600) Remove duplicated hadoop-common with tests classifier exists in bundles
[ https://issues.apache.org/jira/browse/HUDI-2600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-2600. -- Resolution: Done 220bf6a7e6f5cdf0efbbbee9df6852a8b2288570 > Remove duplicated hadoop-common with tests classifier exists in bundles > --- > > Key: HUDI-2600 > URL: https://issues.apache.org/jira/browse/HUDI-2600 > Project: Apache Hudi > Issue Type: Sub-task > Components: Release & Administrative >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > Labels: pull-request-available > Fix For: 0.10.0 > > > We found many duplicated dependencies in the generated dependency list, > `hadoop-common` is one of them: > {code:java} > hadoop-common/org.apache.hadoop/2.7.3//hadoop-common-2.7.3.jar > hadoop-common/org.apache.hadoop/2.7.3/tests/hadoop-common-2.7.3-tests.jar > {code} > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (HUDI-2592) NumberFormatException: Zero length BigInteger when write.precombine.field is decimal type
[ https://issues.apache.org/jira/browse/HUDI-2592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-2592. -- Resolution: Fixed > NumberFormatException: Zero length BigInteger when write.precombine.field is > decimal type > - > > Key: HUDI-2592 > URL: https://issues.apache.org/jira/browse/HUDI-2592 > Project: Apache Hudi > Issue Type: Bug > Components: Common Core >Reporter: Matrix42 >Assignee: Matrix42 >Priority: Major > Labels: pull-request-available > Fix For: 0.10.0, 0.11.0 > > > when write.precombine.field is decimal type,write decimal will be an empty > byte array, when read will throw NumberFormatException: Zero length > BigInteger like below: > {code:java} > 2021-10-20 17:14:03 > java.lang.NumberFormatException: Zero length BigInteger > at java.math.BigInteger.(BigInteger.java:302) > at > org.apache.flink.table.data.DecimalData.fromUnscaledBytes(DecimalData.java:223) > at > org.apache.flink.connectors.hudi.util.AvroToRowDataConverters.lambda$createDecimalConverter$4dc14f00$1(AvroToRowDataConverters.java:158) > at > org.apache.flink.connectors.hudi.util.AvroToRowDataConverters.lambda$createNullableConverter$4568343a$1(AvroToRowDataConverters.java:94) > at > org.apache.flink.connectors.hudi.util.AvroToRowDataConverters.lambda$createRowConverter$68595fbd$1(AvroToRowDataConverters.java:75) > at > org.apache.flink.connectors.hudi.table.format.mor.MergeOnReadInputFormat$1.hasNext(MergeOnReadInputFormat.java:300) > at > org.apache.flink.connectors.hudi.table.format.mor.MergeOnReadInputFormat$LogFileOnlyIterator.reachedEnd(MergeOnReadInputFormat.java:362) > at > org.apache.flink.connectors.hudi.table.format.mor.MergeOnReadInputFormat.reachedEnd(MergeOnReadInputFormat.java:202) > at > org.apache.flink.streaming.api.functions.source.InputFormatSourceFunction.run(InputFormatSourceFunction.java:90) > at > org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:100) > at > org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:63) > at > org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:213) > {code} > analyze: > > HoodieAvroUtils.getNestedFieldVal will invoked to extract precombine field. > next will invoke convertValueForAvroLogicalTypes. when field is decimal > type,the bytebuffer will consumed, we should rewind. > {code:java} > private static Object convertValueForAvroLogicalTypes(Schema fieldSchema, > Object fieldValue) { > if (fieldSchema.getLogicalType() == LogicalTypes.date()) { > return LocalDate.ofEpochDay(Long.parseLong(fieldValue.toString())); > } else if (fieldSchema.getLogicalType() instanceof LogicalTypes.Decimal) { > Decimal dc = (Decimal) fieldSchema.getLogicalType(); > DecimalConversion decimalConversion = new DecimalConversion(); > if (fieldSchema.getType() == Schema.Type.FIXED) { > return decimalConversion.fromFixed((GenericFixed) fieldValue, > fieldSchema, > LogicalTypes.decimal(dc.getPrecision(), dc.getScale())); > } else if (fieldSchema.getType() == Schema.Type.BYTES) { > > //this methoad will consume the byteBuffer > return decimalConversion.fromBytes((ByteBuffer) fieldValue, fieldSchema, > LogicalTypes.decimal(dc.getPrecision(), dc.getScale())); > } > } > return fieldValue; > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (HUDI-2592) NumberFormatException: Zero length BigInteger when write.precombine.field is decimal type
[ https://issues.apache.org/jira/browse/HUDI-2592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang reopened HUDI-2592: > NumberFormatException: Zero length BigInteger when write.precombine.field is > decimal type > - > > Key: HUDI-2592 > URL: https://issues.apache.org/jira/browse/HUDI-2592 > Project: Apache Hudi > Issue Type: Bug > Components: Common Core >Reporter: Matrix42 >Assignee: Matrix42 >Priority: Major > Labels: pull-request-available > Fix For: 0.10.0, 0.11.0 > > > when write.precombine.field is decimal type,write decimal will be an empty > byte array, when read will throw NumberFormatException: Zero length > BigInteger like below: > {code:java} > 2021-10-20 17:14:03 > java.lang.NumberFormatException: Zero length BigInteger > at java.math.BigInteger.(BigInteger.java:302) > at > org.apache.flink.table.data.DecimalData.fromUnscaledBytes(DecimalData.java:223) > at > org.apache.flink.connectors.hudi.util.AvroToRowDataConverters.lambda$createDecimalConverter$4dc14f00$1(AvroToRowDataConverters.java:158) > at > org.apache.flink.connectors.hudi.util.AvroToRowDataConverters.lambda$createNullableConverter$4568343a$1(AvroToRowDataConverters.java:94) > at > org.apache.flink.connectors.hudi.util.AvroToRowDataConverters.lambda$createRowConverter$68595fbd$1(AvroToRowDataConverters.java:75) > at > org.apache.flink.connectors.hudi.table.format.mor.MergeOnReadInputFormat$1.hasNext(MergeOnReadInputFormat.java:300) > at > org.apache.flink.connectors.hudi.table.format.mor.MergeOnReadInputFormat$LogFileOnlyIterator.reachedEnd(MergeOnReadInputFormat.java:362) > at > org.apache.flink.connectors.hudi.table.format.mor.MergeOnReadInputFormat.reachedEnd(MergeOnReadInputFormat.java:202) > at > org.apache.flink.streaming.api.functions.source.InputFormatSourceFunction.run(InputFormatSourceFunction.java:90) > at > org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:100) > at > org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:63) > at > org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:213) > {code} > analyze: > > HoodieAvroUtils.getNestedFieldVal will invoked to extract precombine field. > next will invoke convertValueForAvroLogicalTypes. when field is decimal > type,the bytebuffer will consumed, we should rewind. > {code:java} > private static Object convertValueForAvroLogicalTypes(Schema fieldSchema, > Object fieldValue) { > if (fieldSchema.getLogicalType() == LogicalTypes.date()) { > return LocalDate.ofEpochDay(Long.parseLong(fieldValue.toString())); > } else if (fieldSchema.getLogicalType() instanceof LogicalTypes.Decimal) { > Decimal dc = (Decimal) fieldSchema.getLogicalType(); > DecimalConversion decimalConversion = new DecimalConversion(); > if (fieldSchema.getType() == Schema.Type.FIXED) { > return decimalConversion.fromFixed((GenericFixed) fieldValue, > fieldSchema, > LogicalTypes.decimal(dc.getPrecision(), dc.getScale())); > } else if (fieldSchema.getType() == Schema.Type.BYTES) { > > //this methoad will consume the byteBuffer > return decimalConversion.fromBytes((ByteBuffer) fieldValue, fieldSchema, > LogicalTypes.decimal(dc.getPrecision(), dc.getScale())); > } > } > return fieldValue; > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-2592) NumberFormatException: Zero length BigInteger when write.precombine.field is decimal type
[ https://issues.apache.org/jira/browse/HUDI-2592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17432925#comment-17432925 ] vinoyang commented on HUDI-2592: [~Matrix42] I have given you Jira contributor permission. Thanks for your contribution! > NumberFormatException: Zero length BigInteger when write.precombine.field is > decimal type > - > > Key: HUDI-2592 > URL: https://issues.apache.org/jira/browse/HUDI-2592 > Project: Apache Hudi > Issue Type: Bug > Components: Common Core >Reporter: Matrix42 >Assignee: Matrix42 >Priority: Major > Labels: pull-request-available > Fix For: 0.10.0, 0.11.0 > > > when write.precombine.field is decimal type,write decimal will be an empty > byte array, when read will throw NumberFormatException: Zero length > BigInteger like below: > {code:java} > 2021-10-20 17:14:03 > java.lang.NumberFormatException: Zero length BigInteger > at java.math.BigInteger.(BigInteger.java:302) > at > org.apache.flink.table.data.DecimalData.fromUnscaledBytes(DecimalData.java:223) > at > org.apache.flink.connectors.hudi.util.AvroToRowDataConverters.lambda$createDecimalConverter$4dc14f00$1(AvroToRowDataConverters.java:158) > at > org.apache.flink.connectors.hudi.util.AvroToRowDataConverters.lambda$createNullableConverter$4568343a$1(AvroToRowDataConverters.java:94) > at > org.apache.flink.connectors.hudi.util.AvroToRowDataConverters.lambda$createRowConverter$68595fbd$1(AvroToRowDataConverters.java:75) > at > org.apache.flink.connectors.hudi.table.format.mor.MergeOnReadInputFormat$1.hasNext(MergeOnReadInputFormat.java:300) > at > org.apache.flink.connectors.hudi.table.format.mor.MergeOnReadInputFormat$LogFileOnlyIterator.reachedEnd(MergeOnReadInputFormat.java:362) > at > org.apache.flink.connectors.hudi.table.format.mor.MergeOnReadInputFormat.reachedEnd(MergeOnReadInputFormat.java:202) > at > org.apache.flink.streaming.api.functions.source.InputFormatSourceFunction.run(InputFormatSourceFunction.java:90) > at > org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:100) > at > org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:63) > at > org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:213) > {code} > analyze: > > HoodieAvroUtils.getNestedFieldVal will invoked to extract precombine field. > next will invoke convertValueForAvroLogicalTypes. when field is decimal > type,the bytebuffer will consumed, we should rewind. > {code:java} > private static Object convertValueForAvroLogicalTypes(Schema fieldSchema, > Object fieldValue) { > if (fieldSchema.getLogicalType() == LogicalTypes.date()) { > return LocalDate.ofEpochDay(Long.parseLong(fieldValue.toString())); > } else if (fieldSchema.getLogicalType() instanceof LogicalTypes.Decimal) { > Decimal dc = (Decimal) fieldSchema.getLogicalType(); > DecimalConversion decimalConversion = new DecimalConversion(); > if (fieldSchema.getType() == Schema.Type.FIXED) { > return decimalConversion.fromFixed((GenericFixed) fieldValue, > fieldSchema, > LogicalTypes.decimal(dc.getPrecision(), dc.getScale())); > } else if (fieldSchema.getType() == Schema.Type.BYTES) { > > //this methoad will consume the byteBuffer > return decimalConversion.fromBytes((ByteBuffer) fieldValue, fieldSchema, > LogicalTypes.decimal(dc.getPrecision(), dc.getScale())); > } > } > return fieldValue; > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HUDI-2592) NumberFormatException: Zero length BigInteger when write.precombine.field is decimal type
[ https://issues.apache.org/jira/browse/HUDI-2592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang reassigned HUDI-2592: -- Assignee: Matrix42 > NumberFormatException: Zero length BigInteger when write.precombine.field is > decimal type > - > > Key: HUDI-2592 > URL: https://issues.apache.org/jira/browse/HUDI-2592 > Project: Apache Hudi > Issue Type: Bug > Components: Common Core >Reporter: Matrix42 >Assignee: Matrix42 >Priority: Major > Labels: pull-request-available > Fix For: 0.10.0, 0.11.0 > > > when write.precombine.field is decimal type,write decimal will be an empty > byte array, when read will throw NumberFormatException: Zero length > BigInteger like below: > {code:java} > 2021-10-20 17:14:03 > java.lang.NumberFormatException: Zero length BigInteger > at java.math.BigInteger.(BigInteger.java:302) > at > org.apache.flink.table.data.DecimalData.fromUnscaledBytes(DecimalData.java:223) > at > org.apache.flink.connectors.hudi.util.AvroToRowDataConverters.lambda$createDecimalConverter$4dc14f00$1(AvroToRowDataConverters.java:158) > at > org.apache.flink.connectors.hudi.util.AvroToRowDataConverters.lambda$createNullableConverter$4568343a$1(AvroToRowDataConverters.java:94) > at > org.apache.flink.connectors.hudi.util.AvroToRowDataConverters.lambda$createRowConverter$68595fbd$1(AvroToRowDataConverters.java:75) > at > org.apache.flink.connectors.hudi.table.format.mor.MergeOnReadInputFormat$1.hasNext(MergeOnReadInputFormat.java:300) > at > org.apache.flink.connectors.hudi.table.format.mor.MergeOnReadInputFormat$LogFileOnlyIterator.reachedEnd(MergeOnReadInputFormat.java:362) > at > org.apache.flink.connectors.hudi.table.format.mor.MergeOnReadInputFormat.reachedEnd(MergeOnReadInputFormat.java:202) > at > org.apache.flink.streaming.api.functions.source.InputFormatSourceFunction.run(InputFormatSourceFunction.java:90) > at > org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:100) > at > org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:63) > at > org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:213) > {code} > analyze: > > HoodieAvroUtils.getNestedFieldVal will invoked to extract precombine field. > next will invoke convertValueForAvroLogicalTypes. when field is decimal > type,the bytebuffer will consumed, we should rewind. > {code:java} > private static Object convertValueForAvroLogicalTypes(Schema fieldSchema, > Object fieldValue) { > if (fieldSchema.getLogicalType() == LogicalTypes.date()) { > return LocalDate.ofEpochDay(Long.parseLong(fieldValue.toString())); > } else if (fieldSchema.getLogicalType() instanceof LogicalTypes.Decimal) { > Decimal dc = (Decimal) fieldSchema.getLogicalType(); > DecimalConversion decimalConversion = new DecimalConversion(); > if (fieldSchema.getType() == Schema.Type.FIXED) { > return decimalConversion.fromFixed((GenericFixed) fieldValue, > fieldSchema, > LogicalTypes.decimal(dc.getPrecision(), dc.getScale())); > } else if (fieldSchema.getType() == Schema.Type.BYTES) { > > //this methoad will consume the byteBuffer > return decimalConversion.fromBytes((ByteBuffer) fieldValue, fieldSchema, > LogicalTypes.decimal(dc.getPrecision(), dc.getScale())); > } > } > return fieldValue; > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-2592) NumberFormatException: Zero length BigInteger when write.precombine.field is decimal type
[ https://issues.apache.org/jira/browse/HUDI-2592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang updated HUDI-2592: --- Status: Closed (was: Patch Available) > NumberFormatException: Zero length BigInteger when write.precombine.field is > decimal type > - > > Key: HUDI-2592 > URL: https://issues.apache.org/jira/browse/HUDI-2592 > Project: Apache Hudi > Issue Type: Bug > Components: Common Core >Reporter: Matrix42 >Priority: Major > Labels: pull-request-available > Fix For: 0.10.0, 0.11.0 > > > when write.precombine.field is decimal type,write decimal will be an empty > byte array, when read will throw NumberFormatException: Zero length > BigInteger like below: > {code:java} > 2021-10-20 17:14:03 > java.lang.NumberFormatException: Zero length BigInteger > at java.math.BigInteger.(BigInteger.java:302) > at > org.apache.flink.table.data.DecimalData.fromUnscaledBytes(DecimalData.java:223) > at > org.apache.flink.connectors.hudi.util.AvroToRowDataConverters.lambda$createDecimalConverter$4dc14f00$1(AvroToRowDataConverters.java:158) > at > org.apache.flink.connectors.hudi.util.AvroToRowDataConverters.lambda$createNullableConverter$4568343a$1(AvroToRowDataConverters.java:94) > at > org.apache.flink.connectors.hudi.util.AvroToRowDataConverters.lambda$createRowConverter$68595fbd$1(AvroToRowDataConverters.java:75) > at > org.apache.flink.connectors.hudi.table.format.mor.MergeOnReadInputFormat$1.hasNext(MergeOnReadInputFormat.java:300) > at > org.apache.flink.connectors.hudi.table.format.mor.MergeOnReadInputFormat$LogFileOnlyIterator.reachedEnd(MergeOnReadInputFormat.java:362) > at > org.apache.flink.connectors.hudi.table.format.mor.MergeOnReadInputFormat.reachedEnd(MergeOnReadInputFormat.java:202) > at > org.apache.flink.streaming.api.functions.source.InputFormatSourceFunction.run(InputFormatSourceFunction.java:90) > at > org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:100) > at > org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:63) > at > org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:213) > {code} > analyze: > > HoodieAvroUtils.getNestedFieldVal will invoked to extract precombine field. > next will invoke convertValueForAvroLogicalTypes. when field is decimal > type,the bytebuffer will consumed, we should rewind. > {code:java} > private static Object convertValueForAvroLogicalTypes(Schema fieldSchema, > Object fieldValue) { > if (fieldSchema.getLogicalType() == LogicalTypes.date()) { > return LocalDate.ofEpochDay(Long.parseLong(fieldValue.toString())); > } else if (fieldSchema.getLogicalType() instanceof LogicalTypes.Decimal) { > Decimal dc = (Decimal) fieldSchema.getLogicalType(); > DecimalConversion decimalConversion = new DecimalConversion(); > if (fieldSchema.getType() == Schema.Type.FIXED) { > return decimalConversion.fromFixed((GenericFixed) fieldValue, > fieldSchema, > LogicalTypes.decimal(dc.getPrecision(), dc.getScale())); > } else if (fieldSchema.getType() == Schema.Type.BYTES) { > > //this methoad will consume the byteBuffer > return decimalConversion.fromBytes((ByteBuffer) fieldValue, fieldSchema, > LogicalTypes.decimal(dc.getPrecision(), dc.getScale())); > } > } > return fieldValue; > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-2600) Remove duplicated hadoop-common with tests classifier exists in bundles
vinoyang created HUDI-2600: -- Summary: Remove duplicated hadoop-common with tests classifier exists in bundles Key: HUDI-2600 URL: https://issues.apache.org/jira/browse/HUDI-2600 Project: Apache Hudi Issue Type: Sub-task Components: Release & Administrative Reporter: vinoyang We found many duplicated dependencies in the generated dependency list, `hadoop-common` is one of them: {code:java} hadoop-common/org.apache.hadoop/2.7.3//hadoop-common-2.7.3.jar hadoop-common/org.apache.hadoop/2.7.3/tests/hadoop-common-2.7.3-tests.jar {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HUDI-2600) Remove duplicated hadoop-common with tests classifier exists in bundles
[ https://issues.apache.org/jira/browse/HUDI-2600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang reassigned HUDI-2600: -- Assignee: vinoyang > Remove duplicated hadoop-common with tests classifier exists in bundles > --- > > Key: HUDI-2600 > URL: https://issues.apache.org/jira/browse/HUDI-2600 > Project: Apache Hudi > Issue Type: Sub-task > Components: Release & Administrative >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > > We found many duplicated dependencies in the generated dependency list, > `hadoop-common` is one of them: > {code:java} > hadoop-common/org.apache.hadoop/2.7.3//hadoop-common-2.7.3.jar > hadoop-common/org.apache.hadoop/2.7.3/tests/hadoop-common-2.7.3-tests.jar > {code} > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (HUDI-2507) Generate more dependency list file for other bundles
[ https://issues.apache.org/jira/browse/HUDI-2507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-2507. -- Resolution: Done b480294e792b6344d37560587f8f6e170e210d14 > Generate more dependency list file for other bundles > > > Key: HUDI-2507 > URL: https://issues.apache.org/jira/browse/HUDI-2507 > Project: Apache Hudi > Issue Type: Sub-task > Components: Usability >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > Labels: pull-request-available > Fix For: 0.10.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-2507) Generate more dependency list file for other bundles
[ https://issues.apache.org/jira/browse/HUDI-2507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang updated HUDI-2507: --- Fix Version/s: 0.10.0 > Generate more dependency list file for other bundles > > > Key: HUDI-2507 > URL: https://issues.apache.org/jira/browse/HUDI-2507 > Project: Apache Hudi > Issue Type: Sub-task > Components: Usability >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > Labels: pull-request-available > Fix For: 0.10.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-2508) Build GA for the dependeny diff check workflow
vinoyang created HUDI-2508: -- Summary: Build GA for the dependeny diff check workflow Key: HUDI-2508 URL: https://issues.apache.org/jira/browse/HUDI-2508 Project: Apache Hudi Issue Type: Sub-task Components: Usability Reporter: vinoyang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HUDI-2508) Build GA for the dependeny diff check workflow
[ https://issues.apache.org/jira/browse/HUDI-2508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang reassigned HUDI-2508: -- Assignee: vinoyang > Build GA for the dependeny diff check workflow > -- > > Key: HUDI-2508 > URL: https://issues.apache.org/jira/browse/HUDI-2508 > Project: Apache Hudi > Issue Type: Sub-task > Components: Usability >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HUDI-2507) Generate more dependency list file for other bundles
[ https://issues.apache.org/jira/browse/HUDI-2507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang reassigned HUDI-2507: -- Assignee: vinoyang > Generate more dependency list file for other bundles > > > Key: HUDI-2507 > URL: https://issues.apache.org/jira/browse/HUDI-2507 > Project: Apache Hudi > Issue Type: Sub-task > Components: Usability >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-2507) Generate more dependency list file for other bundles
vinoyang created HUDI-2507: -- Summary: Generate more dependency list file for other bundles Key: HUDI-2507 URL: https://issues.apache.org/jira/browse/HUDI-2507 Project: Apache Hudi Issue Type: Sub-task Components: Usability Reporter: vinoyang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HUDI-2506) Hudi dependency governance
[ https://issues.apache.org/jira/browse/HUDI-2506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang reassigned HUDI-2506: -- Assignee: vinoyang > Hudi dependency governance > -- > > Key: HUDI-2506 > URL: https://issues.apache.org/jira/browse/HUDI-2506 > Project: Apache Hudi > Issue Type: Task > Components: Usability >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (HUDI-2440) Add dependency change diff script for dependency governace
[ https://issues.apache.org/jira/browse/HUDI-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-2440. -- Resolution: Done > Add dependency change diff script for dependency governace > -- > > Key: HUDI-2440 > URL: https://issues.apache.org/jira/browse/HUDI-2440 > Project: Apache Hudi > Issue Type: Sub-task > Components: Usability, Utilities >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > Labels: pull-request-available > Fix For: 0.10.0 > > > Currently, hudi's dependency management is chaotic, e.g. for > `hudi-spark-bundle_2.11`, the dependency list is here: > {code:java} > HikariCP/2.5.1//HikariCP-2.5.1.jar > ST4/4.0.4//ST4-4.0.4.jar > aircompressor/0.15//aircompressor-0.15.jar > annotations/17.0.0//annotations-17.0.0.jar > ant-launcher/1.9.1//ant-launcher-1.9.1.jar > ant/1.6.5//ant-1.6.5.jar > ant/1.9.1//ant-1.9.1.jar > antlr-runtime/3.5.2//antlr-runtime-3.5.2.jar > aopalliance/1.0//aopalliance-1.0.jar > apache-curator/2.7.1//apache-curator-2.7.1.pom > apacheds-i18n/2.0.0-M15//apacheds-i18n-2.0.0-M15.jar > apacheds-kerberos-codec/2.0.0-M15//apacheds-kerberos-codec-2.0.0-M15.jar > api-asn1-api/1.0.0-M20//api-asn1-api-1.0.0-M20.jar > api-util/1.0.0-M20//api-util-1.0.0-M20.jar > asm/3.1//asm-3.1.jar > avatica-metrics/1.8.0//avatica-metrics-1.8.0.jar > avatica/1.8.0//avatica-1.8.0.jar > avro/1.8.2//avro-1.8.2.jar > bonecp/0.8.0.RELEASE//bonecp-0.8.0.RELEASE.jar > calcite-core/1.10.0//calcite-core-1.10.0.jar > calcite-druid/1.10.0//calcite-druid-1.10.0.jar > calcite-linq4j/1.10.0//calcite-linq4j-1.10.0.jar > commons-beanutils-core/1.8.0//commons-beanutils-core-1.8.0.jar > commons-beanutils/1.7.0//commons-beanutils-1.7.0.jar > commons-cli/1.2//commons-cli-1.2.jar > commons-codec/1.4//commons-codec-1.4.jar > commons-collections/3.2.2//commons-collections-3.2.2.jar > commons-compiler/2.7.6//commons-compiler-2.7.6.jar > commons-compress/1.9//commons-compress-1.9.jar > commons-configuration/1.6//commons-configuration-1.6.jar > commons-daemon/1.0.13//commons-daemon-1.0.13.jar > commons-dbcp/1.4//commons-dbcp-1.4.jar > commons-digester/1.8//commons-digester-1.8.jar > commons-el/1.0//commons-el-1.0.jar > commons-httpclient/3.1//commons-httpclient-3.1.jar > commons-io/2.4//commons-io-2.4.jar > commons-lang/2.6//commons-lang-2.6.jar > commons-lang3/3.1//commons-lang3-3.1.jar > commons-logging/1.2//commons-logging-1.2.jar > commons-math/2.2//commons-math-2.2.jar > commons-math3/3.1.1//commons-math3-3.1.1.jar > commons-net/3.1//commons-net-3.1.jar > commons-pool/1.5.4//commons-pool-1.5.4.jar > curator-client/2.7.1//curator-client-2.7.1.jar > curator-framework/2.7.1//curator-framework-2.7.1.jar > curator-recipes/2.7.1//curator-recipes-2.7.1.jar > datanucleus-api-jdo/4.2.4//datanucleus-api-jdo-4.2.4.jar > datanucleus-core/4.1.17//datanucleus-core-4.1.17.jar > datanucleus-rdbms/4.1.19//datanucleus-rdbms-4.1.19.jar > derby/10.10.2.0//derby-10.10.2.0.jar > disruptor/3.3.0//disruptor-3.3.0.jar > dropwizard-metrics-hadoop-metrics2-reporter/0.1.2//dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar > eigenbase-properties/1.1.5//eigenbase-properties-1.1.5.jar > fastutil/7.0.13//fastutil-7.0.13.jar > findbugs-annotations/1.3.9-1//findbugs-annotations-1.3.9-1.jar > fluent-hc/4.4.1//fluent-hc-4.4.1.jar > groovy-all/2.4.4//groovy-all-2.4.4.jar > gson/2.3.1//gson-2.3.1.jar > guava/14.0.1//guava-14.0.1.jar > guice-assistedinject/3.0//guice-assistedinject-3.0.jar > guice-servlet/3.0//guice-servlet-3.0.jar > guice/3.0//guice-3.0.jar > hadoop-annotations/2.7.3//hadoop-annotations-2.7.3.jar > hadoop-auth/2.7.3//hadoop-auth-2.7.3.jar > hadoop-client/2.7.3//hadoop-client-2.7.3.jar > hadoop-common/2.7.3//hadoop-common-2.7.3.jar > hadoop-common/2.7.3/tests/hadoop-common-2.7.3-tests.jar > hadoop-hdfs/2.7.3//hadoop-hdfs-2.7.3.jar > hadoop-hdfs/2.7.3/tests/hadoop-hdfs-2.7.3-tests.jar > hadoop-mapreduce-client-app/2.7.3//hadoop-mapreduce-client-app-2.7.3.jar > hadoop-mapreduce-client-common/2.7.3//hadoop-mapreduce-client-common-2.7.3.jar > hadoop-mapreduce-client-core/2.7.3//hadoop-mapreduce-client-core-2.7.3.jar > hadoop-mapreduce-client-jobclient/2.7.3//hadoop-mapreduce-client-jobclient-2.7.3.jar > hadoop-mapreduce-client-shuffle/2.7.3//hadoop-mapreduce-client-shuffle-2.7.3.jar > hadoop-yarn-api/2.7.3//hadoop-yarn-api-2.7.3.jar > hadoop-yarn-client/2.7.3//hadoop-yarn-client-2.7.3.jar > hadoop-yarn-common/2.7.3//hadoop-yarn-common-2.7.3.jar > hadoop-yarn-registry/2.7.1//hadoop-yarn-registry-2.7.1.jar > hadoop-yarn-server-applicationhistoryservice/2.7.2//hadoop-yarn-server-applicationhistoryservice-2.7.2.jar > hadoop-yarn-server-common/2.7.2//hadoop-yarn-server-common-2.7.2.jar > hadoop-yarn-server-resourcemanager/2.7.2//hadoop-yarn-server-resourcemanager-2.7.2.jar > hadoop-yarn
[jira] [Created] (HUDI-2506) Hudi dependency governance
vinoyang created HUDI-2506: -- Summary: Hudi dependency governance Key: HUDI-2506 URL: https://issues.apache.org/jira/browse/HUDI-2506 Project: Apache Hudi Issue Type: Task Components: Usability Reporter: vinoyang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (HUDI-2440) Add dependency change diff script for dependency governace
[ https://issues.apache.org/jira/browse/HUDI-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang reopened HUDI-2440: > Add dependency change diff script for dependency governace > -- > > Key: HUDI-2440 > URL: https://issues.apache.org/jira/browse/HUDI-2440 > Project: Apache Hudi > Issue Type: Improvement > Components: Usability, Utilities >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > Labels: pull-request-available > Fix For: 0.10.0 > > > Currently, hudi's dependency management is chaotic, e.g. for > `hudi-spark-bundle_2.11`, the dependency list is here: > {code:java} > HikariCP/2.5.1//HikariCP-2.5.1.jar > ST4/4.0.4//ST4-4.0.4.jar > aircompressor/0.15//aircompressor-0.15.jar > annotations/17.0.0//annotations-17.0.0.jar > ant-launcher/1.9.1//ant-launcher-1.9.1.jar > ant/1.6.5//ant-1.6.5.jar > ant/1.9.1//ant-1.9.1.jar > antlr-runtime/3.5.2//antlr-runtime-3.5.2.jar > aopalliance/1.0//aopalliance-1.0.jar > apache-curator/2.7.1//apache-curator-2.7.1.pom > apacheds-i18n/2.0.0-M15//apacheds-i18n-2.0.0-M15.jar > apacheds-kerberos-codec/2.0.0-M15//apacheds-kerberos-codec-2.0.0-M15.jar > api-asn1-api/1.0.0-M20//api-asn1-api-1.0.0-M20.jar > api-util/1.0.0-M20//api-util-1.0.0-M20.jar > asm/3.1//asm-3.1.jar > avatica-metrics/1.8.0//avatica-metrics-1.8.0.jar > avatica/1.8.0//avatica-1.8.0.jar > avro/1.8.2//avro-1.8.2.jar > bonecp/0.8.0.RELEASE//bonecp-0.8.0.RELEASE.jar > calcite-core/1.10.0//calcite-core-1.10.0.jar > calcite-druid/1.10.0//calcite-druid-1.10.0.jar > calcite-linq4j/1.10.0//calcite-linq4j-1.10.0.jar > commons-beanutils-core/1.8.0//commons-beanutils-core-1.8.0.jar > commons-beanutils/1.7.0//commons-beanutils-1.7.0.jar > commons-cli/1.2//commons-cli-1.2.jar > commons-codec/1.4//commons-codec-1.4.jar > commons-collections/3.2.2//commons-collections-3.2.2.jar > commons-compiler/2.7.6//commons-compiler-2.7.6.jar > commons-compress/1.9//commons-compress-1.9.jar > commons-configuration/1.6//commons-configuration-1.6.jar > commons-daemon/1.0.13//commons-daemon-1.0.13.jar > commons-dbcp/1.4//commons-dbcp-1.4.jar > commons-digester/1.8//commons-digester-1.8.jar > commons-el/1.0//commons-el-1.0.jar > commons-httpclient/3.1//commons-httpclient-3.1.jar > commons-io/2.4//commons-io-2.4.jar > commons-lang/2.6//commons-lang-2.6.jar > commons-lang3/3.1//commons-lang3-3.1.jar > commons-logging/1.2//commons-logging-1.2.jar > commons-math/2.2//commons-math-2.2.jar > commons-math3/3.1.1//commons-math3-3.1.1.jar > commons-net/3.1//commons-net-3.1.jar > commons-pool/1.5.4//commons-pool-1.5.4.jar > curator-client/2.7.1//curator-client-2.7.1.jar > curator-framework/2.7.1//curator-framework-2.7.1.jar > curator-recipes/2.7.1//curator-recipes-2.7.1.jar > datanucleus-api-jdo/4.2.4//datanucleus-api-jdo-4.2.4.jar > datanucleus-core/4.1.17//datanucleus-core-4.1.17.jar > datanucleus-rdbms/4.1.19//datanucleus-rdbms-4.1.19.jar > derby/10.10.2.0//derby-10.10.2.0.jar > disruptor/3.3.0//disruptor-3.3.0.jar > dropwizard-metrics-hadoop-metrics2-reporter/0.1.2//dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar > eigenbase-properties/1.1.5//eigenbase-properties-1.1.5.jar > fastutil/7.0.13//fastutil-7.0.13.jar > findbugs-annotations/1.3.9-1//findbugs-annotations-1.3.9-1.jar > fluent-hc/4.4.1//fluent-hc-4.4.1.jar > groovy-all/2.4.4//groovy-all-2.4.4.jar > gson/2.3.1//gson-2.3.1.jar > guava/14.0.1//guava-14.0.1.jar > guice-assistedinject/3.0//guice-assistedinject-3.0.jar > guice-servlet/3.0//guice-servlet-3.0.jar > guice/3.0//guice-3.0.jar > hadoop-annotations/2.7.3//hadoop-annotations-2.7.3.jar > hadoop-auth/2.7.3//hadoop-auth-2.7.3.jar > hadoop-client/2.7.3//hadoop-client-2.7.3.jar > hadoop-common/2.7.3//hadoop-common-2.7.3.jar > hadoop-common/2.7.3/tests/hadoop-common-2.7.3-tests.jar > hadoop-hdfs/2.7.3//hadoop-hdfs-2.7.3.jar > hadoop-hdfs/2.7.3/tests/hadoop-hdfs-2.7.3-tests.jar > hadoop-mapreduce-client-app/2.7.3//hadoop-mapreduce-client-app-2.7.3.jar > hadoop-mapreduce-client-common/2.7.3//hadoop-mapreduce-client-common-2.7.3.jar > hadoop-mapreduce-client-core/2.7.3//hadoop-mapreduce-client-core-2.7.3.jar > hadoop-mapreduce-client-jobclient/2.7.3//hadoop-mapreduce-client-jobclient-2.7.3.jar > hadoop-mapreduce-client-shuffle/2.7.3//hadoop-mapreduce-client-shuffle-2.7.3.jar > hadoop-yarn-api/2.7.3//hadoop-yarn-api-2.7.3.jar > hadoop-yarn-client/2.7.3//hadoop-yarn-client-2.7.3.jar > hadoop-yarn-common/2.7.3//hadoop-yarn-common-2.7.3.jar > hadoop-yarn-registry/2.7.1//hadoop-yarn-registry-2.7.1.jar > hadoop-yarn-server-applicationhistoryservice/2.7.2//hadoop-yarn-server-applicationhistoryservice-2.7.2.jar > hadoop-yarn-server-common/2.7.2//hadoop-yarn-server-common-2.7.2.jar > hadoop-yarn-server-resourcemanager/2.7.2//hadoop-yarn-server-resourcemanager-2.7.2.jar > hadoop-yarn-server-web-pr
[jira] [Updated] (HUDI-2440) Add dependency change diff script for dependency governace
[ https://issues.apache.org/jira/browse/HUDI-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang updated HUDI-2440: --- Parent: HUDI-2506 Issue Type: Sub-task (was: Improvement) > Add dependency change diff script for dependency governace > -- > > Key: HUDI-2440 > URL: https://issues.apache.org/jira/browse/HUDI-2440 > Project: Apache Hudi > Issue Type: Sub-task > Components: Usability, Utilities >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > Labels: pull-request-available > Fix For: 0.10.0 > > > Currently, hudi's dependency management is chaotic, e.g. for > `hudi-spark-bundle_2.11`, the dependency list is here: > {code:java} > HikariCP/2.5.1//HikariCP-2.5.1.jar > ST4/4.0.4//ST4-4.0.4.jar > aircompressor/0.15//aircompressor-0.15.jar > annotations/17.0.0//annotations-17.0.0.jar > ant-launcher/1.9.1//ant-launcher-1.9.1.jar > ant/1.6.5//ant-1.6.5.jar > ant/1.9.1//ant-1.9.1.jar > antlr-runtime/3.5.2//antlr-runtime-3.5.2.jar > aopalliance/1.0//aopalliance-1.0.jar > apache-curator/2.7.1//apache-curator-2.7.1.pom > apacheds-i18n/2.0.0-M15//apacheds-i18n-2.0.0-M15.jar > apacheds-kerberos-codec/2.0.0-M15//apacheds-kerberos-codec-2.0.0-M15.jar > api-asn1-api/1.0.0-M20//api-asn1-api-1.0.0-M20.jar > api-util/1.0.0-M20//api-util-1.0.0-M20.jar > asm/3.1//asm-3.1.jar > avatica-metrics/1.8.0//avatica-metrics-1.8.0.jar > avatica/1.8.0//avatica-1.8.0.jar > avro/1.8.2//avro-1.8.2.jar > bonecp/0.8.0.RELEASE//bonecp-0.8.0.RELEASE.jar > calcite-core/1.10.0//calcite-core-1.10.0.jar > calcite-druid/1.10.0//calcite-druid-1.10.0.jar > calcite-linq4j/1.10.0//calcite-linq4j-1.10.0.jar > commons-beanutils-core/1.8.0//commons-beanutils-core-1.8.0.jar > commons-beanutils/1.7.0//commons-beanutils-1.7.0.jar > commons-cli/1.2//commons-cli-1.2.jar > commons-codec/1.4//commons-codec-1.4.jar > commons-collections/3.2.2//commons-collections-3.2.2.jar > commons-compiler/2.7.6//commons-compiler-2.7.6.jar > commons-compress/1.9//commons-compress-1.9.jar > commons-configuration/1.6//commons-configuration-1.6.jar > commons-daemon/1.0.13//commons-daemon-1.0.13.jar > commons-dbcp/1.4//commons-dbcp-1.4.jar > commons-digester/1.8//commons-digester-1.8.jar > commons-el/1.0//commons-el-1.0.jar > commons-httpclient/3.1//commons-httpclient-3.1.jar > commons-io/2.4//commons-io-2.4.jar > commons-lang/2.6//commons-lang-2.6.jar > commons-lang3/3.1//commons-lang3-3.1.jar > commons-logging/1.2//commons-logging-1.2.jar > commons-math/2.2//commons-math-2.2.jar > commons-math3/3.1.1//commons-math3-3.1.1.jar > commons-net/3.1//commons-net-3.1.jar > commons-pool/1.5.4//commons-pool-1.5.4.jar > curator-client/2.7.1//curator-client-2.7.1.jar > curator-framework/2.7.1//curator-framework-2.7.1.jar > curator-recipes/2.7.1//curator-recipes-2.7.1.jar > datanucleus-api-jdo/4.2.4//datanucleus-api-jdo-4.2.4.jar > datanucleus-core/4.1.17//datanucleus-core-4.1.17.jar > datanucleus-rdbms/4.1.19//datanucleus-rdbms-4.1.19.jar > derby/10.10.2.0//derby-10.10.2.0.jar > disruptor/3.3.0//disruptor-3.3.0.jar > dropwizard-metrics-hadoop-metrics2-reporter/0.1.2//dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar > eigenbase-properties/1.1.5//eigenbase-properties-1.1.5.jar > fastutil/7.0.13//fastutil-7.0.13.jar > findbugs-annotations/1.3.9-1//findbugs-annotations-1.3.9-1.jar > fluent-hc/4.4.1//fluent-hc-4.4.1.jar > groovy-all/2.4.4//groovy-all-2.4.4.jar > gson/2.3.1//gson-2.3.1.jar > guava/14.0.1//guava-14.0.1.jar > guice-assistedinject/3.0//guice-assistedinject-3.0.jar > guice-servlet/3.0//guice-servlet-3.0.jar > guice/3.0//guice-3.0.jar > hadoop-annotations/2.7.3//hadoop-annotations-2.7.3.jar > hadoop-auth/2.7.3//hadoop-auth-2.7.3.jar > hadoop-client/2.7.3//hadoop-client-2.7.3.jar > hadoop-common/2.7.3//hadoop-common-2.7.3.jar > hadoop-common/2.7.3/tests/hadoop-common-2.7.3-tests.jar > hadoop-hdfs/2.7.3//hadoop-hdfs-2.7.3.jar > hadoop-hdfs/2.7.3/tests/hadoop-hdfs-2.7.3-tests.jar > hadoop-mapreduce-client-app/2.7.3//hadoop-mapreduce-client-app-2.7.3.jar > hadoop-mapreduce-client-common/2.7.3//hadoop-mapreduce-client-common-2.7.3.jar > hadoop-mapreduce-client-core/2.7.3//hadoop-mapreduce-client-core-2.7.3.jar > hadoop-mapreduce-client-jobclient/2.7.3//hadoop-mapreduce-client-jobclient-2.7.3.jar > hadoop-mapreduce-client-shuffle/2.7.3//hadoop-mapreduce-client-shuffle-2.7.3.jar > hadoop-yarn-api/2.7.3//hadoop-yarn-api-2.7.3.jar > hadoop-yarn-client/2.7.3//hadoop-yarn-client-2.7.3.jar > hadoop-yarn-common/2.7.3//hadoop-yarn-common-2.7.3.jar > hadoop-yarn-registry/2.7.1//hadoop-yarn-registry-2.7.1.jar > hadoop-yarn-server-applicationhistoryservice/2.7.2//hadoop-yarn-server-applicationhistoryservice-2.7.2.jar > hadoop-yarn-server-common/2.7.2//hadoop-yarn-server-common-2.7.2.jar > hadoop-yarn-server-resourcemanager/2.7.2//hadoop
[jira] [Closed] (HUDI-2440) Add dependency change diff script for dependency governace
[ https://issues.apache.org/jira/browse/HUDI-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-2440. -- Resolution: Implemented 47ed91799943271f219419cf209793a98b3f09b5 > Add dependency change diff script for dependency governace > -- > > Key: HUDI-2440 > URL: https://issues.apache.org/jira/browse/HUDI-2440 > Project: Apache Hudi > Issue Type: Improvement > Components: Usability, Utilities >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > Labels: pull-request-available > Fix For: 0.10.0 > > > Currently, hudi's dependency management is chaotic, e.g. for > `hudi-spark-bundle_2.11`, the dependency list is here: > {code:java} > HikariCP/2.5.1//HikariCP-2.5.1.jar > ST4/4.0.4//ST4-4.0.4.jar > aircompressor/0.15//aircompressor-0.15.jar > annotations/17.0.0//annotations-17.0.0.jar > ant-launcher/1.9.1//ant-launcher-1.9.1.jar > ant/1.6.5//ant-1.6.5.jar > ant/1.9.1//ant-1.9.1.jar > antlr-runtime/3.5.2//antlr-runtime-3.5.2.jar > aopalliance/1.0//aopalliance-1.0.jar > apache-curator/2.7.1//apache-curator-2.7.1.pom > apacheds-i18n/2.0.0-M15//apacheds-i18n-2.0.0-M15.jar > apacheds-kerberos-codec/2.0.0-M15//apacheds-kerberos-codec-2.0.0-M15.jar > api-asn1-api/1.0.0-M20//api-asn1-api-1.0.0-M20.jar > api-util/1.0.0-M20//api-util-1.0.0-M20.jar > asm/3.1//asm-3.1.jar > avatica-metrics/1.8.0//avatica-metrics-1.8.0.jar > avatica/1.8.0//avatica-1.8.0.jar > avro/1.8.2//avro-1.8.2.jar > bonecp/0.8.0.RELEASE//bonecp-0.8.0.RELEASE.jar > calcite-core/1.10.0//calcite-core-1.10.0.jar > calcite-druid/1.10.0//calcite-druid-1.10.0.jar > calcite-linq4j/1.10.0//calcite-linq4j-1.10.0.jar > commons-beanutils-core/1.8.0//commons-beanutils-core-1.8.0.jar > commons-beanutils/1.7.0//commons-beanutils-1.7.0.jar > commons-cli/1.2//commons-cli-1.2.jar > commons-codec/1.4//commons-codec-1.4.jar > commons-collections/3.2.2//commons-collections-3.2.2.jar > commons-compiler/2.7.6//commons-compiler-2.7.6.jar > commons-compress/1.9//commons-compress-1.9.jar > commons-configuration/1.6//commons-configuration-1.6.jar > commons-daemon/1.0.13//commons-daemon-1.0.13.jar > commons-dbcp/1.4//commons-dbcp-1.4.jar > commons-digester/1.8//commons-digester-1.8.jar > commons-el/1.0//commons-el-1.0.jar > commons-httpclient/3.1//commons-httpclient-3.1.jar > commons-io/2.4//commons-io-2.4.jar > commons-lang/2.6//commons-lang-2.6.jar > commons-lang3/3.1//commons-lang3-3.1.jar > commons-logging/1.2//commons-logging-1.2.jar > commons-math/2.2//commons-math-2.2.jar > commons-math3/3.1.1//commons-math3-3.1.1.jar > commons-net/3.1//commons-net-3.1.jar > commons-pool/1.5.4//commons-pool-1.5.4.jar > curator-client/2.7.1//curator-client-2.7.1.jar > curator-framework/2.7.1//curator-framework-2.7.1.jar > curator-recipes/2.7.1//curator-recipes-2.7.1.jar > datanucleus-api-jdo/4.2.4//datanucleus-api-jdo-4.2.4.jar > datanucleus-core/4.1.17//datanucleus-core-4.1.17.jar > datanucleus-rdbms/4.1.19//datanucleus-rdbms-4.1.19.jar > derby/10.10.2.0//derby-10.10.2.0.jar > disruptor/3.3.0//disruptor-3.3.0.jar > dropwizard-metrics-hadoop-metrics2-reporter/0.1.2//dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar > eigenbase-properties/1.1.5//eigenbase-properties-1.1.5.jar > fastutil/7.0.13//fastutil-7.0.13.jar > findbugs-annotations/1.3.9-1//findbugs-annotations-1.3.9-1.jar > fluent-hc/4.4.1//fluent-hc-4.4.1.jar > groovy-all/2.4.4//groovy-all-2.4.4.jar > gson/2.3.1//gson-2.3.1.jar > guava/14.0.1//guava-14.0.1.jar > guice-assistedinject/3.0//guice-assistedinject-3.0.jar > guice-servlet/3.0//guice-servlet-3.0.jar > guice/3.0//guice-3.0.jar > hadoop-annotations/2.7.3//hadoop-annotations-2.7.3.jar > hadoop-auth/2.7.3//hadoop-auth-2.7.3.jar > hadoop-client/2.7.3//hadoop-client-2.7.3.jar > hadoop-common/2.7.3//hadoop-common-2.7.3.jar > hadoop-common/2.7.3/tests/hadoop-common-2.7.3-tests.jar > hadoop-hdfs/2.7.3//hadoop-hdfs-2.7.3.jar > hadoop-hdfs/2.7.3/tests/hadoop-hdfs-2.7.3-tests.jar > hadoop-mapreduce-client-app/2.7.3//hadoop-mapreduce-client-app-2.7.3.jar > hadoop-mapreduce-client-common/2.7.3//hadoop-mapreduce-client-common-2.7.3.jar > hadoop-mapreduce-client-core/2.7.3//hadoop-mapreduce-client-core-2.7.3.jar > hadoop-mapreduce-client-jobclient/2.7.3//hadoop-mapreduce-client-jobclient-2.7.3.jar > hadoop-mapreduce-client-shuffle/2.7.3//hadoop-mapreduce-client-shuffle-2.7.3.jar > hadoop-yarn-api/2.7.3//hadoop-yarn-api-2.7.3.jar > hadoop-yarn-client/2.7.3//hadoop-yarn-client-2.7.3.jar > hadoop-yarn-common/2.7.3//hadoop-yarn-common-2.7.3.jar > hadoop-yarn-registry/2.7.1//hadoop-yarn-registry-2.7.1.jar > hadoop-yarn-server-applicationhistoryservice/2.7.2//hadoop-yarn-server-applicationhistoryservice-2.7.2.jar > hadoop-yarn-server-common/2.7.2//hadoop-yarn-server-common-2.7.2.jar > hadoop-yarn-server-resourcemanager/2.7.2//hadoop
[jira] [Updated] (HUDI-2440) Add dependency change diff script for dependency governace
[ https://issues.apache.org/jira/browse/HUDI-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang updated HUDI-2440: --- Fix Version/s: 0.10.0 > Add dependency change diff script for dependency governace > -- > > Key: HUDI-2440 > URL: https://issues.apache.org/jira/browse/HUDI-2440 > Project: Apache Hudi > Issue Type: Improvement > Components: Usability, Utilities >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > Labels: pull-request-available > Fix For: 0.10.0 > > > Currently, hudi's dependency management is chaotic, e.g. for > `hudi-spark-bundle_2.11`, the dependency list is here: > {code:java} > HikariCP/2.5.1//HikariCP-2.5.1.jar > ST4/4.0.4//ST4-4.0.4.jar > aircompressor/0.15//aircompressor-0.15.jar > annotations/17.0.0//annotations-17.0.0.jar > ant-launcher/1.9.1//ant-launcher-1.9.1.jar > ant/1.6.5//ant-1.6.5.jar > ant/1.9.1//ant-1.9.1.jar > antlr-runtime/3.5.2//antlr-runtime-3.5.2.jar > aopalliance/1.0//aopalliance-1.0.jar > apache-curator/2.7.1//apache-curator-2.7.1.pom > apacheds-i18n/2.0.0-M15//apacheds-i18n-2.0.0-M15.jar > apacheds-kerberos-codec/2.0.0-M15//apacheds-kerberos-codec-2.0.0-M15.jar > api-asn1-api/1.0.0-M20//api-asn1-api-1.0.0-M20.jar > api-util/1.0.0-M20//api-util-1.0.0-M20.jar > asm/3.1//asm-3.1.jar > avatica-metrics/1.8.0//avatica-metrics-1.8.0.jar > avatica/1.8.0//avatica-1.8.0.jar > avro/1.8.2//avro-1.8.2.jar > bonecp/0.8.0.RELEASE//bonecp-0.8.0.RELEASE.jar > calcite-core/1.10.0//calcite-core-1.10.0.jar > calcite-druid/1.10.0//calcite-druid-1.10.0.jar > calcite-linq4j/1.10.0//calcite-linq4j-1.10.0.jar > commons-beanutils-core/1.8.0//commons-beanutils-core-1.8.0.jar > commons-beanutils/1.7.0//commons-beanutils-1.7.0.jar > commons-cli/1.2//commons-cli-1.2.jar > commons-codec/1.4//commons-codec-1.4.jar > commons-collections/3.2.2//commons-collections-3.2.2.jar > commons-compiler/2.7.6//commons-compiler-2.7.6.jar > commons-compress/1.9//commons-compress-1.9.jar > commons-configuration/1.6//commons-configuration-1.6.jar > commons-daemon/1.0.13//commons-daemon-1.0.13.jar > commons-dbcp/1.4//commons-dbcp-1.4.jar > commons-digester/1.8//commons-digester-1.8.jar > commons-el/1.0//commons-el-1.0.jar > commons-httpclient/3.1//commons-httpclient-3.1.jar > commons-io/2.4//commons-io-2.4.jar > commons-lang/2.6//commons-lang-2.6.jar > commons-lang3/3.1//commons-lang3-3.1.jar > commons-logging/1.2//commons-logging-1.2.jar > commons-math/2.2//commons-math-2.2.jar > commons-math3/3.1.1//commons-math3-3.1.1.jar > commons-net/3.1//commons-net-3.1.jar > commons-pool/1.5.4//commons-pool-1.5.4.jar > curator-client/2.7.1//curator-client-2.7.1.jar > curator-framework/2.7.1//curator-framework-2.7.1.jar > curator-recipes/2.7.1//curator-recipes-2.7.1.jar > datanucleus-api-jdo/4.2.4//datanucleus-api-jdo-4.2.4.jar > datanucleus-core/4.1.17//datanucleus-core-4.1.17.jar > datanucleus-rdbms/4.1.19//datanucleus-rdbms-4.1.19.jar > derby/10.10.2.0//derby-10.10.2.0.jar > disruptor/3.3.0//disruptor-3.3.0.jar > dropwizard-metrics-hadoop-metrics2-reporter/0.1.2//dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar > eigenbase-properties/1.1.5//eigenbase-properties-1.1.5.jar > fastutil/7.0.13//fastutil-7.0.13.jar > findbugs-annotations/1.3.9-1//findbugs-annotations-1.3.9-1.jar > fluent-hc/4.4.1//fluent-hc-4.4.1.jar > groovy-all/2.4.4//groovy-all-2.4.4.jar > gson/2.3.1//gson-2.3.1.jar > guava/14.0.1//guava-14.0.1.jar > guice-assistedinject/3.0//guice-assistedinject-3.0.jar > guice-servlet/3.0//guice-servlet-3.0.jar > guice/3.0//guice-3.0.jar > hadoop-annotations/2.7.3//hadoop-annotations-2.7.3.jar > hadoop-auth/2.7.3//hadoop-auth-2.7.3.jar > hadoop-client/2.7.3//hadoop-client-2.7.3.jar > hadoop-common/2.7.3//hadoop-common-2.7.3.jar > hadoop-common/2.7.3/tests/hadoop-common-2.7.3-tests.jar > hadoop-hdfs/2.7.3//hadoop-hdfs-2.7.3.jar > hadoop-hdfs/2.7.3/tests/hadoop-hdfs-2.7.3-tests.jar > hadoop-mapreduce-client-app/2.7.3//hadoop-mapreduce-client-app-2.7.3.jar > hadoop-mapreduce-client-common/2.7.3//hadoop-mapreduce-client-common-2.7.3.jar > hadoop-mapreduce-client-core/2.7.3//hadoop-mapreduce-client-core-2.7.3.jar > hadoop-mapreduce-client-jobclient/2.7.3//hadoop-mapreduce-client-jobclient-2.7.3.jar > hadoop-mapreduce-client-shuffle/2.7.3//hadoop-mapreduce-client-shuffle-2.7.3.jar > hadoop-yarn-api/2.7.3//hadoop-yarn-api-2.7.3.jar > hadoop-yarn-client/2.7.3//hadoop-yarn-client-2.7.3.jar > hadoop-yarn-common/2.7.3//hadoop-yarn-common-2.7.3.jar > hadoop-yarn-registry/2.7.1//hadoop-yarn-registry-2.7.1.jar > hadoop-yarn-server-applicationhistoryservice/2.7.2//hadoop-yarn-server-applicationhistoryservice-2.7.2.jar > hadoop-yarn-server-common/2.7.2//hadoop-yarn-server-common-2.7.2.jar > hadoop-yarn-server-resourcemanager/2.7.2//hadoop-yarn-server-resourcemanager-2.7.2.jar > h
[jira] [Closed] (HUDI-2487) An empty message in Kafka causes a task exception
[ https://issues.apache.org/jira/browse/HUDI-2487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-2487. -- Fix Version/s: (was: 0.9.0) 0.10.0 Resolution: Implemented 9067657a5ff313990c819065ad12d71fa8bb0f06 > An empty message in Kafka causes a task exception > - > > Key: HUDI-2487 > URL: https://issues.apache.org/jira/browse/HUDI-2487 > Project: Apache Hudi > Issue Type: Improvement > Components: DeltaStreamer >Reporter: qianchutao >Assignee: qianchutao >Priority: Major > Labels: easyfix, newbie, pull-request-available > Fix For: 0.10.0 > > Original Estimate: 24h > Remaining Estimate: 24h > > h1. Question: > When I use deltaStreamer to update hive tables in upsert mode from json > data in Kafka to HUDi, if the value of the message body in Kafka is null, the > task throws an exception. > h2. Exception description: > Lost task 0.1 in stage 2.0 (TID 24, > node-group-1UtpO.1f562475-6982-4b16-a50d-d19b0ebff950.com, executor 6): > org.apache.hudi.exception.HoodieException: The value of tmSmp can not be null > at > org.apache.hudi.avro.HoodieAvroUtils.getNestedFieldVal(HoodieAvroUtils.java:463) > at > org.apache.hudi.utilities.deltastreamer.DeltaSync.lambda$readFromSource$d62e16$1(DeltaSync.java:389) > at > org.apache.spark.api.java.JavaPairRDD$$anonfun$toScalaFunction$1.apply(JavaPairRDD.scala:1040) > at scala.collection.Iterator$$anon$11.next(Iterator.scala:410) > at scala.collection.Iterator$$anon$11.next(Iterator.scala:410) > at > org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:196) > at > org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:62) > at > org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:58) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:102) > at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:55) > at org.apache.spark.scheduler.Task.run(Task.scala:123) > at > org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:413) > at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1551) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:419) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > h1. The task Settings: > > {code:java} > hoodie.datasource.write.precombine.field=tmSmp > hoodie.datasource.write.recordkey.field=subOrderId,activityId,ticketId > hoodie.datasource.hive_sync.partition_fields=db,dt > hoodie.datasource.write.partitionpath.field=db:SIMPLE,dt:SIMPLE > hoodie.datasource.write.keygenerator.class=org.apache.hudi.keygen.CustomKeyGenerator > hoodie.datasource.hive_sync.enable=true > hoodie.datasource.meta.sync.enable=true > hoodie.datasource.hive_sync.partition_extractor_class=org.apache.hudi.hive.MultiPartKeysValueExtractor > hoodie.datasource.hive_sync.support_timestamp=true > hoodie.datasource.hive_sync.auto_create_database=true > hoodie.meta.sync.client.tool.class=org.apache.hudi.hive.HiveSyncTool > hoodie.datasource.hive_sync.base_file_format=PARQUET > {code} > > > h1. Spark-submit Script parameter Settings: > > {code:java} > --class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer > --source-ordering-field tmSmp \ > --table-type MERGE_ON_READ \ > --target-table ${TABLE_NAME} \ > --source-class org.apache.hudi.utilities.sources.JsonKafkaSource \ > --schemaprovider-class > org.apache.hudi.utilities.schema.FilebasedSchemaProvider \ > --enable-sync \ > --op UPSERT \ > --continuous \ > {code} > > > So I think some optimizations can be made to prevent task throwing, > such as filtering messages with a null value in Kafka. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (HUDI-2447) Extract common parts from 'if' & Fix typo
[ https://issues.apache.org/jira/browse/HUDI-2447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-2447. -- Resolution: Done > Extract common parts from 'if' & Fix typo > - > > Key: HUDI-2447 > URL: https://issues.apache.org/jira/browse/HUDI-2447 > Project: Apache Hudi > Issue Type: Improvement > Components: Hive Integration >Reporter: 董可伦 >Assignee: 董可伦 >Priority: Minor > Labels: pull-request-available > Fix For: 0.10.0 > > > Extract common parts from 'if' & Fix typo -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-2447) Extract common parts from 'if' & Fix typo
[ https://issues.apache.org/jira/browse/HUDI-2447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang updated HUDI-2447: --- Priority: Minor (was: Major) > Extract common parts from 'if' & Fix typo > - > > Key: HUDI-2447 > URL: https://issues.apache.org/jira/browse/HUDI-2447 > Project: Apache Hudi > Issue Type: Improvement > Components: Hive Integration >Reporter: 董可伦 >Assignee: 董可伦 >Priority: Minor > Labels: pull-request-available > Fix For: 0.10.0 > > > Extract common parts from 'if' & Fix typo -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (HUDI-2434) Add GraphiteReporter reporter periodSeconds config
[ https://issues.apache.org/jira/browse/HUDI-2434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-2434. -- Resolution: Done > Add GraphiteReporter reporter periodSeconds config > -- > > Key: HUDI-2434 > URL: https://issues.apache.org/jira/browse/HUDI-2434 > Project: Apache Hudi > Issue Type: Improvement >Reporter: liujinhui >Assignee: liujinhui >Priority: Major > Labels: pull-request-available > Fix For: 0.10.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-2440) Add dependency change diff script for dependency governace
vinoyang created HUDI-2440: -- Summary: Add dependency change diff script for dependency governace Key: HUDI-2440 URL: https://issues.apache.org/jira/browse/HUDI-2440 Project: Apache Hudi Issue Type: Improvement Components: Utilities Reporter: vinoyang Assignee: vinoyang Currently, hudi's dependency management is chaotic, e.g. for `hudi-spark-bundle_2.11`, the dependency list is here: {code:java} HikariCP/2.5.1//HikariCP-2.5.1.jar ST4/4.0.4//ST4-4.0.4.jar aircompressor/0.15//aircompressor-0.15.jar annotations/17.0.0//annotations-17.0.0.jar ant-launcher/1.9.1//ant-launcher-1.9.1.jar ant/1.6.5//ant-1.6.5.jar ant/1.9.1//ant-1.9.1.jar antlr-runtime/3.5.2//antlr-runtime-3.5.2.jar aopalliance/1.0//aopalliance-1.0.jar apache-curator/2.7.1//apache-curator-2.7.1.pom apacheds-i18n/2.0.0-M15//apacheds-i18n-2.0.0-M15.jar apacheds-kerberos-codec/2.0.0-M15//apacheds-kerberos-codec-2.0.0-M15.jar api-asn1-api/1.0.0-M20//api-asn1-api-1.0.0-M20.jar api-util/1.0.0-M20//api-util-1.0.0-M20.jar asm/3.1//asm-3.1.jar avatica-metrics/1.8.0//avatica-metrics-1.8.0.jar avatica/1.8.0//avatica-1.8.0.jar avro/1.8.2//avro-1.8.2.jar bonecp/0.8.0.RELEASE//bonecp-0.8.0.RELEASE.jar calcite-core/1.10.0//calcite-core-1.10.0.jar calcite-druid/1.10.0//calcite-druid-1.10.0.jar calcite-linq4j/1.10.0//calcite-linq4j-1.10.0.jar commons-beanutils-core/1.8.0//commons-beanutils-core-1.8.0.jar commons-beanutils/1.7.0//commons-beanutils-1.7.0.jar commons-cli/1.2//commons-cli-1.2.jar commons-codec/1.4//commons-codec-1.4.jar commons-collections/3.2.2//commons-collections-3.2.2.jar commons-compiler/2.7.6//commons-compiler-2.7.6.jar commons-compress/1.9//commons-compress-1.9.jar commons-configuration/1.6//commons-configuration-1.6.jar commons-daemon/1.0.13//commons-daemon-1.0.13.jar commons-dbcp/1.4//commons-dbcp-1.4.jar commons-digester/1.8//commons-digester-1.8.jar commons-el/1.0//commons-el-1.0.jar commons-httpclient/3.1//commons-httpclient-3.1.jar commons-io/2.4//commons-io-2.4.jar commons-lang/2.6//commons-lang-2.6.jar commons-lang3/3.1//commons-lang3-3.1.jar commons-logging/1.2//commons-logging-1.2.jar commons-math/2.2//commons-math-2.2.jar commons-math3/3.1.1//commons-math3-3.1.1.jar commons-net/3.1//commons-net-3.1.jar commons-pool/1.5.4//commons-pool-1.5.4.jar curator-client/2.7.1//curator-client-2.7.1.jar curator-framework/2.7.1//curator-framework-2.7.1.jar curator-recipes/2.7.1//curator-recipes-2.7.1.jar datanucleus-api-jdo/4.2.4//datanucleus-api-jdo-4.2.4.jar datanucleus-core/4.1.17//datanucleus-core-4.1.17.jar datanucleus-rdbms/4.1.19//datanucleus-rdbms-4.1.19.jar derby/10.10.2.0//derby-10.10.2.0.jar disruptor/3.3.0//disruptor-3.3.0.jar dropwizard-metrics-hadoop-metrics2-reporter/0.1.2//dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar eigenbase-properties/1.1.5//eigenbase-properties-1.1.5.jar fastutil/7.0.13//fastutil-7.0.13.jar findbugs-annotations/1.3.9-1//findbugs-annotations-1.3.9-1.jar fluent-hc/4.4.1//fluent-hc-4.4.1.jar groovy-all/2.4.4//groovy-all-2.4.4.jar gson/2.3.1//gson-2.3.1.jar guava/14.0.1//guava-14.0.1.jar guice-assistedinject/3.0//guice-assistedinject-3.0.jar guice-servlet/3.0//guice-servlet-3.0.jar guice/3.0//guice-3.0.jar hadoop-annotations/2.7.3//hadoop-annotations-2.7.3.jar hadoop-auth/2.7.3//hadoop-auth-2.7.3.jar hadoop-client/2.7.3//hadoop-client-2.7.3.jar hadoop-common/2.7.3//hadoop-common-2.7.3.jar hadoop-common/2.7.3/tests/hadoop-common-2.7.3-tests.jar hadoop-hdfs/2.7.3//hadoop-hdfs-2.7.3.jar hadoop-hdfs/2.7.3/tests/hadoop-hdfs-2.7.3-tests.jar hadoop-mapreduce-client-app/2.7.3//hadoop-mapreduce-client-app-2.7.3.jar hadoop-mapreduce-client-common/2.7.3//hadoop-mapreduce-client-common-2.7.3.jar hadoop-mapreduce-client-core/2.7.3//hadoop-mapreduce-client-core-2.7.3.jar hadoop-mapreduce-client-jobclient/2.7.3//hadoop-mapreduce-client-jobclient-2.7.3.jar hadoop-mapreduce-client-shuffle/2.7.3//hadoop-mapreduce-client-shuffle-2.7.3.jar hadoop-yarn-api/2.7.3//hadoop-yarn-api-2.7.3.jar hadoop-yarn-client/2.7.3//hadoop-yarn-client-2.7.3.jar hadoop-yarn-common/2.7.3//hadoop-yarn-common-2.7.3.jar hadoop-yarn-registry/2.7.1//hadoop-yarn-registry-2.7.1.jar hadoop-yarn-server-applicationhistoryservice/2.7.2//hadoop-yarn-server-applicationhistoryservice-2.7.2.jar hadoop-yarn-server-common/2.7.2//hadoop-yarn-server-common-2.7.2.jar hadoop-yarn-server-resourcemanager/2.7.2//hadoop-yarn-server-resourcemanager-2.7.2.jar hadoop-yarn-server-web-proxy/2.7.2//hadoop-yarn-server-web-proxy-2.7.2.jar hamcrest-core/1.3//hamcrest-core-1.3.jar hbase-annotations/1.2.3//hbase-annotations-1.2.3.jar hbase-client/1.2.3//hbase-client-1.2.3.jar hbase-common/1.2.3//hbase-common-1.2.3.jar hbase-common/1.2.3/tests/hbase-common-1.2.3-tests.jar hbase-hadoop-compat/1.2.3//hbase-hadoop-compat-1.2.3.jar hbase-hadoop2-compat/1.2.3//hbase-hadoop2-compat-1.2.3.jar hbase-prefix-tree/1.2.3//hbase-prefix-tree-1.2.3.jar hbase-proced
[jira] [Closed] (HUDI-2423) Separate some config logic from HoodieMetricsConfig into HoodieMetricsGraphiteConfig HoodieMetricsJmxConfig
[ https://issues.apache.org/jira/browse/HUDI-2423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-2423. -- Resolution: Done > Separate some config logic from HoodieMetricsConfig into > HoodieMetricsGraphiteConfig HoodieMetricsJmxConfig > --- > > Key: HUDI-2423 > URL: https://issues.apache.org/jira/browse/HUDI-2423 > Project: Apache Hudi > Issue Type: Improvement >Reporter: liujinhui >Assignee: liujinhui >Priority: Major > Labels: pull-request-available > Fix For: 0.10.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-2423) Separate some config logic from HoodieMetricsConfig into HoodieMetricsGraphiteConfig HoodieMetricsJmxConfig
[ https://issues.apache.org/jira/browse/HUDI-2423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang updated HUDI-2423: --- Fix Version/s: 0.10.0 > Separate some config logic from HoodieMetricsConfig into > HoodieMetricsGraphiteConfig HoodieMetricsJmxConfig > --- > > Key: HUDI-2423 > URL: https://issues.apache.org/jira/browse/HUDI-2423 > Project: Apache Hudi > Issue Type: Improvement >Reporter: liujinhui >Assignee: liujinhui >Priority: Major > Labels: pull-request-available > Fix For: 0.10.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-2423) Separate some config logic from HoodieMetricsConfig into HoodieMetricsGraphiteConfig HoodieMetricsJmxConfig
[ https://issues.apache.org/jira/browse/HUDI-2423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang updated HUDI-2423: --- Summary: Separate some config logic from HoodieMetricsConfig into HoodieMetricsGraphiteConfig HoodieMetricsJmxConfig (was: Breakdown HoodieMetricsConfig into HoodieMetricsGraphiteConfig、HoodieMetricsJmxConfig...) > Separate some config logic from HoodieMetricsConfig into > HoodieMetricsGraphiteConfig HoodieMetricsJmxConfig > --- > > Key: HUDI-2423 > URL: https://issues.apache.org/jira/browse/HUDI-2423 > Project: Apache Hudi > Issue Type: Improvement >Reporter: liujinhui >Assignee: liujinhui >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-2410) Fix getDefaultBootstrapIndexClass logical error
[ https://issues.apache.org/jira/browse/HUDI-2410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang updated HUDI-2410: --- Description: {code:java} public static String getDefaultBootstrapIndexClass(Properties props) { String defaultClass = BOOTSTRAP_INDEX_CLASS_NAME.defaultValue(); if ("false".equalsIgnoreCase(props.getProperty(BOOTSTRAP_INDEX_ENABLE.key({ defaultClass = NO_OP_BOOTSTRAP_INDEX_CLASS; } return defaultClass; } {code} When hoodie.bootstrap.index.enable is not passed, the original logic will follow HFileBootstrapIndex, This should not be judged here was: public static String getDefaultBootstrapIndexClass(Properties props) { String defaultClass = BOOTSTRAP_INDEX_CLASS_NAME.defaultValue(); if ("false".equalsIgnoreCase(props.getProperty(BOOTSTRAP_INDEX_ENABLE.key( { defaultClass = NO_OP_BOOTSTRAP_INDEX_CLASS; } return defaultClass; } When hoodie.bootstrap.index.enable is not passed, the original logic will follow HFileBootstrapIndex, This should not be judged here > Fix getDefaultBootstrapIndexClass logical error > --- > > Key: HUDI-2410 > URL: https://issues.apache.org/jira/browse/HUDI-2410 > Project: Apache Hudi > Issue Type: Bug > Components: bootstrap >Reporter: liujinhui >Assignee: liujinhui >Priority: Major > Labels: pull-request-available > Fix For: 0.10.0 > > > > {code:java} > public static String getDefaultBootstrapIndexClass(Properties props) { > String defaultClass = BOOTSTRAP_INDEX_CLASS_NAME.defaultValue(); > if > ("false".equalsIgnoreCase(props.getProperty(BOOTSTRAP_INDEX_ENABLE.key({ > defaultClass = NO_OP_BOOTSTRAP_INDEX_CLASS; > } > return defaultClass; > } > {code} > > When hoodie.bootstrap.index.enable is not passed, the original logic will > follow HFileBootstrapIndex, > This should not be judged here -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (HUDI-2410) Fix getDefaultBootstrapIndexClass logical error
[ https://issues.apache.org/jira/browse/HUDI-2410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-2410. -- Resolution: Fixed 9f3c4a2a7f565f7bcc32189a202a3d400ece23f1 > Fix getDefaultBootstrapIndexClass logical error > --- > > Key: HUDI-2410 > URL: https://issues.apache.org/jira/browse/HUDI-2410 > Project: Apache Hudi > Issue Type: Bug > Components: bootstrap >Reporter: liujinhui >Assignee: liujinhui >Priority: Major > Labels: pull-request-available > Fix For: 0.10.0 > > > public static String getDefaultBootstrapIndexClass(Properties props) { > String defaultClass = BOOTSTRAP_INDEX_CLASS_NAME.defaultValue(); > if > ("false".equalsIgnoreCase(props.getProperty(BOOTSTRAP_INDEX_ENABLE.key( > { defaultClass = NO_OP_BOOTSTRAP_INDEX_CLASS; } > return defaultClass; > } > When hoodie.bootstrap.index.enable is not passed, the original logic will > follow HFileBootstrapIndex, > This should not be judged here -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HUDI-2410) Fix getDefaultBootstrapIndexClass logical error
[ https://issues.apache.org/jira/browse/HUDI-2410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang reassigned HUDI-2410: -- Assignee: liujinhui > Fix getDefaultBootstrapIndexClass logical error > --- > > Key: HUDI-2410 > URL: https://issues.apache.org/jira/browse/HUDI-2410 > Project: Apache Hudi > Issue Type: Bug > Components: bootstrap >Reporter: liujinhui >Assignee: liujinhui >Priority: Major > Labels: pull-request-available > Fix For: 0.10.0 > > > public static String getDefaultBootstrapIndexClass(Properties props) { > String defaultClass = BOOTSTRAP_INDEX_CLASS_NAME.defaultValue(); > if > ("false".equalsIgnoreCase(props.getProperty(BOOTSTRAP_INDEX_ENABLE.key( > { defaultClass = NO_OP_BOOTSTRAP_INDEX_CLASS; } > return defaultClass; > } > When hoodie.bootstrap.index.enable is not passed, the original logic will > follow HFileBootstrapIndex, > This should not be judged here -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (HUDI-2411) Remove unnecessary method overriden and note
[ https://issues.apache.org/jira/browse/HUDI-2411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-2411. -- Resolution: Done 44b9bc145e0d101bcc688f11c6a30ebcbb7a4a7d > Remove unnecessary method overriden and note > - > > Key: HUDI-2411 > URL: https://issues.apache.org/jira/browse/HUDI-2411 > Project: Apache Hudi > Issue Type: Task >Reporter: Xianghu Wang >Assignee: Xianghu Wang >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-2384) Allow log file size more than 2GB
[ https://issues.apache.org/jira/browse/HUDI-2384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang updated HUDI-2384: --- Fix Version/s: 0.10.0 > Allow log file size more than 2GB > - > > Key: HUDI-2384 > URL: https://issues.apache.org/jira/browse/HUDI-2384 > Project: Apache Hudi > Issue Type: Improvement > Components: Common Core >Reporter: XiaoyuGeng >Assignee: XiaoyuGeng >Priority: Minor > Labels: pull-request-available > Fix For: 0.10.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (HUDI-2384) Allow log file size more than 2GB
[ https://issues.apache.org/jira/browse/HUDI-2384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-2384. -- Resolution: Done 21fd6edfe7721c674b40877fbbdbac71b36bf782 > Allow log file size more than 2GB > - > > Key: HUDI-2384 > URL: https://issues.apache.org/jira/browse/HUDI-2384 > Project: Apache Hudi > Issue Type: Improvement > Components: Common Core >Reporter: XiaoyuGeng >Assignee: XiaoyuGeng >Priority: Minor > Labels: pull-request-available > Fix For: 0.10.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (HUDI-2320) Add support ByteArrayDeserializer in AvroKafkaSource
[ https://issues.apache.org/jira/browse/HUDI-2320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-2320. -- Resolution: Done bf5a52e51bbeaa089995335a0a4c55884792e505 > Add support ByteArrayDeserializer in AvroKafkaSource > > > Key: HUDI-2320 > URL: https://issues.apache.org/jira/browse/HUDI-2320 > Project: Apache Hudi > Issue Type: Improvement > Components: DeltaStreamer >Reporter: 董可伦 >Assignee: 董可伦 >Priority: Major > Labels: pull-request-available > Fix For: 0.10.0 > > > When the 'value.serializer' of Kafka Avro Producer is > 'org.apache.kafka.common.serialization.ByteArraySerializer',Use the following > configuration > {code:java} > --source-class org.apache.hudi.utilities.sources.AvroKafkaSource \ > --schemaprovider-class > org.apache.hudi.utilities.schema.JdbcbasedSchemaProvider \ > --hoodie-conf > "hoodie.deltastreamer.source.kafka.value.deserializer.class=org.apache.kafka.common.serialization.ByteArrayDeserializer" > {code} > For now,It will throw an exception:: > {code:java} > java.lang.ClassCastException: [B cannot be cast to > org.apache.avro.generic.GenericRecord{code} > After support ByteArrayDeserializer,Use the configuration above,It works > properly.And there is no need to provide 'schema.registry.url',For example, > we can use the JdbcbasedSchemaProvider to get the sourceSchema -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (HUDI-2225) Add compaction example
[ https://issues.apache.org/jira/browse/HUDI-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-2225. -- Fix Version/s: 0.9.0 Resolution: Done aa857beee00a764cee90d6e790ee4b0ab4ad4862 > Add compaction example > -- > > Key: HUDI-2225 > URL: https://issues.apache.org/jira/browse/HUDI-2225 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: Sagar Sumit >Assignee: Sagar Sumit >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HUDI-2244) Fix database alreadyExistsException while hive sync
[ https://issues.apache.org/jira/browse/HUDI-2244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang reassigned HUDI-2244: -- Assignee: Zheng yunhong > Fix database alreadyExistsException while hive sync > --- > > Key: HUDI-2244 > URL: https://issues.apache.org/jira/browse/HUDI-2244 > Project: Apache Hudi > Issue Type: Bug > Components: Hive Integration >Reporter: Zheng yunhong >Assignee: Zheng yunhong >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > > Fix database alreadyExistsException while hive sync. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (HUDI-2244) Fix database alreadyExistsException while hive sync
[ https://issues.apache.org/jira/browse/HUDI-2244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-2244. -- Resolution: Fixed eedfadeb46d5538bc7efb2c455469f1b42e9385e > Fix database alreadyExistsException while hive sync > --- > > Key: HUDI-2244 > URL: https://issues.apache.org/jira/browse/HUDI-2244 > Project: Apache Hudi > Issue Type: Bug > Components: Hive Integration >Reporter: Zheng yunhong >Assignee: Zheng yunhong >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > > Fix database alreadyExistsException while hive sync. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (HUDI-2230) "Task not serializable" exception due to non-serializable Codahale Timers
[ https://issues.apache.org/jira/browse/HUDI-2230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-2230. -- Resolution: Fixed 8105cf588e28820b9c021c9ed0e59e3f8b6efa71 > "Task not serializable" exception due to non-serializable Codahale Timers > - > > Key: HUDI-2230 > URL: https://issues.apache.org/jira/browse/HUDI-2230 > Project: Apache Hudi > Issue Type: Bug > Components: metrics >Affects Versions: 0.9.0 >Reporter: Dave Hagman >Assignee: Dave Hagman >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > > Steps to reproduce: > * Enable graphite metrics via props file. Example: > {noformat} > hoodie.metrics.on=true > hoodie.metrics.reporter.type=GRAPHITE > hoodie.metrics.graphite.host= > hoodie.metrics.graphite.port= > hoodie.metrics.graphite.metric.prefix= > {noformat} > * Run the Deltastreamer > * Note the following exception: > {noformat} > Exception in thread "main" org.apache.hudi.exception.HoodieException: > org.apache.hudi.exception.HoodieException: Task not serializable > at > org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.lambda$sync$1(HoodieDeltaStreamer.java:165) > at org.apache.hudi.common.util.Option.ifPresent(Option.java:96) > at > org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.sync(HoodieDeltaStreamer.java:160) > at > org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.main(HoodieDeltaStreamer.java:501) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) > at > org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:959) > at > org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180) > at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203) > at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90) > at > org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1038) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1047) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > Caused by: java.util.concurrent.ExecutionException: > org.apache.hudi.exception.HoodieException: Task not serializable > at > java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) > at > java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908) > at > org.apache.hudi.async.HoodieAsyncService.waitForShutdown(HoodieAsyncService.java:90) > at > org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.lambda$sync$1(HoodieDeltaStreamer.java:163) > ... 15 more > Caused by: org.apache.hudi.exception.HoodieException: Task not serializable > at > org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.lambda$startService$0(HoodieDeltaStreamer.java:649) > at > java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: org.apache.spark.SparkException: Task not serializable > at > org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:416) > at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:406) > at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:162) > at org.apache.spark.SparkContext.clean(SparkContext.scala:2502) > at org.apache.spark.rdd.RDD.$anonfun$map$1(RDD.scala:422) > at > org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) > at > org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112) > at org.apache.spark.rdd.RDD.withScope(RDD.scala:414) > at org.apache.spark.rdd.RDD.map(RDD.scala:421) > at org.apache.spark.api.java.JavaRDDLike.map(JavaRDDLike.scala:93) > at org.apache.spark.api.java.JavaRDDLike.map$(JavaRDDLike.scala:92) > at > org.apache.spark.api.java.AbstractJavaRDDLike.map(JavaRDDLike.scala:45) > ... > at > org.apache.hudi.utilities.sources.RowSource.fetchNewData(RowSource.java:43) > at org.apache.hudi.utilities.sources.Source.fetchNext(Source.java:76) > at
[jira] [Updated] (HUDI-2216) the words 'fiels' in the comments is incorrect
[ https://issues.apache.org/jira/browse/HUDI-2216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang updated HUDI-2216: --- Issue Type: Improvement (was: Bug) > the words 'fiels' in the comments is incorrect > -- > > Key: HUDI-2216 > URL: https://issues.apache.org/jira/browse/HUDI-2216 > Project: Apache Hudi > Issue Type: Improvement > Components: Docs >Affects Versions: 0.9.0 >Reporter: 董可伦 >Assignee: 董可伦 >Priority: Major > Labels: documentation, pull-request-available > Fix For: 0.9.0 > > Attachments: HUDI-2216.png > > > the words 'fiels' in the comments of MergeIntoHoodieTableCommand is > incorrect,it should be > 'fields' > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (HUDI-2216) the words 'fiels' in the comments is incorrect
[ https://issues.apache.org/jira/browse/HUDI-2216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-2216. -- Resolution: Done a91296f14a037a148d949b2380ad503677e688c7 > the words 'fiels' in the comments is incorrect > -- > > Key: HUDI-2216 > URL: https://issues.apache.org/jira/browse/HUDI-2216 > Project: Apache Hudi > Issue Type: Improvement > Components: Docs >Affects Versions: 0.9.0 >Reporter: 董可伦 >Assignee: 董可伦 >Priority: Trivial > Labels: documentation, pull-request-available > Fix For: 0.9.0 > > Attachments: HUDI-2216.png > > > the words 'fiels' in the comments of MergeIntoHoodieTableCommand is > incorrect,it should be > 'fields' > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-2216) the words 'fiels' in the comments is incorrect
[ https://issues.apache.org/jira/browse/HUDI-2216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang updated HUDI-2216: --- Priority: Trivial (was: Major) > the words 'fiels' in the comments is incorrect > -- > > Key: HUDI-2216 > URL: https://issues.apache.org/jira/browse/HUDI-2216 > Project: Apache Hudi > Issue Type: Improvement > Components: Docs >Affects Versions: 0.9.0 >Reporter: 董可伦 >Assignee: 董可伦 >Priority: Trivial > Labels: documentation, pull-request-available > Fix For: 0.9.0 > > Attachments: HUDI-2216.png > > > the words 'fiels' in the comments of MergeIntoHoodieTableCommand is > incorrect,it should be > 'fields' > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-2216) the words 'fiels' in the comments is incorrect
[ https://issues.apache.org/jira/browse/HUDI-2216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17386805#comment-17386805 ] vinoyang commented on HUDI-2216: [~dongkelun] I have given you Jira contributor permission. thanks~ > the words 'fiels' in the comments is incorrect > -- > > Key: HUDI-2216 > URL: https://issues.apache.org/jira/browse/HUDI-2216 > Project: Apache Hudi > Issue Type: Bug > Components: Docs >Affects Versions: 0.9.0 >Reporter: 董可伦 >Assignee: 董可伦 >Priority: Major > Labels: documentation, pull-request-available > Fix For: 0.9.0 > > Attachments: HUDI-2216.png > > > the words 'fiels' in the comments of MergeIntoHoodieTableCommand is > incorrect,it should be > 'fields' > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HUDI-2216) the words 'fiels' in the comments is incorrect
[ https://issues.apache.org/jira/browse/HUDI-2216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang reassigned HUDI-2216: -- Assignee: 董可伦 > the words 'fiels' in the comments is incorrect > -- > > Key: HUDI-2216 > URL: https://issues.apache.org/jira/browse/HUDI-2216 > Project: Apache Hudi > Issue Type: Bug > Components: Docs >Affects Versions: 0.9.0 >Reporter: 董可伦 >Assignee: 董可伦 >Priority: Major > Labels: documentation, pull-request-available > Fix For: 0.9.0 > > Attachments: HUDI-2216.png > > > the words 'fiels' in the comments of MergeIntoHoodieTableCommand is > incorrect,it should be > 'fields' > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (HUDI-2213) Remove unnecessary parameter for HoodieMetrics constructor and fix NPE in UT
[ https://issues.apache.org/jira/browse/HUDI-2213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-2213. -- Resolution: Done 71e14cf866d2ddeee7238a3f15b59912c2537943 > Remove unnecessary parameter for HoodieMetrics constructor and fix NPE in UT > > > Key: HUDI-2213 > URL: https://issues.apache.org/jira/browse/HUDI-2213 > Project: Apache Hudi > Issue Type: Improvement > Components: metrics >Reporter: Xuedong Luan >Assignee: Xuedong Luan >Priority: Minor > Labels: pull-request-available > Fix For: 0.9.0 > > > Remove unnecessary parameter table name for HoodieMetrics constructor, > because it can get by config.getTableName(). And also fix some > NullPointerException in metrics UT. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-2213) Remove unnecessary parameter for HoodieMetrics constructor and fix NPE in UT
[ https://issues.apache.org/jira/browse/HUDI-2213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang updated HUDI-2213: --- Fix Version/s: 0.9.0 > Remove unnecessary parameter for HoodieMetrics constructor and fix NPE in UT > > > Key: HUDI-2213 > URL: https://issues.apache.org/jira/browse/HUDI-2213 > Project: Apache Hudi > Issue Type: Improvement > Components: metrics >Reporter: Xuedong Luan >Assignee: Xuedong Luan >Priority: Minor > Labels: pull-request-available > Fix For: 0.9.0 > > > Remove unnecessary parameter table name for HoodieMetrics constructor, > because it can get by config.getTableName(). And also fix some > NullPointerException in metrics UT. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-2211) Fix NullPointerException in TestHoodieConsoleMetrics
[ https://issues.apache.org/jira/browse/HUDI-2211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang updated HUDI-2211: --- Fix Version/s: 0.9.0 > Fix NullPointerException in TestHoodieConsoleMetrics > > > Key: HUDI-2211 > URL: https://issues.apache.org/jira/browse/HUDI-2211 > Project: Apache Hudi > Issue Type: Improvement > Components: metrics >Reporter: Xuedong Luan >Assignee: Xuedong Luan >Priority: Minor > Labels: pull-request-available > Fix For: 0.9.0 > > > java.lang.NullPointerException: Expected a non-null value. Got > nulljava.lang.NullPointerException: Expected a non-null value. Got null at > org.apache.hudi.common.util.Option.(Option.java:65) at > org.apache.hudi.common.util.Option.of(Option.java:75) at > org.apache.hudi.metrics.Metrics.registerHoodieCommonMetrics(Metrics.java:85) > at org.apache.hudi.metrics.Metrics.reportAndCloseReporter(Metrics.java:63) at > org.apache.hudi.metrics.Metrics.shutdown(Metrics.java:109) at > org.apache.hudi.metrics.TestHoodieConsoleMetrics.stop(TestHoodieConsoleMetrics.java:48) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (HUDI-2211) Fix NullPointerException in TestHoodieConsoleMetrics
[ https://issues.apache.org/jira/browse/HUDI-2211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-2211. -- Resolution: Done 6d592c5896d033c7f781d4a9eef4a43916c084ed > Fix NullPointerException in TestHoodieConsoleMetrics > > > Key: HUDI-2211 > URL: https://issues.apache.org/jira/browse/HUDI-2211 > Project: Apache Hudi > Issue Type: Improvement > Components: metrics >Reporter: Xuedong Luan >Assignee: Xuedong Luan >Priority: Minor > Labels: pull-request-available > Fix For: 0.9.0 > > > java.lang.NullPointerException: Expected a non-null value. Got > nulljava.lang.NullPointerException: Expected a non-null value. Got null at > org.apache.hudi.common.util.Option.(Option.java:65) at > org.apache.hudi.common.util.Option.of(Option.java:75) at > org.apache.hudi.metrics.Metrics.registerHoodieCommonMetrics(Metrics.java:85) > at org.apache.hudi.metrics.Metrics.reportAndCloseReporter(Metrics.java:63) at > org.apache.hudi.metrics.Metrics.shutdown(Metrics.java:109) at > org.apache.hudi.metrics.TestHoodieConsoleMetrics.stop(TestHoodieConsoleMetrics.java:48) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (HUDI-2165) Support Transformer for HoodieFlinkStreamer
[ https://issues.apache.org/jira/browse/HUDI-2165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-2165. -- Resolution: Implemented 52524b659d2cb64403e8ba87d2fefe6d536156e9 > Support Transformer for HoodieFlinkStreamer > --- > > Key: HUDI-2165 > URL: https://issues.apache.org/jira/browse/HUDI-2165 > Project: Apache Hudi > Issue Type: New Feature > Components: Flink Integration >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > > Hoodie's delta streamer support {{Transformer}} , we can also provide this > feature for {{HoodieFlinkStreamer}}. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-2165) Support Transformer for HoodieFlinkStreamer
[ https://issues.apache.org/jira/browse/HUDI-2165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang updated HUDI-2165: --- Fix Version/s: 0.9.0 > Support Transformer for HoodieFlinkStreamer > --- > > Key: HUDI-2165 > URL: https://issues.apache.org/jira/browse/HUDI-2165 > Project: Apache Hudi > Issue Type: Improvement > Components: Flink Integration >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > > Hoodie's delta streamer support {{Transformer}} , we can also provide this > feature for {{HoodieFlinkStreamer}}. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-2165) Support Transformer for HoodieFlinkStreamer
[ https://issues.apache.org/jira/browse/HUDI-2165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang updated HUDI-2165: --- Issue Type: New Feature (was: Improvement) > Support Transformer for HoodieFlinkStreamer > --- > > Key: HUDI-2165 > URL: https://issues.apache.org/jira/browse/HUDI-2165 > Project: Apache Hudi > Issue Type: New Feature > Components: Flink Integration >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > > Hoodie's delta streamer support {{Transformer}} , we can also provide this > feature for {{HoodieFlinkStreamer}}. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HUDI-2165) Support Transformer for HoodieFlinkStreamer
[ https://issues.apache.org/jira/browse/HUDI-2165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang reassigned HUDI-2165: -- Assignee: vinoyang > Support Transformer for HoodieFlinkStreamer > --- > > Key: HUDI-2165 > URL: https://issues.apache.org/jira/browse/HUDI-2165 > Project: Apache Hudi > Issue Type: Improvement > Components: Flink Integration >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > > Hoodie's delta streamer support {{Transformer}} , we can also provide this > feature for {{HoodieFlinkStreamer}}. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-2165) Support Transformer for HoodieFlinkStreamer
vinoyang created HUDI-2165: -- Summary: Support Transformer for HoodieFlinkStreamer Key: HUDI-2165 URL: https://issues.apache.org/jira/browse/HUDI-2165 Project: Apache Hudi Issue Type: Improvement Components: Flink Integration Reporter: vinoyang Hoodie's delta streamer support {{Transformer}} , we can also provide this feature for {{HoodieFlinkStreamer}}. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HUDI-2142) Support setting bucket assign parallelism for flink write task
[ https://issues.apache.org/jira/browse/HUDI-2142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang reassigned HUDI-2142: -- Assignee: Zheng yunhong > Support setting bucket assign parallelism for flink write task > -- > > Key: HUDI-2142 > URL: https://issues.apache.org/jira/browse/HUDI-2142 > Project: Apache Hudi > Issue Type: Improvement > Components: Flink Integration >Reporter: Zheng yunhong >Assignee: Zheng yunhong >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > > Support setting bucket assign parallelism for flink write task. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (HUDI-2142) Support setting bucket assign parallelism for flink write task
[ https://issues.apache.org/jira/browse/HUDI-2142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-2142. -- Resolution: Implemented 9b01d2a04520db6230cd16ef2b29013c013b1944 > Support setting bucket assign parallelism for flink write task > -- > > Key: HUDI-2142 > URL: https://issues.apache.org/jira/browse/HUDI-2142 > Project: Apache Hudi > Issue Type: Improvement > Components: Flink Integration >Reporter: Zheng yunhong >Assignee: Zheng yunhong >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > > Support setting bucket assign parallelism for flink write task. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (HUDI-2143) Tweak the default compaction target IO to 500GB when flink async compaction is off
[ https://issues.apache.org/jira/browse/HUDI-2143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-2143. -- Resolution: Done 942a024e74af52e09cabbfe967f5da0ef108bdbb > Tweak the default compaction target IO to 500GB when flink async compaction > is off > -- > > Key: HUDI-2143 > URL: https://issues.apache.org/jira/browse/HUDI-2143 > Project: Apache Hudi > Issue Type: Improvement > Components: Flink Integration >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (HUDI-2147) Remove unused class AvroConvertor in hudi-flink
[ https://issues.apache.org/jira/browse/HUDI-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-2147. -- Resolution: Done 3b2a4f2b6b49e13997292ecafa9accdd3e7b9efd > Remove unused class AvroConvertor in hudi-flink > --- > > Key: HUDI-2147 > URL: https://issues.apache.org/jira/browse/HUDI-2147 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Xianghu Wang >Assignee: Xianghu Wang >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-2147) Remove unused class AvroConvertor in hudi-flink
[ https://issues.apache.org/jira/browse/HUDI-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang updated HUDI-2147: --- Fix Version/s: 0.9.0 > Remove unused class AvroConvertor in hudi-flink > --- > > Key: HUDI-2147 > URL: https://issues.apache.org/jira/browse/HUDI-2147 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Xianghu Wang >Assignee: Xianghu Wang >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (HUDI-2087) Support Append only in Flink stream
[ https://issues.apache.org/jira/browse/HUDI-2087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-2087. -- Fix Version/s: 0.9.0 Resolution: Done 371526789d663dee85041eb31c27c52c81ef87ef > Support Append only in Flink stream > --- > > Key: HUDI-2087 > URL: https://issues.apache.org/jira/browse/HUDI-2087 > Project: Apache Hudi > Issue Type: Improvement > Components: Flink Integration >Reporter: yuzhaojing >Assignee: yuzhaojing >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > Attachments: image-2021-07-08-22-04-30-039.png, > image-2021-07-08-22-04-40-018.png > > > It is necessary to support append mode in flink stream, as the data lake > should be able to write log type data as parquet high performance without > merge. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (HUDI-2136) Fix packet conflict when flink-sql-connector-hive and hudi-flink-bundle both in flink lib
[ https://issues.apache.org/jira/browse/HUDI-2136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-2136. -- Resolution: Fixed 047d956e01b6d7c92320686d8321b2bbe9d2188e > Fix packet conflict when flink-sql-connector-hive and hudi-flink-bundle both > in flink lib > - > > Key: HUDI-2136 > URL: https://issues.apache.org/jira/browse/HUDI-2136 > Project: Apache Hudi > Issue Type: Bug > Components: Flink Integration >Reporter: Zheng yunhong >Assignee: Zheng yunhong >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > > Fix packet conflict when flink-sql-connector-hive and hudi-flink-bundle both > in flink lib. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HUDI-2136) Fix packet conflict when flink-sql-connector-hive and hudi-flink-bundle both in flink lib
[ https://issues.apache.org/jira/browse/HUDI-2136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang reassigned HUDI-2136: -- Assignee: Zheng yunhong > Fix packet conflict when flink-sql-connector-hive and hudi-flink-bundle both > in flink lib > - > > Key: HUDI-2136 > URL: https://issues.apache.org/jira/browse/HUDI-2136 > Project: Apache Hudi > Issue Type: Bug > Components: Flink Integration >Reporter: Zheng yunhong >Assignee: Zheng yunhong >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > > Fix packet conflict when flink-sql-connector-hive and hudi-flink-bundle both > in flink lib. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (HUDI-2111) Update docs about bootstrap support configure KeyGenerator by type
[ https://issues.apache.org/jira/browse/HUDI-2111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-2111. -- Resolution: Done 0a6e48dd23c73a1ef34852396291cfec388bb0ca > Update docs about bootstrap support configure KeyGenerator by type > -- > > Key: HUDI-2111 > URL: https://issues.apache.org/jira/browse/HUDI-2111 > Project: Apache Hudi > Issue Type: Task >Reporter: Xianghu Wang >Assignee: Xianghu Wang >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (HUDI-2106) Fix flink batch compaction bug while user don't set compaction tasks
[ https://issues.apache.org/jira/browse/HUDI-2106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-2106. -- Resolution: Fixed bc313727e3e89640edad85364022e057c9864ee9 > Fix flink batch compaction bug while user don't set compaction tasks > > > Key: HUDI-2106 > URL: https://issues.apache.org/jira/browse/HUDI-2106 > Project: Apache Hudi > Issue Type: Bug > Components: Flink Integration >Reporter: Zheng yunhong >Assignee: Zheng yunhong >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > > There is a bug in flink batch compaction while we did not set compaction > tasks, the compaction tasks would always default value instead of > compactionPlan operations size. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HUDI-2106) Fix flink batch compaction bug while user don't set compaction tasks
[ https://issues.apache.org/jira/browse/HUDI-2106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang reassigned HUDI-2106: -- Assignee: Zheng yunhong > Fix flink batch compaction bug while user don't set compaction tasks > > > Key: HUDI-2106 > URL: https://issues.apache.org/jira/browse/HUDI-2106 > Project: Apache Hudi > Issue Type: Bug > Components: Flink Integration >Reporter: Zheng yunhong >Assignee: Zheng yunhong >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > > There is a bug in flink batch compaction while we did not set compaction > tasks, the compaction tasks would always default value instead of > compactionPlan operations size. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-2132) Make coordinator events as POJO for efficient serialization
[ https://issues.apache.org/jira/browse/HUDI-2132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17375078#comment-17375078 ] vinoyang commented on HUDI-2132: 32bd8ce088e0f1d82577575ac048e1a44d44e380 > Make coordinator events as POJO for efficient serialization > --- > > Key: HUDI-2132 > URL: https://issues.apache.org/jira/browse/HUDI-2132 > Project: Apache Hudi > Issue Type: Improvement > Components: Flink Integration >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (HUDI-1930) Bootstrap support configure KeyGenerator by type
[ https://issues.apache.org/jira/browse/HUDI-1930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-1930. -- Resolution: Implemented > Bootstrap support configure KeyGenerator by type > > > Key: HUDI-1930 > URL: https://issues.apache.org/jira/browse/HUDI-1930 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: Xianghu Wang >Assignee: Xianghu Wang >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1930) Bootstrap support configure KeyGenerator by type
[ https://issues.apache.org/jira/browse/HUDI-1930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang updated HUDI-1930: --- Fix Version/s: 0.9.0 > Bootstrap support configure KeyGenerator by type > > > Key: HUDI-1930 > URL: https://issues.apache.org/jira/browse/HUDI-1930 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: Xianghu Wang >Assignee: Xianghu Wang >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (HUDI-2112) Support reading pure logs file group for flink batch reader after compaction
[ https://issues.apache.org/jira/browse/HUDI-2112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-2112. -- Resolution: Done 7462fdefc39c75dca986d65551d871b8c47d4f55 > Support reading pure logs file group for flink batch reader after compaction > > > Key: HUDI-2112 > URL: https://issues.apache.org/jira/browse/HUDI-2112 > Project: Apache Hudi > Issue Type: Improvement > Components: Flink Integration >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (HUDI-2088) Missing Partition Fields And PreCombineField In Hoodie Properties For Table Written By Flink
[ https://issues.apache.org/jira/browse/HUDI-2088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-2088. -- Resolution: Fixed b34d53fa9c1821e7e7191ede1b99bdc96212ce3e > Missing Partition Fields And PreCombineField In Hoodie Properties For Table > Written By Flink > > > Key: HUDI-2088 > URL: https://issues.apache.org/jira/browse/HUDI-2088 > Project: Apache Hudi > Issue Type: Bug > Components: Flink Integration, Spark Integration >Reporter: pengzhiwei >Assignee: pengzhiwei >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > > Currently we have missed the partition fields and preCombineField in > hoodie.properites when init the table by flink which will cause two problems: > 1、spark partition prune will not work if missing he partition fields. > 2、spark query mor table will return incorrect result if missing the > preCombineField. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (HUDI-2052) Support load logfile in BootstrapFunction
[ https://issues.apache.org/jira/browse/HUDI-2052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-2052. -- Resolution: Implemented 07e93de8b49560eee23237817fc24fbe763f2891 > Support load logfile in BootstrapFunction > - > > Key: HUDI-2052 > URL: https://issues.apache.org/jira/browse/HUDI-2052 > Project: Apache Hudi > Issue Type: Improvement > Components: Flink Integration >Reporter: yuzhaojing >Assignee: yuzhaojing >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > > Support load logfile in BootstrapFunction -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-2052) Support load logfile in BootstrapFunction
[ https://issues.apache.org/jira/browse/HUDI-2052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang updated HUDI-2052: --- Fix Version/s: 0.9.0 > Support load logfile in BootstrapFunction > - > > Key: HUDI-2052 > URL: https://issues.apache.org/jira/browse/HUDI-2052 > Project: Apache Hudi > Issue Type: Improvement > Components: Flink Integration >Reporter: yuzhaojing >Assignee: yuzhaojing >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > > Support load logfile in BootstrapFunction -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (HUDI-2103) Add rebalance before index bootstrap
[ https://issues.apache.org/jira/browse/HUDI-2103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-2103. -- Fix Version/s: 0.9.0 Resolution: Done 1cbf43b6e7731c68661c74bdfa41399f6398172f > Add rebalance before index bootstrap > > > Key: HUDI-2103 > URL: https://issues.apache.org/jira/browse/HUDI-2103 > Project: Apache Hudi > Issue Type: Improvement > Components: Flink Integration >Reporter: yuzhaojing >Assignee: yuzhaojing >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > > When use flink sql upsert to hudi, user always set parallelism larger than > kafak partition num. Now bootstrap operator need at least one element to > trigger loading. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-2092) Fix NPE caused by FlinkStreamerConfig#writePartitionUrlEncode null value
[ https://issues.apache.org/jira/browse/HUDI-2092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang updated HUDI-2092: --- Fix Version/s: 0.9.0 > Fix NPE caused by FlinkStreamerConfig#writePartitionUrlEncode null value > > > Key: HUDI-2092 > URL: https://issues.apache.org/jira/browse/HUDI-2092 > Project: Apache Hudi > Issue Type: Bug >Reporter: Xianghu Wang >Assignee: Xianghu Wang >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (HUDI-2092) Fix NPE caused by FlinkStreamerConfig#writePartitionUrlEncode null value
[ https://issues.apache.org/jira/browse/HUDI-2092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-2092. -- Resolution: Fixed 202887b8ca27eb6de808ba7a2e737b13ae9eb8c0 > Fix NPE caused by FlinkStreamerConfig#writePartitionUrlEncode null value > > > Key: HUDI-2092 > URL: https://issues.apache.org/jira/browse/HUDI-2092 > Project: Apache Hudi > Issue Type: Bug >Reporter: Xianghu Wang >Assignee: Xianghu Wang >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (HUDI-2067) Sync all the options of FlinkOptions to FlinkStreamerConfig
[ https://issues.apache.org/jira/browse/HUDI-2067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-2067. -- Resolution: Implemented 34fc8a8880b0da5adb82b6eb7678f141af54ca18 > Sync all the options of FlinkOptions to FlinkStreamerConfig > --- > > Key: HUDI-2067 > URL: https://issues.apache.org/jira/browse/HUDI-2067 > Project: Apache Hudi > Issue Type: Improvement > Components: Flink Integration >Reporter: Danny Chen >Assignee: Vinay >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > > Sync the options so that the {{HoodieFlinkStreamer}} can have more config > options. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-2067) Sync all the options of FlinkOptions to FlinkStreamerConfig
[ https://issues.apache.org/jira/browse/HUDI-2067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang updated HUDI-2067: --- Issue Type: Improvement (was: Task) > Sync all the options of FlinkOptions to FlinkStreamerConfig > --- > > Key: HUDI-2067 > URL: https://issues.apache.org/jira/browse/HUDI-2067 > Project: Apache Hudi > Issue Type: Improvement > Components: Flink Integration >Reporter: Danny Chen >Assignee: Vinay >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > > Sync the options so that the {{HoodieFlinkStreamer}} can have more config > options. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (HUDI-2074) Use while loop instead of recursive call in MergeOnReadInputFormat#MergeIterator to avoid StackOverflow
[ https://issues.apache.org/jira/browse/HUDI-2074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-2074. -- Resolution: Done d24341d10ca49ed52fc0c1c86a164fbfb57327d4 > Use while loop instead of recursive call in > MergeOnReadInputFormat#MergeIterator to avoid StackOverflow > --- > > Key: HUDI-2074 > URL: https://issues.apache.org/jira/browse/HUDI-2074 > Project: Apache Hudi > Issue Type: Improvement > Components: Flink Integration >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (HUDI-2068) Skip the assign state for SmallFileAssign when the state can not assign initially
[ https://issues.apache.org/jira/browse/HUDI-2068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-2068. -- Resolution: Fixed e64fe5505487f3ee591b2b5d044c2c57989f8991 > Skip the assign state for SmallFileAssign when the state can not assign > initially > - > > Key: HUDI-2068 > URL: https://issues.apache.org/jira/browse/HUDI-2068 > Project: Apache Hudi > Issue Type: Bug > Components: Flink Integration >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (HUDI-2062) Catch FileNotFoundException in WriteProfiles #getCommitMetadataSafely
[ https://issues.apache.org/jira/browse/HUDI-2062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-2062. -- Fix Version/s: 0.9.0 Resolution: Fixed 218f2a6df8a41279d904a76b594d22f4d66d17c2 > Catch FileNotFoundException in WriteProfiles #getCommitMetadataSafely > - > > Key: HUDI-2062 > URL: https://issues.apache.org/jira/browse/HUDI-2062 > Project: Apache Hudi > Issue Type: Bug > Components: Flink Integration >Reporter: yuzhaojing >Assignee: yuzhaojing >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > > The function WriteProfiles #getCommitMetadataSafely expect get instant > safely, if instant deleted by cleaner that ignore it. > But in case, FileNotFoundException will throw out of try catch. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-2062) Catch IOException in WriteProfiles #getCommitMetadataSafely
[ https://issues.apache.org/jira/browse/HUDI-2062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17367982#comment-17367982 ] vinoyang commented on HUDI-2062: Hi [~yuzhaojing] What's the exception and stack trace did you meet? Can you share it in the "Description" section? > Catch IOException in WriteProfiles #getCommitMetadataSafely > --- > > Key: HUDI-2062 > URL: https://issues.apache.org/jira/browse/HUDI-2062 > Project: Apache Hudi > Issue Type: Bug > Components: Flink Integration >Reporter: yuzhaojing >Assignee: yuzhaojing >Priority: Major > Labels: pull-request-available > > Catch IOException in WriteProfiles #getCommitMetadataSafely -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (HUDI-1826) Add ORC support in HoodieSnapshotExporter
[ https://issues.apache.org/jira/browse/HUDI-1826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-1826. -- Fix Version/s: 0.9.0 Resolution: Fixed 43b9c1fa1caf97f6fb2baf68e350615541ea0a0c > Add ORC support in HoodieSnapshotExporter > - > > Key: HUDI-1826 > URL: https://issues.apache.org/jira/browse/HUDI-1826 > Project: Apache Hudi > Issue Type: Sub-task > Components: Storage Management >Reporter: Teresa Kang >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)