[jira] [Commented] (HUDI-1566) Typo in account request caused wrong name in Apache id
[ https://issues.apache.org/jira/browse/HUDI-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17276787#comment-17276787 ] wangxianghu commented on HUDI-1566: --- [~clr] Yes ,I found it later can you help ping somebody to handle it, I tried, but seems have no permission. I need this id to active my permission to merge code into hudi project thanks :) > Typo in account request caused wrong name in Apache id > -- > > Key: HUDI-1566 > URL: https://issues.apache.org/jira/browse/HUDI-1566 > Project: Apache Hudi > Issue Type: Bug >Reporter: Craig L Russell >Priority: Major > > Due to a typo in the request, the Apache id wangxinghu was created. > The id should be wangxianghu. > Thanks for fixing this. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-1566) Typo in account request caused wrong name in Apache id
[ https://issues.apache.org/jira/browse/HUDI-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17276777#comment-17276777 ] wangxianghu commented on HUDI-1566: --- Hi [~clr]，is there any progress on correcting the id ? > Typo in account request caused wrong name in Apache id > -- > > Key: HUDI-1566 > URL: https://issues.apache.org/jira/browse/HUDI-1566 > Project: Apache Hudi > Issue Type: Bug >Reporter: Craig L Russell >Priority: Major > > Due to a typo in the request, the Apache id wangxinghu was created. > The id should be wangxianghu. > Thanks for fixing this. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-1566) Typo in account request caused wrong name in Apache id
[ https://issues.apache.org/jira/browse/HUDI-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17275540#comment-17275540 ] wangxianghu commented on HUDI-1566: --- [~vinoth] can you help me correct the id > Typo in account request caused wrong name in Apache id > -- > > Key: HUDI-1566 > URL: https://issues.apache.org/jira/browse/HUDI-1566 > Project: Apache Hudi > Issue Type: Bug >Reporter: Craig L Russell >Priority: Major > > Due to a typo in the request, the Apache id wangxinghu was created. > The id should be wangxianghu. > Thanks for fixing this. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HUDI-1556) Add App Id and App name to HoodieDeltaStreamerMetrics
[ https://issues.apache.org/jira/browse/HUDI-1556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu reassigned HUDI-1556: - Assignee: liujinhui > Add App Id and App name to HoodieDeltaStreamerMetrics > - > > Key: HUDI-1556 > URL: https://issues.apache.org/jira/browse/HUDI-1556 > Project: Apache Hudi > Issue Type: Task >Reporter: wangxianghu >Assignee: liujinhui >Priority: Major > Fix For: 0.8.0 > > > we need something unique to relate Metric data to spark job -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-1556) Add App Id and App name to HoodieDeltaStreamerMetrics
wangxianghu created HUDI-1556: - Summary: Add App Id and App name to HoodieDeltaStreamerMetrics Key: HUDI-1556 URL: https://issues.apache.org/jira/browse/HUDI-1556 Project: Apache Hudi Issue Type: Task Reporter: wangxianghu Fix For: 0.8.0 we need something unique to relate Metric data to spark job -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-1547) CI intermittent failure: TestJsonStringToHoodieRecordMapFunction.testMapFunction
[ https://issues.apache.org/jira/browse/HUDI-1547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17271766#comment-17271766 ] wangxianghu commented on HUDI-1547: --- [~vinoth] I can take it > CI intermittent failure: > TestJsonStringToHoodieRecordMapFunction.testMapFunction > - > > Key: HUDI-1547 > URL: https://issues.apache.org/jira/browse/HUDI-1547 > Project: Apache Hudi > Issue Type: Bug > Components: Release Administrative >Affects Versions: 0.8.0 >Reporter: sivabalan narayanan >Assignee: wangxianghu >Priority: Major > Labels: user-support-issues > > [https://github.com/apache/hudi/issues/2467] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HUDI-1547) CI intermittent failure: TestJsonStringToHoodieRecordMapFunction.testMapFunction
[ https://issues.apache.org/jira/browse/HUDI-1547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu reassigned HUDI-1547: - Assignee: wangxianghu > CI intermittent failure: > TestJsonStringToHoodieRecordMapFunction.testMapFunction > - > > Key: HUDI-1547 > URL: https://issues.apache.org/jira/browse/HUDI-1547 > Project: Apache Hudi > Issue Type: Bug > Components: Release Administrative >Affects Versions: 0.8.0 >Reporter: sivabalan narayanan >Assignee: wangxianghu >Priority: Major > Labels: user-support-issues > > [https://github.com/apache/hudi/issues/2467] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1544) Add unit test against HoodieFlinkStreamer
[ https://issues.apache.org/jira/browse/HUDI-1544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu updated HUDI-1544: -- Summary: Add unit test against HoodieFlinkStreamer (was: Add HoodieFlinkStreamer unit test) > Add unit test against HoodieFlinkStreamer > - > > Key: HUDI-1544 > URL: https://issues.apache.org/jira/browse/HUDI-1544 > Project: Apache Hudi > Issue Type: Test >Reporter: wangxianghu >Priority: Major > Fix For: 0.8.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HUDI-1544) Add unit test against HoodieFlinkStreamer
[ https://issues.apache.org/jira/browse/HUDI-1544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu reassigned HUDI-1544: - Assignee: wangxianghu > Add unit test against HoodieFlinkStreamer > - > > Key: HUDI-1544 > URL: https://issues.apache.org/jira/browse/HUDI-1544 > Project: Apache Hudi > Issue Type: Test >Reporter: wangxianghu >Assignee: wangxianghu >Priority: Major > Fix For: 0.8.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-1544) Add HoodieFlinkStreamer unit test
wangxianghu created HUDI-1544: - Summary: Add HoodieFlinkStreamer unit test Key: HUDI-1544 URL: https://issues.apache.org/jira/browse/HUDI-1544 Project: Apache Hudi Issue Type: Test Reporter: wangxianghu Fix For: 0.8.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1543) Fix NPE using HoodieFlinkStreamer to etl data from kafka to hudi
[ https://issues.apache.org/jira/browse/HUDI-1543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu updated HUDI-1543: -- Summary: Fix NPE using HoodieFlinkStreamer to etl data from kafka to hudi (was: Fix HoodieFlinkStreamer NPE after HUDI-1511 merged) > Fix NPE using HoodieFlinkStreamer to etl data from kafka to hudi > > > Key: HUDI-1543 > URL: https://issues.apache.org/jira/browse/HUDI-1543 > Project: Apache Hudi > Issue Type: Bug >Reporter: wangxianghu >Assignee: wangxianghu >Priority: Major > > ``` > java.lang.NullPointerException: Keyed state can only be used on a 'keyed > stream', i.e., after a 'keyBy()' operation. > at org.apache.flink.util.Preconditions.checkNotNull(Preconditions.java:75) > at > org.apache.flink.streaming.api.operators.StreamingRuntimeContext.checkPreconditionsAndGetKeyedStateStore(StreamingRuntimeContext.java:223) > at > org.apache.flink.streaming.api.operators.StreamingRuntimeContext.getMapState(StreamingRuntimeContext.java:216) > at > org.apache.hudi.index.state.FlinkInMemoryStateIndex.(FlinkInMemoryStateIndex.java:58) > at > org.apache.hudi.index.FlinkHoodieIndex.createIndex(FlinkHoodieIndex.java:61) > at > org.apache.hudi.client.HoodieFlinkWriteClient.createIndex(HoodieFlinkWriteClient.java:75) > at > org.apache.hudi.client.AbstractHoodieWriteClient.(AbstractHoodieWriteClient.java:136) > at > org.apache.hudi.client.AbstractHoodieWriteClient.(AbstractHoodieWriteClient.java:120) > at > org.apache.hudi.client.HoodieFlinkWriteClient.(HoodieFlinkWriteClient.java:62) > at > org.apache.hudi.operator.InstantGenerateOperator.open(InstantGenerateOperator.java:115) > at > org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:291) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:479) > at > org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:47) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:475) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:528) > at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546) > at java.lang.Thread.run(Thread.java:748) > ``` -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-1543) Fix HoodieFlinkStreamer NPE after HUDI-1511 merged
wangxianghu created HUDI-1543: - Summary: Fix HoodieFlinkStreamer NPE after HUDI-1511 merged Key: HUDI-1543 URL: https://issues.apache.org/jira/browse/HUDI-1543 Project: Apache Hudi Issue Type: Bug Reporter: wangxianghu Assignee: wangxianghu ``` java.lang.NullPointerException: Keyed state can only be used on a 'keyed stream', i.e., after a 'keyBy()' operation. at org.apache.flink.util.Preconditions.checkNotNull(Preconditions.java:75) at org.apache.flink.streaming.api.operators.StreamingRuntimeContext.checkPreconditionsAndGetKeyedStateStore(StreamingRuntimeContext.java:223) at org.apache.flink.streaming.api.operators.StreamingRuntimeContext.getMapState(StreamingRuntimeContext.java:216) at org.apache.hudi.index.state.FlinkInMemoryStateIndex.(FlinkInMemoryStateIndex.java:58) at org.apache.hudi.index.FlinkHoodieIndex.createIndex(FlinkHoodieIndex.java:61) at org.apache.hudi.client.HoodieFlinkWriteClient.createIndex(HoodieFlinkWriteClient.java:75) at org.apache.hudi.client.AbstractHoodieWriteClient.(AbstractHoodieWriteClient.java:136) at org.apache.hudi.client.AbstractHoodieWriteClient.(AbstractHoodieWriteClient.java:120) at org.apache.hudi.client.HoodieFlinkWriteClient.(HoodieFlinkWriteClient.java:62) at org.apache.hudi.operator.InstantGenerateOperator.open(InstantGenerateOperator.java:115) at org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:291) at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:479) at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:47) at org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:475) at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:528) at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546) at java.lang.Thread.run(Thread.java:748) ``` -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HUDI-1207) Add kafka implementation of write commit callback to Spark datasources
[ https://issues.apache.org/jira/browse/HUDI-1207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu reassigned HUDI-1207: - Assignee: Trevorzhang (was: wangxianghu) > Add kafka implementation of write commit callback to Spark datasources > -- > > Key: HUDI-1207 > URL: https://issues.apache.org/jira/browse/HUDI-1207 > Project: Apache Hudi > Issue Type: Task >Reporter: wangxianghu >Assignee: Trevorzhang >Priority: Major > Fix For: 0.7.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (HUDI-1466) Migrate CI/CD from travis to Azure pipeline
[ https://issues.apache.org/jira/browse/HUDI-1466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu closed HUDI-1466. - Resolution: Duplicate > Migrate CI/CD from travis to Azure pipeline > > > Key: HUDI-1466 > URL: https://issues.apache.org/jira/browse/HUDI-1466 > Project: Apache Hudi > Issue Type: New Feature >Reporter: wangxianghu >Assignee: wangxianghu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1466) Migrate CI/CD from travis to Azure pipeline
[ https://issues.apache.org/jira/browse/HUDI-1466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu updated HUDI-1466: -- Status: Open (was: New) > Migrate CI/CD from travis to Azure pipeline > > > Key: HUDI-1466 > URL: https://issues.apache.org/jira/browse/HUDI-1466 > Project: Apache Hudi > Issue Type: New Feature >Reporter: wangxianghu >Assignee: wangxianghu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-1531) Introduce HoodiePartitionCleaner to delete specific partition
wangxianghu created HUDI-1531: - Summary: Introduce HoodiePartitionCleaner to delete specific partition Key: HUDI-1531 URL: https://issues.apache.org/jira/browse/HUDI-1531 Project: Apache Hudi Issue Type: New Feature Reporter: wangxianghu Assignee: wangxianghu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1514) Avoid raw type use for parameter of interface
[
https://issues.apache.org/jira/browse/HUDI-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
wangxianghu updated HUDI-1514:
--
Fix Version/s: (was: 0.6.1)
0.7.0
> Avoid raw type use for parameter of interface
> ---
>
> Key: HUDI-1514
> URL: https://issues.apache.org/jira/browse/HUDI-1514
> Project: Apache Hudi
> Issue Type: Improvement
> Components: Utilities
>Reporter: puyvqi
>Priority: Major
> Fix For: 0.7.0
>
> Attachments: image-2021-01-07-17-23-46-202.png
>
>
> org.apache.hudi.utilities.transform.Transformer method api:
> Dataset {color:#ffc66d}apply{color}(JavaSparkContext jsc{color:#cc7832},
> {color}SparkSession sparkSession{color:#cc7832}, {color}Dataset
> rowDataset{color:#cc7832}, {color}TypedProperties
> properties){color:#cc7832};{color}
> {color:#172b4d}it's a bad idea to use raw type parameter(without a type
> parameter or wildcard) in interface ,especially Dataset[T] is define in
> scala,when implementing this interface in scala. it turn out to be an
> compilation error:
> {color}
> {code:java}
> //overriding method apply in trait Transformer of type (x$1:
> org.apache.spark.api.java.JavaSparkContext, x$2:
> org.apache.spark.sql.SparkSession, x$3:
> org.apache.spark.sql.Dataset[org.apache.spark.sql.Row], x$4:
> org.apache.hudi.common.config.TypedProperties)org.apache.spark.sql.Dataset;
> method apply has incompatible type
> override def apply(jsc: JavaSparkContext, sparkSession: SparkSession,
> rowDataset: Dataset[Row], properties: TypedProperties): Dataset[_] =
> rowDataset
> {code}
> {code:java}
> //method apply has incompatible type
> override def apply(jsc: JavaSparkContext, sparkSession: SparkSession,
> rowDataset: Dataset[Row], properties: TypedProperties): Dataset[Row] =
> rowDataset
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HUDI-1511) InstantGenerateOperator support multiple parallelism
[ https://issues.apache.org/jira/browse/HUDI-1511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu reassigned HUDI-1511: - Assignee: loukey_j > InstantGenerateOperator support multiple parallelism > > > Key: HUDI-1511 > URL: https://issues.apache.org/jira/browse/HUDI-1511 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: wangxianghu >Assignee: loukey_j >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (HUDI-1511) InstantGenerateOperator support multiple parallelism
[ https://issues.apache.org/jira/browse/HUDI-1511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu reopened HUDI-1511: --- > InstantGenerateOperator support multiple parallelism > > > Key: HUDI-1511 > URL: https://issues.apache.org/jira/browse/HUDI-1511 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: wangxianghu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (HUDI-1511) InstantGenerateOperator support multiple parallelism
[ https://issues.apache.org/jira/browse/HUDI-1511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu closed HUDI-1511. - Resolution: Duplicate > InstantGenerateOperator support multiple parallelism > > > Key: HUDI-1511 > URL: https://issues.apache.org/jira/browse/HUDI-1511 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: wangxianghu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-1511) InstantGenerateOperator support multiple parallelism
wangxianghu created HUDI-1511: - Summary: InstantGenerateOperator support multiple parallelism Key: HUDI-1511 URL: https://issues.apache.org/jira/browse/HUDI-1511 Project: Apache Hudi Issue Type: Sub-task Reporter: wangxianghu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1506) Fix wrong exception thrown in HoodieAvroUtils
[
https://issues.apache.org/jira/browse/HUDI-1506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
wangxianghu updated HUDI-1506:
--
Description:
{code:java}
//
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure:
Task 4 in stage 4.0 failed 4 times, most recent failure: Lost task 4.3 in stage
4.0 (TID 24, al-prd-dtp-data-lake-10-0-88-26, executor 4):
org.apache.hudi.exception.HoodieException: etlDatetime(Part -etlDatetime) field
not found in record. Acceptable fields were :[vin, uuid, commercialType,
businessType, vehicleNo, plateColor, vehicleColor, engineId, nextFixDate,
feePrintId, transArea, createTime, updateTime, registerDate, curVehicleNo,
reportVehicleNo, model, checkDate, certifyDateA, certifyDateB, certificate,
transAgency, transAgencyNet, transDateStart, transDateStop, insurCom, insurNum,
insurType, insurCount, insurEff, insurExp, insurCreateTime, insurUpdateTime,
curVehicleCertno, reportVehicleCertno, seats, brand, vehicleType, fuelType,
engineDisplace, photo, enginePower, gpsBrand, gpsModel, gpsImei,
gpsInstallDate, curDriverUuid, reportDrivers, curTimeOn, curTimeOff, timeFrom,
timeTo, ownerName, fixState, checkState, photoId, photoIdUrl, fareType,
wheelBase, vehicleTec, vehicleSafe, lesseeName, lesseeCode, sdcOperationType,
hivePartition, etlDatetime]
{code}
we can see the `etlDatetime` do exist. it is caused by null value acturally
was:Caused by: org.apache.spark.SparkException: Job aborted due to stage
failure: Task 4 in stage 4.0 failed 4 times, most recent failure: Lost task 4.3
in stage 4.0 (TID 24, al-prd-dtp-data-lake-10-0-88-26, executor 4):
org.apache.hudi.exception.HoodieException: etlDatetime(Part -etlDatetime) field
not found in record. Acceptable fields were :[vin, uuid, commercialType,
businessType, vehicleNo, plateColor, vehicleColor, engineId, nextFixDate,
feePrintId, transArea, createTime, updateTime, registerDate, curVehicleNo,
reportVehicleNo, model, checkDate, certifyDateA, certifyDateB, certificate,
transAgency, transAgencyNet, transDateStart, transDateStop, insurCom, insurNum,
insurType, insurCount, insurEff, insurExp, insurCreateTime, insurUpdateTime,
curVehicleCertno, reportVehicleCertno, seats, brand, vehicleType, fuelType,
engineDisplace, photo, enginePower, gpsBrand, gpsModel, gpsImei,
gpsInstallDate, curDriverUuid, reportDrivers, curTimeOn, curTimeOff, timeFrom,
timeTo, ownerName, fixState, checkState, photoId, photoIdUrl, fareType,
wheelBase, vehicleTec, vehicleSafe, lesseeName, lesseeCode, sdcOperationType,
hivePartition, etlDatetime]
> Fix wrong exception thrown in HoodieAvroUtils
> -
>
> Key: HUDI-1506
> URL: https://issues.apache.org/jira/browse/HUDI-1506
> Project: Apache Hudi
> Issue Type: Bug
>Reporter: wangxianghu
>Assignee: wangxianghu
>Priority: Major
>
> {code:java}
> //
> Caused by: org.apache.spark.SparkException: Job aborted due to stage failure:
> Task 4 in stage 4.0 failed 4 times, most recent failure: Lost task 4.3 in
> stage 4.0 (TID 24, al-prd-dtp-data-lake-10-0-88-26, executor 4):
> org.apache.hudi.exception.HoodieException: etlDatetime(Part -etlDatetime)
> field not found in record. Acceptable fields were :[vin, uuid,
> commercialType, businessType, vehicleNo, plateColor, vehicleColor, engineId,
> nextFixDate, feePrintId, transArea, createTime, updateTime, registerDate,
> curVehicleNo, reportVehicleNo, model, checkDate, certifyDateA, certifyDateB,
> certificate, transAgency, transAgencyNet, transDateStart, transDateStop,
> insurCom, insurNum, insurType, insurCount, insurEff, insurExp,
> insurCreateTime, insurUpdateTime, curVehicleCertno, reportVehicleCertno,
> seats, brand, vehicleType, fuelType, engineDisplace, photo, enginePower,
> gpsBrand, gpsModel, gpsImei, gpsInstallDate, curDriverUuid, reportDrivers,
> curTimeOn, curTimeOff, timeFrom, timeTo, ownerName, fixState, checkState,
> photoId, photoIdUrl, fareType, wheelBase, vehicleTec, vehicleSafe,
> lesseeName, lesseeCode, sdcOperationType, hivePartition, etlDatetime]
> {code}
> we can see the `etlDatetime` do exist. it is caused by null value acturally
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HUDI-1506) Fix wrong exception thrown in HoodieAvroUtils
wangxianghu created HUDI-1506: - Summary: Fix wrong exception thrown in HoodieAvroUtils Key: HUDI-1506 URL: https://issues.apache.org/jira/browse/HUDI-1506 Project: Apache Hudi Issue Type: Bug Reporter: wangxianghu Assignee: wangxianghu Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 4 in stage 4.0 failed 4 times, most recent failure: Lost task 4.3 in stage 4.0 (TID 24, al-prd-dtp-data-lake-10-0-88-26, executor 4): org.apache.hudi.exception.HoodieException: etlDatetime(Part -etlDatetime) field not found in record. Acceptable fields were :[vin, uuid, commercialType, businessType, vehicleNo, plateColor, vehicleColor, engineId, nextFixDate, feePrintId, transArea, createTime, updateTime, registerDate, curVehicleNo, reportVehicleNo, model, checkDate, certifyDateA, certifyDateB, certificate, transAgency, transAgencyNet, transDateStart, transDateStop, insurCom, insurNum, insurType, insurCount, insurEff, insurExp, insurCreateTime, insurUpdateTime, curVehicleCertno, reportVehicleCertno, seats, brand, vehicleType, fuelType, engineDisplace, photo, enginePower, gpsBrand, gpsModel, gpsImei, gpsInstallDate, curDriverUuid, reportDrivers, curTimeOn, curTimeOff, timeFrom, timeTo, ownerName, fixState, checkState, photoId, photoIdUrl, fareType, wheelBase, vehicleTec, vehicleSafe, lesseeName, lesseeCode, sdcOperationType, hivePartition, etlDatetime] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-981) Use rocksDB as flink state backend
[ https://issues.apache.org/jira/browse/HUDI-981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu updated HUDI-981: - Status: Open (was: New) > Use rocksDB as flink state backend > -- > > Key: HUDI-981 > URL: https://issues.apache.org/jira/browse/HUDI-981 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: wangxianghu >Assignee: chijunqing >Priority: Major > > Use rocksDB as flink state backend -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1485) Fix Deletes issued without any prior commits exception
[
https://issues.apache.org/jira/browse/HUDI-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
wangxianghu updated HUDI-1485:
--
Description:
step to reproduce:
just use the delete demo in the quickstart
```
// spark-shell
// fetch total records count
spark.sql("select uuid, partitionpath from hudi_trips_snapshot").count()
// fetch two records to be deleted
val ds = spark.sql("select uuid, partitionpath from
hudi_trips_snapshot").limit(2)
// issue deletes
val deletes = dataGen.generateDeletes(ds.collectAsList())
val df = spark.read.json(spark.sparkContext.parallelize(deletes, 2))
df.write.format("hudi").
options(getQuickstartWriteConfigs).
option(OPERATION_OPT_KEY,"delete").
option(PRECOMBINE_FIELD_OPT_KEY, "ts").
option(RECORDKEY_FIELD_OPT_KEY, "uuid").
option(PARTITIONPATH_FIELD_OPT_KEY, "partitionpath").
option(TABLE_NAME, tableName).
mode(Append).
save(basePath)
```
was:
use the delete demo in quickstart
```
// spark-shell
// fetch total records count
spark.sql("select uuid, partitionpath from hudi_trips_snapshot").count()
// fetch two records to be deleted
val ds = spark.sql("select uuid, partitionpath from
hudi_trips_snapshot").limit(2)
// issue deletes
val deletes = dataGen.generateDeletes(ds.collectAsList())
val df = spark.read.json(spark.sparkContext.parallelize(deletes, 2))
df.write.format("hudi").
options(getQuickstartWriteConfigs).
option(OPERATION_OPT_KEY,"delete").
option(PRECOMBINE_FIELD_OPT_KEY, "ts").
option(RECORDKEY_FIELD_OPT_KEY, "uuid").
option(PARTITIONPATH_FIELD_OPT_KEY, "partitionpath").
option(TABLE_NAME, tableName).
mode(Append).
save(basePath)
```
> Fix Deletes issued without any prior commits exception
> --
>
> Key: HUDI-1485
> URL: https://issues.apache.org/jira/browse/HUDI-1485
> Project: Apache Hudi
> Issue Type: Bug
>Reporter: wangxianghu
>Assignee: wangxianghu
>Priority: Major
>
> step to reproduce:
> just use the delete demo in the quickstart
> ```
> // spark-shell
> // fetch total records count
> spark.sql("select uuid, partitionpath from hudi_trips_snapshot").count()
> // fetch two records to be deleted
> val ds = spark.sql("select uuid, partitionpath from
> hudi_trips_snapshot").limit(2)
> // issue deletes
> val deletes = dataGen.generateDeletes(ds.collectAsList())
> val df = spark.read.json(spark.sparkContext.parallelize(deletes, 2))
> df.write.format("hudi").
> options(getQuickstartWriteConfigs).
> option(OPERATION_OPT_KEY,"delete").
> option(PRECOMBINE_FIELD_OPT_KEY, "ts").
> option(RECORDKEY_FIELD_OPT_KEY, "uuid").
> option(PARTITIONPATH_FIELD_OPT_KEY, "partitionpath").
> option(TABLE_NAME, tableName).
> mode(Append).
> save(basePath)
> ```
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HUDI-1485) Fix Deletes issued without any prior commits exception
wangxianghu created HUDI-1485:
-
Summary: Fix Deletes issued without any prior commits exception
Key: HUDI-1485
URL: https://issues.apache.org/jira/browse/HUDI-1485
Project: Apache Hudi
Issue Type: Bug
Reporter: wangxianghu
Assignee: wangxianghu
use the delete demo in quickstart
```
// spark-shell
// fetch total records count
spark.sql("select uuid, partitionpath from hudi_trips_snapshot").count()
// fetch two records to be deleted
val ds = spark.sql("select uuid, partitionpath from
hudi_trips_snapshot").limit(2)
// issue deletes
val deletes = dataGen.generateDeletes(ds.collectAsList())
val df = spark.read.json(spark.sparkContext.parallelize(deletes, 2))
df.write.format("hudi").
options(getQuickstartWriteConfigs).
option(OPERATION_OPT_KEY,"delete").
option(PRECOMBINE_FIELD_OPT_KEY, "ts").
option(RECORDKEY_FIELD_OPT_KEY, "uuid").
option(PARTITIONPATH_FIELD_OPT_KEY, "partitionpath").
option(TABLE_NAME, tableName).
mode(Append).
save(basePath)
```
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HUDI-1471) Make QuickStartUtils generate deletes according to specific ts
wangxianghu created HUDI-1471: - Summary: Make QuickStartUtils generate deletes according to specific ts Key: HUDI-1471 URL: https://issues.apache.org/jira/browse/HUDI-1471 Project: Apache Hudi Issue Type: Task Reporter: wangxianghu Assignee: wangxianghu Since `QuickStartUtils.generateRandomValue` method do not set `ts` to `0.0` any more, method `QuickStartUtils.generateInserts` should generate records according to specofic `ts ` instead of `0.0` -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-1467) Promote Powered by chapter to top level menu
wangxianghu created HUDI-1467: - Summary: Promote Powered by chapter to top level menu Key: HUDI-1467 URL: https://issues.apache.org/jira/browse/HUDI-1467 Project: Apache Hudi Issue Type: Improvement Reporter: wangxianghu Assignee: wangxianghu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-1466) Migrate CI/CD from travis to Azure pipeline
wangxianghu created HUDI-1466: - Summary: Migrate CI/CD from travis to Azure pipeline Key: HUDI-1466 URL: https://issues.apache.org/jira/browse/HUDI-1466 Project: Apache Hudi Issue Type: New Feature Reporter: wangxianghu Assignee: wangxianghu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HUDI-1439) Remove scala dependency from hudi-client-common
[ https://issues.apache.org/jira/browse/HUDI-1439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu resolved HUDI-1439. --- Resolution: Fixed Fixed via master branch : 236d1b0dec409efb9c7a9febd260b1060da6a9c1 > Remove scala dependency from hudi-client-common > --- > > Key: HUDI-1439 > URL: https://issues.apache.org/jira/browse/HUDI-1439 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: wangxianghu >Assignee: shenh062326 >Priority: Major > Labels: pull-request-available > Fix For: 0.7.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HUDI-1447) DeltaStreamer kafka source supports consuming from specified timestamp
[ https://issues.apache.org/jira/browse/HUDI-1447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu reassigned HUDI-1447: - Assignee: liujinhui (was: wangxianghu) > DeltaStreamer kafka source supports consuming from specified timestamp > -- > > Key: HUDI-1447 > URL: https://issues.apache.org/jira/browse/HUDI-1447 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer >Reporter: wangxianghu >Assignee: liujinhui >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-1447) DeltaStreamer kafka source supports consuming from specified timestamp
wangxianghu created HUDI-1447: - Summary: DeltaStreamer kafka source supports consuming from specified timestamp Key: HUDI-1447 URL: https://issues.apache.org/jira/browse/HUDI-1447 Project: Apache Hudi Issue Type: New Feature Components: DeltaStreamer Reporter: wangxianghu Assignee: wangxianghu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HUDI-1438) Move DataSourceOptions to hudi-client-common to reuse more code
[ https://issues.apache.org/jira/browse/HUDI-1438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu reassigned HUDI-1438: - Assignee: wangxianghu > Move DataSourceOptions to hudi-client-common to reuse more code > --- > > Key: HUDI-1438 > URL: https://issues.apache.org/jira/browse/HUDI-1438 > Project: Apache Hudi > Issue Type: Improvement >Reporter: wangxianghu >Assignee: wangxianghu >Priority: Major > Fix For: 0.7.0 > > > DataSourceOptions holds lots of options about read and write configs, it > should be moved to `hudi-client-common` module for code reuse purpose -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HUDI-1438) Move DataSourceOptions to hudi-client-common to reuse more code
[ https://issues.apache.org/jira/browse/HUDI-1438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu reassigned HUDI-1438: - Assignee: liujinhui (was: wangxianghu) > Move DataSourceOptions to hudi-client-common to reuse more code > --- > > Key: HUDI-1438 > URL: https://issues.apache.org/jira/browse/HUDI-1438 > Project: Apache Hudi > Issue Type: Improvement >Reporter: wangxianghu >Assignee: liujinhui >Priority: Major > Fix For: 0.7.0 > > > DataSourceOptions holds lots of options about read and write configs, it > should be moved to `hudi-client-common` module for code reuse purpose -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1439) Remove scala dependency from hudi-client-common
[ https://issues.apache.org/jira/browse/HUDI-1439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu updated HUDI-1439: -- Summary: Remove scala dependency from hudi-client-common (was: Remove scala denpendency from hudi-client-common) > Remove scala dependency from hudi-client-common > --- > > Key: HUDI-1439 > URL: https://issues.apache.org/jira/browse/HUDI-1439 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: wangxianghu >Assignee: shenh062326 >Priority: Major > Fix For: 0.7.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HUDI-1439) Remove scala denpendency from hudi-client-common
[ https://issues.apache.org/jira/browse/HUDI-1439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu reassigned HUDI-1439: - Assignee: shenh062326 > Remove scala denpendency from hudi-client-common > > > Key: HUDI-1439 > URL: https://issues.apache.org/jira/browse/HUDI-1439 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: wangxianghu >Assignee: shenh062326 >Priority: Major > Fix For: 0.7.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-1439) Remove scala denpendency from hudi-client-common
wangxianghu created HUDI-1439: - Summary: Remove scala denpendency from hudi-client-common Key: HUDI-1439 URL: https://issues.apache.org/jira/browse/HUDI-1439 Project: Apache Hudi Issue Type: Sub-task Reporter: wangxianghu Fix For: 0.7.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1438) Move DataSourceOptions to hudi-client-common to reuse more code
[ https://issues.apache.org/jira/browse/HUDI-1438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu updated HUDI-1438: -- Summary: Move DataSourceOptions to hudi-client-common to reuse more code (was: Move DataSourceOptions to hudi-client-common to reuse code) > Move DataSourceOptions to hudi-client-common to reuse more code > --- > > Key: HUDI-1438 > URL: https://issues.apache.org/jira/browse/HUDI-1438 > Project: Apache Hudi > Issue Type: Improvement >Reporter: wangxianghu >Priority: Major > Fix For: 0.7.0 > > > DataSourceOptions holds lots of options about read and write configs, it > should be moved to `hudi-client-common` module for code reuse purpose -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1438) Move DataSourceOptions to hudi-client-common to reuse code
[ https://issues.apache.org/jira/browse/HUDI-1438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu updated HUDI-1438: -- Issue Type: Improvement (was: Bug) > Move DataSourceOptions to hudi-client-common to reuse code > -- > > Key: HUDI-1438 > URL: https://issues.apache.org/jira/browse/HUDI-1438 > Project: Apache Hudi > Issue Type: Improvement >Reporter: wangxianghu >Priority: Major > Fix For: 0.7.0 > > > DataSourceOptions holds lots of options about read and write configs, it > should be moved to `hudi-client-common` module for code reuse purpose -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-1438) Move DataSourceOptions to hudi-client-common to reuse code
wangxianghu created HUDI-1438: - Summary: Move DataSourceOptions to hudi-client-common to reuse code Key: HUDI-1438 URL: https://issues.apache.org/jira/browse/HUDI-1438 Project: Apache Hudi Issue Type: Bug Reporter: wangxianghu Fix For: 0.7.0 DataSourceOptions holds lots of options about read and write configs, it should be moved to `hudi-client-common` module for code reuse purpose -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1402) Add example about how to use flink engine to quickstart
[ https://issues.apache.org/jira/browse/HUDI-1402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu updated HUDI-1402: -- Summary: Add example about how to use flink engine to quickstart (was: Add a blog about how to use flink engine) > Add example about how to use flink engine to quickstart > > > Key: HUDI-1402 > URL: https://issues.apache.org/jira/browse/HUDI-1402 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: wangxianghu >Assignee: wangxianghu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-1433) Flink engine support sync data to hive table
wangxianghu created HUDI-1433: - Summary: Flink engine support sync data to hive table Key: HUDI-1433 URL: https://issues.apache.org/jira/browse/HUDI-1433 Project: Apache Hudi Issue Type: Sub-task Reporter: wangxianghu Assignee: wangxianghu Fix For: 0.7.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-1421) Improvement of failure recovery for HoodieFlinkStreamer
wangxianghu created HUDI-1421: - Summary: Improvement of failure recovery for HoodieFlinkStreamer Key: HUDI-1421 URL: https://issues.apache.org/jira/browse/HUDI-1421 Project: Apache Hudi Issue Type: Sub-task Reporter: wangxianghu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1335) Introduce FlinkHoodieSimpleIndex to hudi-flink-client
[ https://issues.apache.org/jira/browse/HUDI-1335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu updated HUDI-1335: -- Status: Open (was: New) > Introduce FlinkHoodieSimpleIndex to hudi-flink-client > - > > Key: HUDI-1335 > URL: https://issues.apache.org/jira/browse/HUDI-1335 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: wangxianghu >Assignee: wangxianghu >Priority: Major > Labels: pull-request-available > Fix For: 0.7.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1336) Introduce FlinkHoodieGlobalSimpleIndex to hudi-flink-client
[ https://issues.apache.org/jira/browse/HUDI-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu updated HUDI-1336: -- Status: Open (was: New) > Introduce FlinkHoodieGlobalSimpleIndex to hudi-flink-client > --- > > Key: HUDI-1336 > URL: https://issues.apache.org/jira/browse/HUDI-1336 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: wangxianghu >Assignee: wangxianghu >Priority: Major > Fix For: 0.7.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1333) Introduce FlinkHoodieGlobalBloomIndex to hudi-flink-client
[ https://issues.apache.org/jira/browse/HUDI-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu updated HUDI-1333: -- Status: Open (was: New) > Introduce FlinkHoodieGlobalBloomIndex to hudi-flink-client > -- > > Key: HUDI-1333 > URL: https://issues.apache.org/jira/browse/HUDI-1333 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: wangxianghu >Assignee: Gary Li >Priority: Major > Fix For: 0.7.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HUDI-1333) Introduce FlinkHoodieGlobalBloomIndex to hudi-flink-client
[ https://issues.apache.org/jira/browse/HUDI-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu reassigned HUDI-1333: - Assignee: Gary Li (was: wangxianghu) > Introduce FlinkHoodieGlobalBloomIndex to hudi-flink-client > -- > > Key: HUDI-1333 > URL: https://issues.apache.org/jira/browse/HUDI-1333 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: wangxianghu >Assignee: Gary Li >Priority: Major > Fix For: 0.7.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-1332) Introduce FlinkHoodieBloomIndex to hudi-flink-client
[ https://issues.apache.org/jira/browse/HUDI-1332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17239119#comment-17239119 ] wangxianghu commented on HUDI-1332: --- Hi [~garyli] , I have not started it yet, thanks for picking it up :D assigned it to you. > Introduce FlinkHoodieBloomIndex to hudi-flink-client > > > Key: HUDI-1332 > URL: https://issues.apache.org/jira/browse/HUDI-1332 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: wangxianghu >Assignee: Gary Li >Priority: Major > Fix For: 0.7.0 > > > a flink implementation for bloom index -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HUDI-1332) Introduce FlinkHoodieBloomIndex to hudi-flink-client
[ https://issues.apache.org/jira/browse/HUDI-1332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu reassigned HUDI-1332: - Assignee: Gary Li (was: wangxianghu) > Introduce FlinkHoodieBloomIndex to hudi-flink-client > > > Key: HUDI-1332 > URL: https://issues.apache.org/jira/browse/HUDI-1332 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: wangxianghu >Assignee: Gary Li >Priority: Major > Fix For: 0.7.0 > > > a flink implementation for bloom index -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1417) [DOCS] Qcon Landing practice of Apache Hudi in T3go added to site
[ https://issues.apache.org/jira/browse/HUDI-1417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu updated HUDI-1417: -- Status: Open (was: New) > [DOCS] Qcon Landing practice of Apache Hudi in T3go added to site > - > > Key: HUDI-1417 > URL: https://issues.apache.org/jira/browse/HUDI-1417 > Project: Apache Hudi > Issue Type: Task >Reporter: wangxianghu >Assignee: wangxianghu >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HUDI-1417) [DOCS] Qcon Landing practice of Apache Hudi in T3go added to site
[ https://issues.apache.org/jira/browse/HUDI-1417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu resolved HUDI-1417. --- Resolution: Fixed > [DOCS] Qcon Landing practice of Apache Hudi in T3go added to site > - > > Key: HUDI-1417 > URL: https://issues.apache.org/jira/browse/HUDI-1417 > Project: Apache Hudi > Issue Type: Task >Reporter: wangxianghu >Assignee: wangxianghu >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-1417) [DOCS] Qcon Landing practice of Apache Hudi in T3go added to site
[ https://issues.apache.org/jira/browse/HUDI-1417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17239043#comment-17239043 ] wangxianghu commented on HUDI-1417: --- done via asf-site branch : ce24323b0577d125405620c9a757e95709309c32 > [DOCS] Qcon Landing practice of Apache Hudi in T3go added to site > - > > Key: HUDI-1417 > URL: https://issues.apache.org/jira/browse/HUDI-1417 > Project: Apache Hudi > Issue Type: Task >Reporter: wangxianghu >Assignee: wangxianghu >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1417) [DOCS] Qcon Landing practice of Apache Hudi in T3go added to site
[ https://issues.apache.org/jira/browse/HUDI-1417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu updated HUDI-1417: -- Summary: [DOCS] Qcon Landing practice of Apache Hudi in T3go added to site (was: [DOCS] Qcon Apache Hudi's landing practice in T3go added to site) > [DOCS] Qcon Landing practice of Apache Hudi in T3go added to site > - > > Key: HUDI-1417 > URL: https://issues.apache.org/jira/browse/HUDI-1417 > Project: Apache Hudi > Issue Type: Task >Reporter: wangxianghu >Assignee: wangxianghu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-1417) [DOCS] Qcon Apache Hudi's landing practice in T3go added to site
wangxianghu created HUDI-1417: - Summary: [DOCS] Qcon Apache Hudi's landing practice in T3go added to site Key: HUDI-1417 URL: https://issues.apache.org/jira/browse/HUDI-1417 Project: Apache Hudi Issue Type: Task Reporter: wangxianghu Assignee: wangxianghu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HUDI-981) Use rocksDB as flink state backend
[ https://issues.apache.org/jira/browse/HUDI-981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu reassigned HUDI-981: Assignee: chijunqing (was: wangxianghu) > Use rocksDB as flink state backend > -- > > Key: HUDI-981 > URL: https://issues.apache.org/jira/browse/HUDI-981 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: wangxianghu >Assignee: chijunqing >Priority: Major > > Use rocksDB as flink state backend -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1412) Make HoodieWriteConfig support setting different default value according to engine type
[ https://issues.apache.org/jira/browse/HUDI-1412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu updated HUDI-1412: -- Description: Currently, `HoodieIndexConfig` set its default index type to bloom, which is suitable for spark engine. But，since hoodie has supported flink engine, we should make the default values setted according to engine user used. > Make HoodieWriteConfig support setting different default value according to > engine type > --- > > Key: HUDI-1412 > URL: https://issues.apache.org/jira/browse/HUDI-1412 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: wangxianghu >Assignee: wangxianghu >Priority: Major > > Currently, `HoodieIndexConfig` set its default index type to bloom, which is > suitable for spark engine. > But，since hoodie has supported flink engine, we should make the default > values setted according to engine user used. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1412) Make HoodieWriteConfig support setting different default value according to engine type
[ https://issues.apache.org/jira/browse/HUDI-1412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu updated HUDI-1412: -- Summary: Make HoodieWriteConfig support setting different default value according to engine type (was: Make HoodieConfig support setting different default value according to engine type) > Make HoodieWriteConfig support setting different default value according to > engine type > --- > > Key: HUDI-1412 > URL: https://issues.apache.org/jira/browse/HUDI-1412 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: wangxianghu >Assignee: wangxianghu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1412) Make HoodieConfig support setting different default value according to engine type
[ https://issues.apache.org/jira/browse/HUDI-1412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu updated HUDI-1412: -- Summary: Make HoodieConfig support setting different default value according to engine type (was: Make HoodieIndexConfig support setting different default index type according to engine type) > Make HoodieConfig support setting different default value according to engine > type > -- > > Key: HUDI-1412 > URL: https://issues.apache.org/jira/browse/HUDI-1412 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: wangxianghu >Assignee: wangxianghu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-1412) Make HoodieIndexConfig support setting different default index type according to engine type
wangxianghu created HUDI-1412: - Summary: Make HoodieIndexConfig support setting different default index type according to engine type Key: HUDI-1412 URL: https://issues.apache.org/jira/browse/HUDI-1412 Project: Apache Hudi Issue Type: Sub-task Reporter: wangxianghu Assignee: wangxianghu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1405) Make HoodieFlinkStreamer support read props from local fileSystem
[ https://issues.apache.org/jira/browse/HUDI-1405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu updated HUDI-1405: -- Summary: Make HoodieFlinkStreamer support read props from local fileSystem (was: Make HudiFlinkStreamer support read props from local fileSystem) > Make HoodieFlinkStreamer support read props from local fileSystem > - > > Key: HUDI-1405 > URL: https://issues.apache.org/jira/browse/HUDI-1405 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: wangxianghu >Assignee: Suneel Marthi >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1403) Decouple HoodieFlinkStreamer from Kafka to support more sources
[ https://issues.apache.org/jira/browse/HUDI-1403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu updated HUDI-1403: -- Description: currently, `HoodieFlinkStreamer` support read from kafka only. we should enhance it to support more sources. BTW, we can reuse lots of code from `hudi-utilities` here was: currently, `HudiFlinkStreamer` support read from kafka only. we should enhance it to support more sources. BTW, we can reuse lots of code from `hudi-utilities` here > Decouple HoodieFlinkStreamer from Kafka to support more sources > --- > > Key: HUDI-1403 > URL: https://issues.apache.org/jira/browse/HUDI-1403 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: wangxianghu >Assignee: Suneel Marthi >Priority: Major > > currently, `HoodieFlinkStreamer` support read from kafka only. we should > enhance it to support more sources. > BTW, we can reuse lots of code from `hudi-utilities` here -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1403) Decouple HoodieFlinkStreamer from Kafka to support more sources
[ https://issues.apache.org/jira/browse/HUDI-1403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu updated HUDI-1403: -- Summary: Decouple HoodieFlinkStreamer from Kafka to support more sources (was: Decouple HoodieiFlinkStreamer from Kafka to support more sources) > Decouple HoodieFlinkStreamer from Kafka to support more sources > --- > > Key: HUDI-1403 > URL: https://issues.apache.org/jira/browse/HUDI-1403 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: wangxianghu >Assignee: Suneel Marthi >Priority: Major > > currently, `HudiFlinkStreamer` support read from kafka only. we should > enhance it to support more sources. > BTW, we can reuse lots of code from `hudi-utilities` here -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1403) Decouple HoodieiFlinkStreamer from Kafka to support more sources
[ https://issues.apache.org/jira/browse/HUDI-1403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu updated HUDI-1403: -- Summary: Decouple HoodieiFlinkStreamer from Kafka to support more sources (was: Decouple HudiFlinkStreamer from Kafka to support more sources) > Decouple HoodieiFlinkStreamer from Kafka to support more sources > > > Key: HUDI-1403 > URL: https://issues.apache.org/jira/browse/HUDI-1403 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: wangxianghu >Assignee: Suneel Marthi >Priority: Major > > currently, `HudiFlinkStreamer` support read from kafka only. we should > enhance it to support more sources. > BTW, we can reuse lots of code from `hudi-utilities` here -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-909) Integrate hudi with flink engine
[ https://issues.apache.org/jira/browse/HUDI-909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu updated HUDI-909: - Description: Integrate hudi with flink engine (was: Introduce hudi-flink-client module to support flink engine based on new abstraction) > Integrate hudi with flink engine > > > Key: HUDI-909 > URL: https://issues.apache.org/jira/browse/HUDI-909 > Project: Apache Hudi > Issue Type: Task >Reporter: wangxianghu >Assignee: wangxianghu >Priority: Major > Fix For: 0.6.1 > > > Integrate hudi with flink engine -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-1405) Make HudiFlinkStreamer support read props from local fileSystem
wangxianghu created HUDI-1405: - Summary: Make HudiFlinkStreamer support read props from local fileSystem Key: HUDI-1405 URL: https://issues.apache.org/jira/browse/HUDI-1405 Project: Apache Hudi Issue Type: Sub-task Reporter: wangxianghu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1403) Decouple HudiFlinkStreamer from Kafka to support more sources
[ https://issues.apache.org/jira/browse/HUDI-1403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu updated HUDI-1403: -- Description: currently, `HudiFlinkStreamer` support read from kafka only. we should enhance it to support more sources. BTW, we can reuse lots of code from `hudi-utilities` here > Decouple HudiFlinkStreamer from Kafka to support more sources > - > > Key: HUDI-1403 > URL: https://issues.apache.org/jira/browse/HUDI-1403 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: wangxianghu >Priority: Major > > currently, `HudiFlinkStreamer` support read from kafka only. we should > enhance it to support more sources. > BTW, we can reuse lots of code from `hudi-utilities` here -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HUDI-1402) Add a blog about how to use flink engine
[ https://issues.apache.org/jira/browse/HUDI-1402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu reassigned HUDI-1402: - Assignee: wangxianghu > Add a blog about how to use flink engine > > > Key: HUDI-1402 > URL: https://issues.apache.org/jira/browse/HUDI-1402 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: wangxianghu >Assignee: wangxianghu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-1404) Make flink engine support bulkinsert operation
wangxianghu created HUDI-1404: - Summary: Make flink engine support bulkinsert operation Key: HUDI-1404 URL: https://issues.apache.org/jira/browse/HUDI-1404 Project: Apache Hudi Issue Type: Sub-task Reporter: wangxianghu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-1403) Decouple HudiFlinkStreamer from Kafka to support more sources
wangxianghu created HUDI-1403: - Summary: Decouple HudiFlinkStreamer from Kafka to support more sources Key: HUDI-1403 URL: https://issues.apache.org/jira/browse/HUDI-1403 Project: Apache Hudi Issue Type: Sub-task Reporter: wangxianghu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-1402) Add a blog about how to use flink engine
wangxianghu created HUDI-1402: - Summary: Add a blog about how to use flink engine Key: HUDI-1402 URL: https://issues.apache.org/jira/browse/HUDI-1402 Project: Apache Hudi Issue Type: Sub-task Reporter: wangxianghu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-909) Integerate hudi with flink engine
[ https://issues.apache.org/jira/browse/HUDI-909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu updated HUDI-909: - Summary: Integerate hudi with flink engine (was: Introduce hudi-flink-client module to support flink engine) > Integerate hudi with flink engine > - > > Key: HUDI-909 > URL: https://issues.apache.org/jira/browse/HUDI-909 > Project: Apache Hudi > Issue Type: Task >Reporter: wangxianghu >Assignee: wangxianghu >Priority: Major > Fix For: 0.6.1 > > > Introduce hudi-flink-client module to support flink engine based on new > abstraction -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-909) Integrate hudi with flink engine
[ https://issues.apache.org/jira/browse/HUDI-909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu updated HUDI-909: - Summary: Integrate hudi with flink engine (was: Integerate hudi with flink engine) > Integrate hudi with flink engine > > > Key: HUDI-909 > URL: https://issues.apache.org/jira/browse/HUDI-909 > Project: Apache Hudi > Issue Type: Task >Reporter: wangxianghu >Assignee: wangxianghu >Priority: Major > Fix For: 0.6.1 > > > Introduce hudi-flink-client module to support flink engine based on new > abstraction -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1400) Replace Operation enum with WriteOperationType
[ https://issues.apache.org/jira/browse/HUDI-1400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu updated HUDI-1400: -- Description: currently, we use `org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.Operation` to represent operation type in `hudi-utilities` and `hudi-integ-test`module, while `hudi-common` use `org.apache.hudi.common.model.WriteOperationType`. we can unify them. was: currently, we use `org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.Operation` to represent operation type in `hudi-utilities` module, while `hudi-common` use `org.apache.hudi.common.model.WriteOperationType`. we can unify them. > Replace Operation enum with WriteOperationType > -- > > Key: HUDI-1400 > URL: https://issues.apache.org/jira/browse/HUDI-1400 > Project: Apache Hudi > Issue Type: Improvement >Reporter: wangxianghu >Assignee: wangxianghu >Priority: Major > > currently, we use > `org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.Operation` to > represent operation type in `hudi-utilities` and `hudi-integ-test`module, > while `hudi-common` use `org.apache.hudi.common.model.WriteOperationType`. > we can unify them. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1400) Replace Operation enum with WriteOperationType
[ https://issues.apache.org/jira/browse/HUDI-1400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu updated HUDI-1400: -- Summary: Replace Operation enum with WriteOperationType (was: Replace Operation enum with WriteOperationType in hudi-utilities) > Replace Operation enum with WriteOperationType > -- > > Key: HUDI-1400 > URL: https://issues.apache.org/jira/browse/HUDI-1400 > Project: Apache Hudi > Issue Type: Improvement >Reporter: wangxianghu >Assignee: wangxianghu >Priority: Major > > currently, we use > `org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.Operation` to > represent operation type in `hudi-utilities` module, while `hudi-common` use > `org.apache.hudi.common.model.WriteOperationType`. > we can unify them. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1400) Replace Operation enum with WriteOperationType in hudi-utilities
[ https://issues.apache.org/jira/browse/HUDI-1400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu updated HUDI-1400: -- Description: currently, we use `org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.Operation` to represent operation type in `hudi-utilities` module, while `hudi-common` use `org.apache.hudi.common.model.WriteOperationType`. we can unify them. > Replace Operation enum with WriteOperationType in hudi-utilities > > > Key: HUDI-1400 > URL: https://issues.apache.org/jira/browse/HUDI-1400 > Project: Apache Hudi > Issue Type: Improvement >Reporter: wangxianghu >Assignee: wangxianghu >Priority: Major > > currently, we use > `org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.Operation` to > represent operation type in `hudi-utilities` module, while `hudi-common` use > `org.apache.hudi.common.model.WriteOperationType`. > we can unify them. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HUDI-1400) Replace Operation enum with WriteOperationType in hudi-utilities
[ https://issues.apache.org/jira/browse/HUDI-1400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu reassigned HUDI-1400: - Assignee: wangxianghu > Replace Operation enum with WriteOperationType in hudi-utilities > > > Key: HUDI-1400 > URL: https://issues.apache.org/jira/browse/HUDI-1400 > Project: Apache Hudi > Issue Type: Improvement >Reporter: wangxianghu >Assignee: wangxianghu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-1400) Replace Operation enum with WriteOperationType in hudi-utilities
wangxianghu created HUDI-1400: - Summary: Replace Operation enum with WriteOperationType in hudi-utilities Key: HUDI-1400 URL: https://issues.apache.org/jira/browse/HUDI-1400 Project: Apache Hudi Issue Type: Improvement Reporter: wangxianghu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1337) Deduplicate data in one batch for flink engine
[ https://issues.apache.org/jira/browse/HUDI-1337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu updated HUDI-1337: -- Summary: Deduplicate data in one batch for flink engine (was: Introduce FlinkInMemoryHashIndex to hudi-flink-client) > Deduplicate data in one batch for flink engine > -- > > Key: HUDI-1337 > URL: https://issues.apache.org/jira/browse/HUDI-1337 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: wangxianghu >Assignee: liujinhui >Priority: Major > Fix For: 0.7.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HUDI-982) Make flink engine support MOR table
[ https://issues.apache.org/jira/browse/HUDI-982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu reassigned HUDI-982: Assignee: liujinhui (was: wangxianghu) > Make flink engine support MOR table > --- > > Key: HUDI-982 > URL: https://issues.apache.org/jira/browse/HUDI-982 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: wangxianghu >Assignee: liujinhui >Priority: Major > > Make flink engine support MOR table -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-982) Make flink engine support MOR table
[ https://issues.apache.org/jira/browse/HUDI-982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu updated HUDI-982: - Summary: Make flink engine support MOR table (was: Add commit action implemetation to hudi-flink-client) > Make flink engine support MOR table > --- > > Key: HUDI-982 > URL: https://issues.apache.org/jira/browse/HUDI-982 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: wangxianghu >Assignee: wangxianghu >Priority: Major > > Add commit action implemetation to hudi-flink-client -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-982) Make flink engine support MOR table
[ https://issues.apache.org/jira/browse/HUDI-982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu updated HUDI-982: - Description: Make flink engine support MOR table (was: Add commit action implemetation to hudi-flink-client) > Make flink engine support MOR table > --- > > Key: HUDI-982 > URL: https://issues.apache.org/jira/browse/HUDI-982 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: wangxianghu >Assignee: wangxianghu >Priority: Major > > Make flink engine support MOR table -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-981) Use rocksDB as flink state backend
[ https://issues.apache.org/jira/browse/HUDI-981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu updated HUDI-981: - Description: Use rocksDB as flink state backend (was: Add async compact implemetation to hudi-flink-client) > Use rocksDB as flink state backend > -- > > Key: HUDI-981 > URL: https://issues.apache.org/jira/browse/HUDI-981 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: wangxianghu >Assignee: wangxianghu >Priority: Major > > Use rocksDB as flink state backend -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-981) Use rocksDB as flink state backend
[ https://issues.apache.org/jira/browse/HUDI-981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu updated HUDI-981: - Summary: Use rocksDB as flink state backend (was: Add async compact implemetation to hudi-flink-client) > Use rocksDB as flink state backend > -- > > Key: HUDI-981 > URL: https://issues.apache.org/jira/browse/HUDI-981 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: wangxianghu >Assignee: wangxianghu >Priority: Major > > Add async compact implemetation to hudi-flink-client -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-1377) clean duplicate code in HoodieSparkSqlWriter
[ https://issues.apache.org/jira/browse/HUDI-1377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17229629#comment-17229629 ] wangxianghu commented on HUDI-1377: --- [~wangshikai] please assign this ticket to yourself，then we can close it. > clean duplicate code in HoodieSparkSqlWriter > > > Key: HUDI-1377 > URL: https://issues.apache.org/jira/browse/HUDI-1377 > Project: Apache Hudi > Issue Type: Improvement > Components: Spark Integration >Reporter: karl wang >Priority: Major > Labels: pull-request-available > > clean duplicate code in HoodieSparkSqlWriter -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-1377) clean duplicate code in HoodieSparkSqlWriter
[ https://issues.apache.org/jira/browse/HUDI-1377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17229626#comment-17229626 ] wangxianghu commented on HUDI-1377: --- done via master branch : 430d4b428e7c5b325c7414a187f9cda158c2758a > clean duplicate code in HoodieSparkSqlWriter > > > Key: HUDI-1377 > URL: https://issues.apache.org/jira/browse/HUDI-1377 > Project: Apache Hudi > Issue Type: Improvement > Components: Spark Integration >Reporter: karl wang >Priority: Major > Labels: pull-request-available > > clean duplicate code in HoodieSparkSqlWriter -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1377) clean duplicate code in HoodieSparkSqlWriter
[ https://issues.apache.org/jira/browse/HUDI-1377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu updated HUDI-1377: -- Status: Open (was: New) > clean duplicate code in HoodieSparkSqlWriter > > > Key: HUDI-1377 > URL: https://issues.apache.org/jira/browse/HUDI-1377 > Project: Apache Hudi > Issue Type: Improvement > Components: Spark Integration >Reporter: karl wang >Priority: Major > Labels: pull-request-available > > clean duplicate code in HoodieSparkSqlWriter -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-912) Refactor and relocate KeyGenerator to support more engines
[ https://issues.apache.org/jira/browse/HUDI-912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17225041#comment-17225041 ] wangxianghu commented on HUDI-912: -- done via master branch : d160abb43740e0bcdf40458c345ecd2d74e6698c > Refactor and relocate KeyGenerator to support more engines > -- > > Key: HUDI-912 > URL: https://issues.apache.org/jira/browse/HUDI-912 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: wangxianghu >Assignee: wangxianghu >Priority: Major > Labels: pull-request-available > > Currently, `keyGenerator`s are implemented in `hudi-spark` module, they can > only be used by spark engine. > Since `keyGenerator` is a core tool for hudi, they should be > engine-independent. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HUDI-912) Refactor and relocate KeyGenerator to support more engines
[ https://issues.apache.org/jira/browse/HUDI-912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu resolved HUDI-912. -- Resolution: Fixed > Refactor and relocate KeyGenerator to support more engines > -- > > Key: HUDI-912 > URL: https://issues.apache.org/jira/browse/HUDI-912 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: wangxianghu >Assignee: wangxianghu >Priority: Major > Labels: pull-request-available > > Currently, `keyGenerator`s are implemented in `hudi-spark` module, they can > only be used by spark engine. > Since `keyGenerator` is a core tool for hudi, they should be > engine-independent. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-912) Refactor and relocate KeyGenerator to support more engines
[ https://issues.apache.org/jira/browse/HUDI-912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu updated HUDI-912: - Status: Open (was: New) > Refactor and relocate KeyGenerator to support more engines > -- > > Key: HUDI-912 > URL: https://issues.apache.org/jira/browse/HUDI-912 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: wangxianghu >Assignee: wangxianghu >Priority: Major > Labels: pull-request-available > > Currently, `keyGenerator`s are implemented in `hudi-spark` module, they can > only be used by spark engine. > Since `keyGenerator` is a core tool for hudi, they should be > engine-independent. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-913) Update docs about KeyGenerator
[ https://issues.apache.org/jira/browse/HUDI-913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu updated HUDI-913: - Description: update default values about `KeyGenerator` was: Add read client implemetation to hudi-flink-client > Update docs about KeyGenerator > -- > > Key: HUDI-913 > URL: https://issues.apache.org/jira/browse/HUDI-913 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: wangxianghu >Assignee: wangxianghu >Priority: Major > > update default values about `KeyGenerator` > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-913) Update docs about KeyGenerator
[ https://issues.apache.org/jira/browse/HUDI-913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu updated HUDI-913: - Summary: Update docs about KeyGenerator (was: Add read client implemetation to hudi-flink-client) > Update docs about KeyGenerator > -- > > Key: HUDI-913 > URL: https://issues.apache.org/jira/browse/HUDI-913 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: wangxianghu >Assignee: wangxianghu >Priority: Major > > Add read client implemetation to hudi-flink-client > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-912) Refactor and relocate KeyGenerator to support more engines
[ https://issues.apache.org/jira/browse/HUDI-912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu updated HUDI-912: - Summary: Refactor and relocate KeyGenerator to support more engines (was: Refactor KeyGenerator and relocate them to support more engines) > Refactor and relocate KeyGenerator to support more engines > -- > > Key: HUDI-912 > URL: https://issues.apache.org/jira/browse/HUDI-912 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: wangxianghu >Assignee: wangxianghu >Priority: Major > > Currently, `keyGenerator`s are implemented in `hudi-spark` module, they can > only be used by spark engine. > Since `keyGenerator` is a core tool for hudi, they should be > engine-independent. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-912) Refactor KeyGenerator and relocate them to support more engines
[ https://issues.apache.org/jira/browse/HUDI-912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu updated HUDI-912: - Description: Currently, `keyGenerator`s are implemented in `hudi-spark` module, they can only be used by spark engine. Since `keyGenerator` is a core tool for hudi, they should be engine-independent. was: Currently, `keyGenerator`s are implemented in `hudi-spark` module, they can only be used by spark engine. Since `keyGenerator` is a core tool for hudi, there should be engine-independent. > Refactor KeyGenerator and relocate them to support more engines > --- > > Key: HUDI-912 > URL: https://issues.apache.org/jira/browse/HUDI-912 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: wangxianghu >Assignee: wangxianghu >Priority: Major > > Currently, `keyGenerator`s are implemented in `hudi-spark` module, they can > only be used by spark engine. > Since `keyGenerator` is a core tool for hudi, they should be > engine-independent. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-912) Refactor KeyGenerator and relocate them to support more engines
[ https://issues.apache.org/jira/browse/HUDI-912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu updated HUDI-912: - Description: Currently, `keyGenerator`s are implemented in `hudi-spark` module, they can only be used by spark engine. Since `keyGenerator` is a core tool for hudi, there should be engine-independent. was:Add write client implemetation to hudi-flink-client > Refactor KeyGenerator and relocate them to support more engines > --- > > Key: HUDI-912 > URL: https://issues.apache.org/jira/browse/HUDI-912 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: wangxianghu >Assignee: wangxianghu >Priority: Major > > Currently, `keyGenerator`s are implemented in `hudi-spark` module, they can > only be used by spark engine. > Since `keyGenerator` is a core tool for hudi, there should be > engine-independent. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-912) Refactor KeyGenerator and relocate them to support more engines
[ https://issues.apache.org/jira/browse/HUDI-912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu updated HUDI-912: - Summary: Refactor KeyGenerator and relocate them to support more engines (was: Add write client implemetation to hudi-flink-client) > Refactor KeyGenerator and relocate them to support more engines > --- > > Key: HUDI-912 > URL: https://issues.apache.org/jira/browse/HUDI-912 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: wangxianghu >Assignee: wangxianghu >Priority: Major > > Add write client implemetation to hudi-flink-client -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1327) Introduce base implementation of hudi-flink-client
[ https://issues.apache.org/jira/browse/HUDI-1327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu updated HUDI-1327: -- Summary: Introduce base implementation of hudi-flink-client (was: Introduce base implemetation of hudi-flink-client) > Introduce base implementation of hudi-flink-client > -- > > Key: HUDI-1327 > URL: https://issues.apache.org/jira/browse/HUDI-1327 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: wangxianghu >Assignee: wangxianghu >Priority: Major > Labels: pull-request-available > Fix For: 0.6.1 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-911) Add Blog about Hudi-Spark decoupling and Flink integration design
[ https://issues.apache.org/jira/browse/HUDI-911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu updated HUDI-911: - Summary: Add Blog about Hudi-Spark decoupling and Flink integration design (was: Add table implemetation to hudi-flink-client) > Add Blog about Hudi-Spark decoupling and Flink integration design > - > > Key: HUDI-911 > URL: https://issues.apache.org/jira/browse/HUDI-911 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: wangxianghu >Assignee: wangxianghu >Priority: Major > > Add table implemetation to hudi-flink-client -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-911) Add Blog about Hudi-Spark decoupling and Flink integration design
[ https://issues.apache.org/jira/browse/HUDI-911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu updated HUDI-911: - Description: Add a blog to Introduction Hudi-Spark decoupling and Flink integration design (was: Add table implemetation to hudi-flink-client) > Add Blog about Hudi-Spark decoupling and Flink integration design > - > > Key: HUDI-911 > URL: https://issues.apache.org/jira/browse/HUDI-911 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: wangxianghu >Assignee: wangxianghu >Priority: Major > > Add a blog to Introduction Hudi-Spark decoupling and Flink integration design -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-910) Introduce HoodieWriteInput for hudi write client
[ https://issues.apache.org/jira/browse/HUDI-910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu updated HUDI-910: - Status: Open (was: New) > Introduce HoodieWriteInput for hudi write client > > > Key: HUDI-910 > URL: https://issues.apache.org/jira/browse/HUDI-910 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: wangxianghu >Assignee: wangxianghu >Priority: Major > Labels: pull-request-available > > h1. Introduce HoodieWriteInput for hudi write client -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-910) Add Blog about Hudi-Spark decoupling and Flink integration design
[ https://issues.apache.org/jira/browse/HUDI-910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu updated HUDI-910: - Description: Intruduce FlinkHoodieIndex to hudi-flink-client (was: Abstract implemetation of HoodieIndex for flink) > Add Blog about Hudi-Spark decoupling and Flink integration design > - > > Key: HUDI-910 > URL: https://issues.apache.org/jira/browse/HUDI-910 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: wangxianghu >Assignee: wangxianghu >Priority: Major > Labels: pull-request-available > > Intruduce FlinkHoodieIndex to hudi-flink-client -- This message was sent by Atlassian Jira (v8.3.4#803005)
