[jira] [Commented] (HUDI-561) hudi partition path config

2020-01-20 Thread liujinhui (Jira)
[ https://issues.apache.org/jira/browse/HUDI-561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019886#comment-17019886 ] liujinhui commented on HUDI-561: There is also a problem with using the transformer, it will modify the

[jira] [Commented] (HUDI-561) hudi partition path config

2020-01-21 Thread liujinhui (Jira)
[ https://issues.apache.org/jira/browse/HUDI-561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020719#comment-17020719 ] liujinhui commented on HUDI-561: If it is time format data, we can create a time string format. Users can

[jira] [Commented] (HUDI-561) hudi partition path config

2020-01-20 Thread liujinhui (Jira)
[ https://issues.apache.org/jira/browse/HUDI-561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019841#comment-17019841 ] liujinhui commented on HUDI-561: [~yanghua] [~vinoth] > hudi partition path config >

[jira] [Created] (HUDI-561) hudi partition path config

2020-01-20 Thread liujinhui (Jira)
liujinhui created HUDI-561: -- Summary: hudi partition path config Key: HUDI-561 URL: https://issues.apache.org/jira/browse/HUDI-561 Project: Apache Hudi (incubating) Issue Type: Improvement

[jira] [Commented] (HUDI-76) CSV Source support for Hudi Delta Streamer

2020-01-09 Thread liujinhui (Jira)
[ https://issues.apache.org/jira/browse/HUDI-76?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17012411#comment-17012411 ] liujinhui commented on HUDI-76: --- [~guoyihua] I tried it, it seems that it will cause data misalignment and

[jira] [Created] (HUDI-471) hudi quickstart spark-shell local

2019-12-26 Thread liujinhui (Jira)
liujinhui created HUDI-471: -- Summary: hudi quickstart spark-shell local Key: HUDI-471 URL: https://issues.apache.org/jira/browse/HUDI-471 Project: Apache Hudi (incubating) Issue Type: Improvement

[jira] [Created] (HUDI-507) Support \ t split hdfs source

2020-01-07 Thread liujinhui (Jira)
liujinhui created HUDI-507: -- Summary: Support \ t split hdfs source Key: HUDI-507 URL: https://issues.apache.org/jira/browse/HUDI-507 Project: Apache Hudi (incubating) Issue Type: Improvement

[jira] [Commented] (HUDI-471) hudi quickstart spark-shell local

2020-01-08 Thread liujinhui (Jira)
[ https://issues.apache.org/jira/browse/HUDI-471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010450#comment-17010450 ] liujinhui commented on HUDI-471: This does not need to continue to pay attention to, there is no problem in

[jira] [Updated] (HUDI-471) hudi quickstart spark-shell local

2020-01-08 Thread liujinhui (Jira)
[ https://issues.apache.org/jira/browse/HUDI-471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liujinhui updated HUDI-471: --- Status: Open (was: New) > hudi quickstart spark-shell local > - > >

[jira] [Assigned] (HUDI-471) hudi quickstart spark-shell local

2020-01-08 Thread liujinhui (Jira)
[ https://issues.apache.org/jira/browse/HUDI-471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liujinhui reassigned HUDI-471: -- Assignee: liujinhui > hudi quickstart spark-shell local > - > >

[jira] [Closed] (HUDI-471) hudi quickstart spark-shell local

2020-01-08 Thread liujinhui (Jira)
[ https://issues.apache.org/jira/browse/HUDI-471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liujinhui closed HUDI-471. -- Resolution: Not A Problem This does not need to continue to pay attention to, there is no problem in itself >

[jira] [Commented] (HUDI-507) Support \ t split hdfs source

2020-01-07 Thread liujinhui (Jira)
[ https://issues.apache.org/jira/browse/HUDI-507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010247#comment-17010247 ] liujinhui commented on HUDI-507: [~vinoth]  _please give me the contributor permission, Email sent before,

[jira] [Issue Comment Deleted] (HUDI-507) Support \ t split hdfs source

2020-01-07 Thread liujinhui (Jira)
[ https://issues.apache.org/jira/browse/HUDI-507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liujinhui updated HUDI-507: --- Comment: was deleted (was: [~vinoth]  _please give me the contributor permission, Email sent before, but not

[jira] [Updated] (HUDI-507) Support \ t split hdfs source

2020-01-07 Thread liujinhui (Jira)
[ https://issues.apache.org/jira/browse/HUDI-507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liujinhui updated HUDI-507: --- Description: hi,hudi   Current Hudi data source does not support HDFS file data splitting with \ t

[jira] [Updated] (HUDI-507) Support \ t split hdfs source

2020-01-07 Thread liujinhui (Jira)
[ https://issues.apache.org/jira/browse/HUDI-507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liujinhui updated HUDI-507: --- Description: hi,hudi   Current Hudi data source does not support HDFS file data splitting with \ t

[jira] [Updated] (HUDI-507) Support \ t split hdfs source

2020-01-07 Thread liujinhui (Jira)
[ https://issues.apache.org/jira/browse/HUDI-507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liujinhui updated HUDI-507: --- Description: hi,hudi   Current Hudi data source does not support HDFS file data splitting with \ t

[jira] [Commented] (HUDI-507) Support \ t split hdfs source

2020-01-07 Thread liujinhui (Jira)
[ https://issues.apache.org/jira/browse/HUDI-507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010246#comment-17010246 ] liujinhui commented on HUDI-507: [~vinoth]  _please give me the contributor permission, Email sent before,

[jira] [Commented] (HUDI-76) CSV Source support for Hudi Delta Streamer

2020-01-08 Thread liujinhui (Jira)
[ https://issues.apache.org/jira/browse/HUDI-76?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010658#comment-17010658 ] liujinhui commented on HUDI-76: --- [~guoyihua]  hello, CSV to ROW I see your implementation, I think the key

[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes

2020-05-18 Thread liujinhui (Jira)
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110824#comment-17110824 ] liujinhui commented on HUDI-648: Hello, are there any good ideas and design suggestions for this proposal?

[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes

2020-03-18 Thread liujinhui (Jira)
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17062242#comment-17062242 ] liujinhui commented on HUDI-648: Hello, I also encountered this problem recently. Occasionally the kafka

[jira] [Created] (HUDI-869) Add support for alluxio

2020-05-08 Thread liujinhui (Jira)
liujinhui created HUDI-869: -- Summary: Add support for alluxio Key: HUDI-869 URL: https://issues.apache.org/jira/browse/HUDI-869 Project: Apache Hudi (incubating) Issue Type: New Feature

[jira] [Reopened] (HUDI-869) Add support for alluxio

2020-05-08 Thread liujinhui (Jira)
[ https://issues.apache.org/jira/browse/HUDI-869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liujinhui reopened HUDI-869: > Add support for alluxio > > > Key: HUDI-869 > URL:

[jira] [Updated] (HUDI-869) Add support for alluxio

2020-05-08 Thread liujinhui (Jira)
[ https://issues.apache.org/jira/browse/HUDI-869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liujinhui updated HUDI-869: --- Status: Closed (was: Patch Available) > Add support for alluxio > > >

[jira] [Updated] (HUDI-869) Add support for alluxio

2020-05-08 Thread liujinhui (Jira)
[ https://issues.apache.org/jira/browse/HUDI-869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liujinhui updated HUDI-869: --- Status: Open (was: New) > Add support for alluxio > > > Key:

[jira] [Updated] (HUDI-869) Add support for alluxio

2020-05-08 Thread liujinhui (Jira)
[ https://issues.apache.org/jira/browse/HUDI-869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liujinhui updated HUDI-869: --- Status: Patch Available (was: In Progress) > Add support for alluxio > > >

[jira] [Updated] (HUDI-869) Add support for alluxio

2020-05-08 Thread liujinhui (Jira)
[ https://issues.apache.org/jira/browse/HUDI-869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liujinhui updated HUDI-869: --- Status: In Progress (was: Open) > Add support for alluxio > > >

[jira] [Updated] (HUDI-561) hudi partition path config

2020-05-09 Thread liujinhui (Jira)
[ https://issues.apache.org/jira/browse/HUDI-561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liujinhui updated HUDI-561: --- Status: Open (was: New) > hudi partition path config > -- > > Key:

[jira] [Updated] (HUDI-507) Support \ t split hdfs source

2020-05-09 Thread liujinhui (Jira)
[ https://issues.apache.org/jira/browse/HUDI-507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liujinhui updated HUDI-507: --- Status: Open (was: New) > Support \ t split hdfs source > - > >

[jira] [Closed] (HUDI-561) hudi partition path config

2020-05-09 Thread liujinhui (Jira)
[ https://issues.apache.org/jira/browse/HUDI-561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liujinhui closed HUDI-561. -- Resolution: Duplicate > hudi partition path config > -- > > Key:

[jira] [Closed] (HUDI-507) Support \ t split hdfs source

2020-05-09 Thread liujinhui (Jira)
[ https://issues.apache.org/jira/browse/HUDI-507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liujinhui closed HUDI-507. -- Resolution: Duplicate > Support \ t split hdfs source > - > > Key:

[jira] [Updated] (HUDI-914) support different target data clusters

2020-05-19 Thread liujinhui (Jira)
[ https://issues.apache.org/jira/browse/HUDI-914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liujinhui updated HUDI-914: --- Status: Open (was: New) > support different target data clusters > -- > >

[jira] [Commented] (HUDI-914) support different target data clusters

2020-05-19 Thread liujinhui (Jira)
[ https://issues.apache.org/jira/browse/HUDI-914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111257#comment-17111257 ] liujinhui commented on HUDI-914: [~yanghua]  > support different target data clusters >

[jira] [Created] (HUDI-914) support different target data clusters

2020-05-19 Thread liujinhui (Jira)
liujinhui created HUDI-914: -- Summary: support different target data clusters Key: HUDI-914 URL: https://issues.apache.org/jira/browse/HUDI-914 Project: Apache Hudi (incubating) Issue Type: New

[jira] [Commented] (HUDI-914) support different target data clusters

2020-05-19 Thread liujinhui (Jira)
[ https://issues.apache.org/jira/browse/HUDI-914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111259#comment-17111259 ] liujinhui commented on HUDI-914: [~vinothchandar] > support different target data clusters >

[jira] [Updated] (HUDI-918) Hudi can't get data

2020-05-21 Thread liujinhui (Jira)
[ https://issues.apache.org/jira/browse/HUDI-918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liujinhui updated HUDI-918: --- Status: Open (was: New) > Hudi can't get data > --- > > Key: HUDI-918 >

[jira] [Created] (HUDI-918) Hudi can't get data

2020-05-21 Thread liujinhui (Jira)
liujinhui created HUDI-918: -- Summary: Hudi can't get data Key: HUDI-918 URL: https://issues.apache.org/jira/browse/HUDI-918 Project: Apache Hudi (incubating) Issue Type: Bug Components:

[jira] [Updated] (HUDI-918) Fix kafkaOffsetGen can not read kafka data bug

2020-05-21 Thread liujinhui (Jira)
[ https://issues.apache.org/jira/browse/HUDI-918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liujinhui updated HUDI-918: --- Description: When the sourcelimit is less than the number of Kafka partitions, Hudi cannot get the data

[jira] [Updated] (HUDI-918) deltastreamer bug is no new data

2020-05-21 Thread liujinhui (Jira)
[ https://issues.apache.org/jira/browse/HUDI-918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liujinhui updated HUDI-918: --- Summary: deltastreamer bug is no new data (was: Hudi can't get data) > deltastreamer bug is no new data >

[jira] [Updated] (HUDI-1006) deltastreamer set auto.offset.reset=latest can't consume data

2020-06-08 Thread liujinhui (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liujinhui updated HUDI-1006: Status: Open (was: New) > deltastreamer set auto.offset.reset=latest can't consume data >

[jira] [Created] (HUDI-1006) deltastreamer set auto.offset.reset=latest can't consume data

2020-06-08 Thread liujinhui (Jira)
liujinhui created HUDI-1006: --- Summary: deltastreamer set auto.offset.reset=latest can't consume data Key: HUDI-1006 URL: https://issues.apache.org/jira/browse/HUDI-1006 Project: Apache Hudi Issue

[jira] [Assigned] (HUDI-1007) When earliestOffsets is greater than checkpoint, Hudi will not be able to successfully consume data

2020-06-08 Thread liujinhui (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liujinhui reassigned HUDI-1007: --- Assignee: liujinhui > When earliestOffsets is greater than checkpoint, Hudi will not be able to >

[jira] [Created] (HUDI-1007) When earliestOffsets is greater than checkpoint, Hudi will not be able to successfully consume data

2020-06-08 Thread liujinhui (Jira)
liujinhui created HUDI-1007: --- Summary: When earliestOffsets is greater than checkpoint, Hudi will not be able to successfully consume data Key: HUDI-1007 URL: https://issues.apache.org/jira/browse/HUDI-1007

[jira] [Updated] (HUDI-1007) When earliestOffsets is greater than checkpoint, Hudi will not be able to successfully consume data

2020-06-08 Thread liujinhui (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liujinhui updated HUDI-1007: Description: Use deltastreamer to consume kafka, When earliestOffsets is greater than checkpoint, Hudi

[jira] [Commented] (HUDI-1007) When earliestOffsets is greater than checkpoint, Hudi will not be able to successfully consume data

2020-06-08 Thread liujinhui (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128400#comment-17128400 ] liujinhui commented on HUDI-1007: - Yes, every run will check the offset of the earliest in the offect

[jira] [Commented] (HUDI-1007) When earliestOffsets is greater than checkpoint, Hudi will not be able to successfully consume data

2020-06-08 Thread liujinhui (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128395#comment-17128395 ] liujinhui commented on HUDI-1007: - # This test case is really special and requires a production

[jira] [Closed] (HUDI-918) Fix kafkaOffsetGen can not read kafka data bug

2020-06-08 Thread liujinhui (Jira)
[ https://issues.apache.org/jira/browse/HUDI-918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liujinhui closed HUDI-918. -- > Fix kafkaOffsetGen can not read kafka data bug > -- > >

[jira] [Resolved] (HUDI-918) Fix kafkaOffsetGen can not read kafka data bug

2020-06-08 Thread liujinhui (Jira)
[ https://issues.apache.org/jira/browse/HUDI-918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liujinhui resolved HUDI-918. Resolution: Fixed > Fix kafkaOffsetGen can not read kafka data bug >

[jira] [Commented] (HUDI-1007) When earliestOffsets is greater than checkpoint, Hudi will not be able to successfully consume data

2020-06-08 Thread liujinhui (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128220#comment-17128220 ] liujinhui commented on HUDI-1007: - *[~vinoth]  What is your idea?* > When earliestOffsets is greater than

[jira] [Assigned] (HUDI-1006) deltastreamer set auto.offset.reset=latest can't consume data

2020-06-08 Thread liujinhui (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liujinhui reassigned HUDI-1006: --- Assignee: Tianye Li (was: liujinhui) > deltastreamer set auto.offset.reset=latest can't consume

[jira] [Commented] (HUDI-1006) deltastreamer set auto.offset.reset=latest can't consume data

2020-06-08 Thread liujinhui (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128197#comment-17128197 ] liujinhui commented on HUDI-1006: - [~Litianye]  It's up to you to fix this problem > deltastreamer set

[jira] [Commented] (HUDI-914) support different target data clusters

2020-06-12 Thread liujinhui (Jira)
[ https://issues.apache.org/jira/browse/HUDI-914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17134637#comment-17134637 ] liujinhui commented on HUDI-914: The deltastreamer task always runs on a certain cluster, but the

[jira] [Commented] (HUDI-1007) When earliestOffsets is greater than checkpoint, Hudi will not be able to successfully consume data

2020-06-12 Thread liujinhui (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17134634#comment-17134634 ] liujinhui commented on HUDI-1007: - Caused by: org.apache.spark.SparkException: Job aborted due to stage

[jira] [Commented] (HUDI-1007) When earliestOffsets is greater than checkpoint, Hudi will not be able to successfully consume data

2020-06-12 Thread liujinhui (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17134635#comment-17134635 ] liujinhui commented on HUDI-1007: - I think that starting from the latest offect can indeed solve this

[jira] [Updated] (HUDI-1007) When earliestOffsets is greater than checkpoint, Hudi will not be able to successfully consume data

2020-06-08 Thread liujinhui (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liujinhui updated HUDI-1007: Component/s: DeltaStreamer > When earliestOffsets is greater than checkpoint, Hudi will not be able to >

[jira] [Updated] (HUDI-1007) When earliestOffsets is greater than checkpoint, Hudi will not be able to successfully consume data

2020-06-08 Thread liujinhui (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liujinhui updated HUDI-1007: Status: Open (was: New) > When earliestOffsets is greater than checkpoint, Hudi will not be able to >

[jira] [Commented] (HUDI-914) support different target data clusters

2020-06-08 Thread liujinhui (Jira)
[ https://issues.apache.org/jira/browse/HUDI-914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128757#comment-17128757 ] liujinhui commented on HUDI-914: Due to the needs of some business parties, they only want the hudi dataset

[jira] [Commented] (HUDI-1007) When earliestOffsets is greater than checkpoint, Hudi will not be able to successfully consume data

2020-06-16 Thread liujinhui (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17136660#comment-17136660 ] liujinhui commented on HUDI-1007: -   So the best way at present is to discover the data delay early

[jira] [Commented] (HUDI-1007) When earliestOffsets is greater than checkpoint, Hudi will not be able to successfully consume data

2020-06-16 Thread liujinhui (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17136655#comment-17136655 ] liujinhui commented on HUDI-1007: - > in your case, is this true? does setting the flag help actually? coz