[jira] [Created] (KUDU-3356) Support vectorization evaluation while scanning
LiFu He created KUDU-3356: - Summary: Support vectorization evaluation while scanning Key: KUDU-3356 URL: https://issues.apache.org/jira/browse/KUDU-3356 Project: Kudu Issue Type: New Feature Reporter: LiFu He Kudu is a column storage system, vectorization can speed up scanning. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (KUDU-3355) Move outdated range partitions directly to hdfs
[ https://issues.apache.org/jira/browse/KUDU-3355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] LiFu He updated KUDU-3355: -- Description: There is a feature in ClickHouse that allows MergeTree table to move range partitions from local disks to remote storage(hdfs/s3). [https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/mergetree/#table_engine-mergetree-creating-a-table] [https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/mergetree/#mergetree-table-ttl] If kudu supports this type of functionarility, maybe it will make the combination of kudu and hdfs smoother, and even integrate into the data lake, compete with iceburg/hudi/delta. I have read [~granthenke]'s blog [Transparent Hierarchical Storage Management with Apache Kudu and Impala|https://blog.cloudera.com/transparent-hierarchical-storage-management-with-apache-kudu-and-impala/], thanks for his detailed description, but I think this feature is another option which release the dependence on computing engines. What's your opinion? was: There is a feature in ClickHouse that allows MergeTree table to move range partitions from local disks to remote storage(hdfs/s3). [https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/mergetree/#table_engine-mergetree-creating-a-table] [https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/mergetree/#mergetree-table-ttl] If kudu supports this type of functionarility, maybe it will make the combination of kudu and hdfs smoother, and even integrate into the data lake, compete with iceburg/hudi/delta. I have read [~granthenke]'s blog [Transparent Hierarchical Storage Management with Apache Kudu and Impala|https://blog.cloudera.com/transparent-hierarchical-storage-management-with-apache-kudu-and-impala/], thanks for his detailed description, but I think this feature is another option. What's your opinion? > Move outdated range partitions directly to hdfs > --- > > Key: KUDU-3355 > URL: https://issues.apache.org/jira/browse/KUDU-3355 > Project: Kudu > Issue Type: New Feature >Reporter: LiFu He >Priority: Minor > > There is a feature in ClickHouse that allows MergeTree table to move range > partitions from local disks to remote storage(hdfs/s3). > [https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/mergetree/#table_engine-mergetree-creating-a-table] > [https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/mergetree/#mergetree-table-ttl] > If kudu supports this type of functionarility, maybe it will make the > combination of kudu and hdfs smoother, and even integrate into the data lake, > compete with iceburg/hudi/delta. > I have read [~granthenke]'s blog [Transparent Hierarchical Storage Management > with Apache Kudu and > Impala|https://blog.cloudera.com/transparent-hierarchical-storage-management-with-apache-kudu-and-impala/], > thanks for his detailed description, but I think this feature is another > option which release the dependence on computing engines. > What's your opinion? > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (KUDU-3355) Move outdated range partitions directly to hdfs
[ https://issues.apache.org/jira/browse/KUDU-3355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] LiFu He updated KUDU-3355: -- Summary: Move outdated range partitions directly to hdfs (was: Move range partitions that are out of date to hdfs directly) > Move outdated range partitions directly to hdfs > --- > > Key: KUDU-3355 > URL: https://issues.apache.org/jira/browse/KUDU-3355 > Project: Kudu > Issue Type: New Feature >Reporter: LiFu He >Priority: Minor > > There is a feature in ClickHouse that allows MergeTree table to move range > partitions from local disks to remote storage(hdfs/s3). > [https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/mergetree/#table_engine-mergetree-creating-a-table] > [https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/mergetree/#mergetree-table-ttl] > If kudu supports this type of functionarility, maybe it will make the > combination of kudu and hdfs smoother, and even integrate into the data lake, > compete with iceburg/hudi/delta. > I have read [~granthenke]'s blog [Transparent Hierarchical Storage Management > with Apache Kudu and > Impala|https://blog.cloudera.com/transparent-hierarchical-storage-management-with-apache-kudu-and-impala/], > thanks for his detailed description, but I think this feature is another > option. > What's your opinion? > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (KUDU-3355) Move range partitions that are out of date to hdfs directly
LiFu He created KUDU-3355: - Summary: Move range partitions that are out of date to hdfs directly Key: KUDU-3355 URL: https://issues.apache.org/jira/browse/KUDU-3355 Project: Kudu Issue Type: New Feature Reporter: LiFu He There is a feature in ClickHouse that allows MergeTree table to move range partitions from local disks to remote storage(hdfs/s3). [https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/mergetree/#table_engine-mergetree-creating-a-table] [https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/mergetree/#mergetree-table-ttl] If kudu supports this type of functionarility, maybe it will make the combination of kudu and hdfs smoother, and even integrate into the data lake, compete with iceburg/hudi/delta. I have read [~granthenke]'s blog [Transparent Hierarchical Storage Management with Apache Kudu and Impala|https://blog.cloudera.com/transparent-hierarchical-storage-management-with-apache-kudu-and-impala/], thanks for his detailed description, but I think this feature is another option. What's your opinion? -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (KUDU-3125) kudu-master will crash during initialization while kerberos is enabled
[ https://issues.apache.org/jira/browse/KUDU-3125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] LiFu He updated KUDU-3125: -- Description: We met this issue in our test cluster, the ouput info is just like below: k5_mutex_lock: Received error 22 (Invalid argument) kudu-master: ../../../include/k5-thread.h:376: k5_mutex_lock: Assertion `r == 0' failed ... SIGARBORT ... ... Aborted (core dumped) After some analysis, we found that the krb5-lib we are using is 1.16.1 and it seems it has a bug: [https://web.mit.edu/kerberos/krb5-1.16/] So, we switched the krb5-lib to version 1.17.x and the crash was gone. Here I file this jira for the users who *would* meet this issue. was: We met this issue in our test cluster, the ouput info is just like below: k5_mutex_lock: Received error 22 (Invalid argument) kudu-master: ../../../include/k5-thread.h:376: k5_mutex_lock: Assertion `r == 0' failed ... SIGARBORT ... ... Aborted (core dumped) After some analysis, we found that the krb5-lib we are using is 1.16.1 and it seems it has a bug: [https://web.mit.edu/kerberos/krb5-1.16/] So, we switched the krb5-lib to version 1.17.x and the crash was gone. Here I file this jira for the users who *will* meet this issue. > kudu-master will crash during initialization while kerberos is enabled > -- > > Key: KUDU-3125 > URL: https://issues.apache.org/jira/browse/KUDU-3125 > Project: Kudu > Issue Type: Bug > Components: master >Reporter: LiFu He >Assignee: LiFu He >Priority: Major > Fix For: NA > > > We met this issue in our test cluster, the ouput info is just like below: > > k5_mutex_lock: Received error 22 (Invalid argument) > kudu-master: ../../../include/k5-thread.h:376: k5_mutex_lock: Assertion `r == > 0' failed > ... > SIGARBORT ... > ... > Aborted (core dumped) > > After some analysis, we found that the krb5-lib we are using is 1.16.1 and it > seems it has a bug: [https://web.mit.edu/kerberos/krb5-1.16/] > So, we switched the krb5-lib to version 1.17.x and the crash was gone. > Here I file this jira for the users who *would* meet this issue. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KUDU-3125) kudu-master will crash during initialization while kerberos is enabled
[ https://issues.apache.org/jira/browse/KUDU-3125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] LiFu He resolved KUDU-3125. --- Fix Version/s: NA Assignee: LiFu He Resolution: Won't Fix It's krb5-lib's bug when it is 1.16.1. > kudu-master will crash during initialization while kerberos is enabled > -- > > Key: KUDU-3125 > URL: https://issues.apache.org/jira/browse/KUDU-3125 > Project: Kudu > Issue Type: Bug > Components: master >Reporter: LiFu He >Assignee: LiFu He >Priority: Major > Fix For: NA > > > We met this issue in our test cluster, the ouput info is just like below: > > k5_mutex_lock: Received error 22 (Invalid argument) > kudu-master: ../../../include/k5-thread.h:376: k5_mutex_lock: Assertion `r == > 0' failed > ... > SIGARBORT ... > ... > Aborted (core dumped) > > After some analysis, we found that the krb5-lib we are using is 1.16.1 and it > seems it has a bug: [https://web.mit.edu/kerberos/krb5-1.16/] > So, we switched the krb5-lib to version 1.17.x and the crash was gone. > Here I file this jira for the users who *will* meet this issue. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-3125) kudu-master will crash during initialization while kerberos is enabled
[ https://issues.apache.org/jira/browse/KUDU-3125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] LiFu He updated KUDU-3125: -- Description: We met this issue in our test cluster, the ouput info is just like below: k5_mutex_lock: Received error 22 (Invalid argument) kudu-master: ../../../include/k5-thread.h:376: k5_mutex_lock: Assertion `r == 0' failed ... SIGARBORT ... ... Aborted (core dumped) After some analysis, we found that the krb5-lib we are using is 1.16.1 and it seems it has a bug: [https://web.mit.edu/kerberos/krb5-1.16/] So, we switched the krb5-lib to version 1.17.x and the crash was gone. Here I file this jira for the users who *will* meet this issue. was: We met this issue in our test cluster, the ouput info is just like below: k5_mutex_lock: Received error 22 (Invalid argument) kudu-master: ../../../include/k5-thread.h:376: k5_mutex_lock: Assertion `r == 0' failed ... *** SIGARBORT ... ... Aborted (core dumped) After some analysis, we found that the krb5-lib we are using is 1.16.1 and it seems it has a bug: [https://web.mit.edu/kerberos/krb5-1.16/] So, we switched the krb5-lib to version 1.17.x and the crash was gone. Here I file this jira for the users who *will* meet this issue. > kudu-master will crash during initialization while kerberos is enabled > -- > > Key: KUDU-3125 > URL: https://issues.apache.org/jira/browse/KUDU-3125 > Project: Kudu > Issue Type: Bug > Components: master >Reporter: LiFu He >Priority: Major > > We met this issue in our test cluster, the ouput info is just like below: > > k5_mutex_lock: Received error 22 (Invalid argument) > kudu-master: ../../../include/k5-thread.h:376: k5_mutex_lock: Assertion `r == > 0' failed > ... > SIGARBORT ... > ... > Aborted (core dumped) > > After some analysis, we found that the krb5-lib we are using is 1.16.1 and it > seems it has a bug: [https://web.mit.edu/kerberos/krb5-1.16/] > So, we switched the krb5-lib to version 1.17.x and the crash was gone. > Here I file this jira for the users who *will* meet this issue. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KUDU-3125) kudu-master will crash during initialization while kerberos is enabled
LiFu He created KUDU-3125: - Summary: kudu-master will crash during initialization while kerberos is enabled Key: KUDU-3125 URL: https://issues.apache.org/jira/browse/KUDU-3125 Project: Kudu Issue Type: Bug Components: master Reporter: LiFu He We met this issue in our test cluster, the ouput info is just like below: k5_mutex_lock: Received error 22 (Invalid argument) kudu-master: ../../../include/k5-thread.h:376: k5_mutex_lock: Assertion `r == 0' failed ... *** SIGARBORT ... ... Aborted (core dumped) After some analysis, we found that the krb5-lib we are using is 1.16.1 and it seems it has a bug: [https://web.mit.edu/kerberos/krb5-1.16/] So, we switched the krb5-lib to version 1.17.x and the crash was gone. Here I file this jira for the users who *will* meet this issue. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KUDU-3110) tserver data folder too large
[ https://issues.apache.org/jira/browse/KUDU-3110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17090420#comment-17090420 ] LiFu He commented on KUDU-3110: --- Are there many updates? How about '[–tablet_history_max_age_sec|[https://kudu.apache.org/releases/1.7.1/docs/configuration_reference.html#kudu-tserver_tablet_history_max_age_sec]]' ? In addition, use this command to check the data info if possible: [https://kudu.apache.org/releases/1.7.1/docs/configuration_reference.html#kudu-tserver_tablet_history_max_age_sec] > tserver data folder too large > - > > Key: KUDU-3110 > URL: https://issues.apache.org/jira/browse/KUDU-3110 > Project: Kudu > Issue Type: Bug > Components: tserver >Affects Versions: 1.7.1 >Reporter: SeaAndHill >Priority: Critical > Attachments: kudu use disk.png > > > there is about 100,000 rows in one table , the kudu tserver data directory > use 50G -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KUDU-1994) Automatically Create New Range Partitions When Needed
[
https://issues.apache.org/jira/browse/KUDU-1994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17090400#comment-17090400
]
LiFu He commented on KUDU-1994:
---
Yes, go ahead : )
> Automatically Create New Range Partitions When Needed
> -
>
> Key: KUDU-1994
> URL: https://issues.apache.org/jira/browse/KUDU-1994
> Project: Kudu
> Issue Type: Improvement
>Affects Versions: 1.3.0
>Reporter: Alan Jackoway
>Assignee: Thomas D'Silva
>Priority: Major
> Labels: roadmap-candidate
>
> We have a few Kudu tables where we use a range-partitioned timestamp as part
> of the key. The intention of this is to keep data locality for data that is
> likely to be scanned together, such as events in a timeseries.
> Currently we create these with a partitions that look like this:
> {noformat}
> RANGE (ts) (
> PARTITION 0 <= VALUES < 142008840,
> PARTITION 142008840 <= VALUES < 142786080,
> PARTITION 142786080 <= VALUES < 143572320,
> PARTITION 143572320 <= VALUES < 144367200,
> PARTITION 144367200 <= VALUES < 145162440,
> PARTITION 145162440 <= VALUES < 145948320,
> PARTITION 145948320 <= VALUES < 146734560,
> PARTITION 146734560 <= VALUES < 147529440,
> PARTITION 147529440 <= VALUES < 148324680,
> PARTITION 148324680 <= VALUES < 149103360,
> PARTITION 149103360 <= VALUES < 149889600,
> PARTITION 149889600 <= VALUES < 150684480
> )
> {noformat}
> The problem is that as time goes on we have to choose to either create empty
> partitions in advance of when we are writing data or risk forgetting to
> create a partition and having writes of new data fail.
> Ideally, Kudu would have a way to indicate the size of the partitions (in
> this example 3 months converted to milliseconds) and then automatically
> create new partitions when new data comes in that needs the partition.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (KUDU-1994) Automatically Create New Range Partitions When Needed
[
https://issues.apache.org/jira/browse/KUDU-1994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
LiFu He reassigned KUDU-1994:
-
Assignee: Thomas D'Silva (was: LiFu He)
> Automatically Create New Range Partitions When Needed
> -
>
> Key: KUDU-1994
> URL: https://issues.apache.org/jira/browse/KUDU-1994
> Project: Kudu
> Issue Type: Improvement
>Affects Versions: 1.3.0
>Reporter: Alan Jackoway
>Assignee: Thomas D'Silva
>Priority: Major
> Labels: roadmap-candidate
>
> We have a few Kudu tables where we use a range-partitioned timestamp as part
> of the key. The intention of this is to keep data locality for data that is
> likely to be scanned together, such as events in a timeseries.
> Currently we create these with a partitions that look like this:
> {noformat}
> RANGE (ts) (
> PARTITION 0 <= VALUES < 142008840,
> PARTITION 142008840 <= VALUES < 142786080,
> PARTITION 142786080 <= VALUES < 143572320,
> PARTITION 143572320 <= VALUES < 144367200,
> PARTITION 144367200 <= VALUES < 145162440,
> PARTITION 145162440 <= VALUES < 145948320,
> PARTITION 145948320 <= VALUES < 146734560,
> PARTITION 146734560 <= VALUES < 147529440,
> PARTITION 147529440 <= VALUES < 148324680,
> PARTITION 148324680 <= VALUES < 149103360,
> PARTITION 149103360 <= VALUES < 149889600,
> PARTITION 149889600 <= VALUES < 150684480
> )
> {noformat}
> The problem is that as time goes on we have to choose to either create empty
> partitions in advance of when we are writing data or risk forgetting to
> create a partition and having writes of new data fail.
> Ideally, Kudu would have a way to indicate the size of the partitions (in
> this example 3 months converted to milliseconds) and then automatically
> create new partitions when new data comes in that needs the partition.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (KUDU-3070) allow skip open block manager in cli cmeta rewrite_raft_config operation
[
https://issues.apache.org/jira/browse/KUDU-3070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17054713#comment-17054713
]
LiFu He edited comment on KUDU-3070 at 3/9/20, 7:27 AM:
Interesting. In our cases, we will use ['change_config
move_replica'|https://kudu.apache.org/docs/command_line_tools_reference.html#change_config-move_replica]
to move the tablet from one tserver to another in the same cluster, and use
'insert into ... select from ...' to move the whole table(tablets) from one
cluster to another.
was (Author: helifu):
Interesting. In our cases, we will use '[change_config move_replica|and use
'insert into ... select from ...' to move the table(tablets) from one cluster
to another.]' to move the tablet from one tserver to another in the same
cluster, and use 'insert into ... select from ...' to move the whole
table(tablets) from one cluster to another.
> allow skip open block manager in cli cmeta rewrite_raft_config operation
>
>
> Key: KUDU-3070
> URL: https://issues.apache.org/jira/browse/KUDU-3070
> Project: Kudu
> Issue Type: New Feature
> Components: CLI
>Reporter: wangningito
>Priority: Minor
>
> I'm in a bigdata company which served over 1000+ company, we adopted kudu as
> main or auxiliary storage engine, some of our customers are just small
> startups, they had a lot of data but too much nodes are expensive to them.
> So some of cases are based on: few nodes, much data and maybe not compacted
> well data.
> In our scenarios, there exists some migration cases
> # from standalone tserver to another standalone tserver
> # from 3 nodes tserver cluster to another 3 nodes tserver
> In the past, we have to do something like this
> {code:java}
> // First, download tablet data via kudu local_replica copy_from_remote
> // then rewrite all the raft info for each tablet
> echo ${tablet_id_list} | xargs -i kudu local_replica cmeta
> rewrite_raft_config {} PEER_INFO -fs_data_dirs=xxx -fs_wal_dir=yyy{code}
> Download data via copy_from_remote is blazing fast.
> However sometimes it takes us a lot of time to rewrite raft info of all
> tablet, 30s - 60s per tablet as I witnessed. Sometimes it could take more
> time if the data were not fully compacted. So sometimes it take us 2 hours to
> download tablet data, but 6 hours to rewrite meta.
> I noticed some code fragment in RewriteRaftConfig function
> {code:java}
> FsManager fs_manager(env, FsManagerOpts());
> RETURN_NOT_OK(fs_manager.Open());{code}
> This means I have to open the fs_data_dirs and fs_wal_dir 100 times if I want
> to rewrite raft of 100 tablets.
> To saving the overhead of each operation, we can just skip opening block
> manager for rewrite_raft_config, cause all the operations only happened on
> meta files.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (KUDU-3070) allow skip open block manager in cli cmeta rewrite_raft_config operation
[
https://issues.apache.org/jira/browse/KUDU-3070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17054713#comment-17054713
]
LiFu He edited comment on KUDU-3070 at 3/9/20, 7:26 AM:
Interesting. In our cases, we will use '[change_config move_replica|and use
'insert into ... select from ...' to move the table(tablets) from one cluster
to another.]' to move the tablet from one tserver to another in the same
cluster, and use 'insert into ... select from ...' to move the whole
table(tablets) from one cluster to another.
was (Author: helifu):
Interesting. In our cases, we will use 'change_config move_replica' to move the
tablet from one tserver to another in the same cluster, and use 'insert into
... select from ...' to move the whole table(tablets) from one cluster to
another.
> allow skip open block manager in cli cmeta rewrite_raft_config operation
>
>
> Key: KUDU-3070
> URL: https://issues.apache.org/jira/browse/KUDU-3070
> Project: Kudu
> Issue Type: New Feature
> Components: CLI
>Reporter: wangningito
>Priority: Minor
>
> I'm in a bigdata company which served over 1000+ company, we adopted kudu as
> main or auxiliary storage engine, some of our customers are just small
> startups, they had a lot of data but too much nodes are expensive to them.
> So some of cases are based on: few nodes, much data and maybe not compacted
> well data.
> In our scenarios, there exists some migration cases
> # from standalone tserver to another standalone tserver
> # from 3 nodes tserver cluster to another 3 nodes tserver
> In the past, we have to do something like this
> {code:java}
> // First, download tablet data via kudu local_replica copy_from_remote
> // then rewrite all the raft info for each tablet
> echo ${tablet_id_list} | xargs -i kudu local_replica cmeta
> rewrite_raft_config {} PEER_INFO -fs_data_dirs=xxx -fs_wal_dir=yyy{code}
> Download data via copy_from_remote is blazing fast.
> However sometimes it takes us a lot of time to rewrite raft info of all
> tablet, 30s - 60s per tablet as I witnessed. Sometimes it could take more
> time if the data were not fully compacted. So sometimes it take us 2 hours to
> download tablet data, but 6 hours to rewrite meta.
> I noticed some code fragment in RewriteRaftConfig function
> {code:java}
> FsManager fs_manager(env, FsManagerOpts());
> RETURN_NOT_OK(fs_manager.Open());{code}
> This means I have to open the fs_data_dirs and fs_wal_dir 100 times if I want
> to rewrite raft of 100 tablets.
> To saving the overhead of each operation, we can just skip opening block
> manager for rewrite_raft_config, cause all the operations only happened on
> meta files.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KUDU-3070) allow skip open block manager in cli cmeta rewrite_raft_config operation
[
https://issues.apache.org/jira/browse/KUDU-3070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17054713#comment-17054713
]
LiFu He commented on KUDU-3070:
---
Interesting. In our cases, we will use 'change_config move_replica' to move the
tablet from one tserver to another in the same cluster, and use 'insert into
... select from ...' to move the whole table(tablets) from one cluster to
another.
> allow skip open block manager in cli cmeta rewrite_raft_config operation
>
>
> Key: KUDU-3070
> URL: https://issues.apache.org/jira/browse/KUDU-3070
> Project: Kudu
> Issue Type: New Feature
> Components: CLI
>Reporter: wangningito
>Priority: Minor
>
> I'm in a bigdata company which served over 1000+ company, we adopted kudu as
> main or auxiliary storage engine, some of our customers are just small
> startups, they had a lot of data but too much nodes are expensive to them.
> So some of cases are based on: few nodes, much data and maybe not compacted
> well data.
> In our scenarios, there exists some migration cases
> # from standalone tserver to another standalone tserver
> # from 3 nodes tserver cluster to another 3 nodes tserver
> In the past, we have to do something like this
> {code:java}
> // First, download tablet data via kudu local_replica copy_from_remote
> // then rewrite all the raft info for each tablet
> echo ${tablet_id_list} | xargs -i kudu local_replica cmeta
> rewrite_raft_config {} PEER_INFO -fs_data_dirs=xxx -fs_wal_dir=yyy{code}
> Download data via copy_from_remote is blazing fast.
> However sometimes it takes us a lot of time to rewrite raft info of all
> tablet, 30s - 60s per tablet as I witnessed. Sometimes it could take more
> time if the data were not fully compacted. So sometimes it take us 2 hours to
> download tablet data, but 6 hours to rewrite meta.
> I noticed some code fragment in RewriteRaftConfig function
> {code:java}
> FsManager fs_manager(env, FsManagerOpts());
> RETURN_NOT_OK(fs_manager.Open());{code}
> This means I have to open the fs_data_dirs and fs_wal_dir 100 times if I want
> to rewrite raft of 100 tablets.
> To saving the overhead of each operation, we can just skip opening block
> manager for rewrite_raft_config, cause all the operations only happened on
> meta files.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KUDU-2986) Incorrect value for the 'live_row_count' metric with pre-1.11.0 tables
[ https://issues.apache.org/jira/browse/KUDU-2986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039729#comment-17039729 ] LiFu He commented on KUDU-2986: --- It seems this jira can be closed. > Incorrect value for the 'live_row_count' metric with pre-1.11.0 tables > -- > > Key: KUDU-2986 > URL: https://issues.apache.org/jira/browse/KUDU-2986 > Project: Kudu > Issue Type: Bug > Components: CLI, client, master, metrics >Affects Versions: 1.11.0 >Reporter: YifanZhang >Assignee: LiFu He >Priority: Major > > When we upgraded the cluster with pre-1.11.0 tables, we got inconsistent > values for the 'live_row_count' metric of these tables: > When visiting masterURL:port/metrics, we got 0 for old tables, and got a > positive integer for a old table with a newly added partition, which is the > count of rows in the newly added partition. > When getting table statistics via `kudu table statistics` CLI tool, we got 0 > for old tables and the old table with a new parition. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KUDU-2986) Incorrect value for the 'live_row_count' metric with pre-1.11.0 tables
[ https://issues.apache.org/jira/browse/KUDU-2986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019394#comment-17019394 ] LiFu He commented on KUDU-2986: --- Here is a confusing use case: # we have a cluster with 1 master(1.12.x), 2 tservers(1.10.x) , and 1 tserver(1.12.x); # create a table with single tablet and RF = 3; # If the leader is on 1.12.x, we can see the statistics, but if the leader is on 1.10.x, we can't see the statistics.:( > Incorrect value for the 'live_row_count' metric with pre-1.11.0 tables > -- > > Key: KUDU-2986 > URL: https://issues.apache.org/jira/browse/KUDU-2986 > Project: Kudu > Issue Type: Bug > Components: CLI, client, master, metrics >Affects Versions: 1.11.0 >Reporter: YifanZhang >Assignee: LiFu He >Priority: Major > > When we upgraded the cluster with pre-1.11.0 tables, we got inconsistent > values for the 'live_row_count' metric of these tables: > When visiting masterURL:port/metrics, we got 0 for old tables, and got a > positive integer for a old table with a newly added partition, which is the > count of rows in the newly added partition. > When getting table statistics via `kudu table statistics` CLI tool, we got 0 > for old tables and the old table with a new parition. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-3044) Failed to start the mini_cluster while the pid is greater than 262144
[
https://issues.apache.org/jira/browse/KUDU-3044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
LiFu He updated KUDU-3044:
--
Description:
I met this issue while doing some tests and below is the output:
{code:java}
// code placeholder
WARNING: Logging before InitGoogleLogging() is written to STDERR
F0120 15:16:07.215484 573686 net_util.cc:485] Check failed: pid < 1 << kPidBits
(573686 vs. 262144) PID 573686 is more than 18 bits wide
*** Check failure stack trace: ***
*** Aborted at 1579504567 (unix time) try "date -d @1579504567" if you are
using GNU date ***
PC: @ 0x7f44b85b2067 gsignal
*** SIGABRT (@0x7d60008c0f6) received by PID 573686 (TID 0x7f44badae880) from
PID 573686; stack trace: ***
@ 0x7f44ba0e2890 (unknown)
@ 0x7f44b85b2067 gsignal
@ 0x7f44b85b3448 abort
@ 0xc2bdf9 google::logging_fail()
@ 0xc2d6ed google::LogMessage::Fail()
@ 0xc2f5bd google::LogMessage::SendToLog()
@ 0xc2d229 google::LogMessage::Flush()
@ 0xc3005f google::LogMessageFatal::~LogMessageFatal()
@ 0x2b1477b kudu::GetBindIpForDaemon()
@ 0xcbc26d kudu::cluster::MiniCluster::ReserveDaemonSocket()
@ 0xcb94f6 kudu::cluster::InternalMiniCluster::StartMasters()
@ 0xcbace0 kudu::cluster::InternalMiniCluster::Start()
@ 0xc20d01 kudu::tserver::TsTabletManagerITest::StartCluster()
@ 0xc15462
kudu::tserver::TsTabletManagerITest_TestTableStats_Test::TestBody()
@ 0x2a6b9c8
testing::internal::HandleExceptionsInMethodIfSupported<>()
@ 0x2a59e92 testing::Test::Run()
@ 0x2a59fd8 testing::TestInfo::Run()
@ 0x2a5a0b5 testing::TestCase::Run()
@ 0x2a609c8 testing::internal::UnitTestImpl::RunAllTests()
@ 0x2a6bed8
testing::internal::HandleExceptionsInMethodIfSupported<>()
@ 0x2a5a18d testing::UnitTest::Run()
@ 0xbad3ba main
@ 0x7f44b859eb45 __libc_start_main
@ 0xc0eb38 (unknown)
Aborted (core dumped)
{code}
was:
I hit this issue while doing some tests and below is the output:
{code:java}
// code placeholder
WARNING: Logging before InitGoogleLogging() is written to STDERR
F0120 15:16:07.215484 573686 net_util.cc:485] Check failed: pid < 1 << kPidBits
(573686 vs. 262144) PID 573686 is more than 18 bits wide
*** Check failure stack trace: ***
*** Aborted at 1579504567 (unix time) try "date -d @1579504567" if you are
using GNU date ***
PC: @ 0x7f44b85b2067 gsignal
*** SIGABRT (@0x7d60008c0f6) received by PID 573686 (TID 0x7f44badae880) from
PID 573686; stack trace: ***
@ 0x7f44ba0e2890 (unknown)
@ 0x7f44b85b2067 gsignal
@ 0x7f44b85b3448 abort
@ 0xc2bdf9 google::logging_fail()
@ 0xc2d6ed google::LogMessage::Fail()
@ 0xc2f5bd google::LogMessage::SendToLog()
@ 0xc2d229 google::LogMessage::Flush()
@ 0xc3005f google::LogMessageFatal::~LogMessageFatal()
@ 0x2b1477b kudu::GetBindIpForDaemon()
@ 0xcbc26d kudu::cluster::MiniCluster::ReserveDaemonSocket()
@ 0xcb94f6 kudu::cluster::InternalMiniCluster::StartMasters()
@ 0xcbace0 kudu::cluster::InternalMiniCluster::Start()
@ 0xc20d01 kudu::tserver::TsTabletManagerITest::StartCluster()
@ 0xc15462
kudu::tserver::TsTabletManagerITest_TestTableStats_Test::TestBody()
@ 0x2a6b9c8
testing::internal::HandleExceptionsInMethodIfSupported<>()
@ 0x2a59e92 testing::Test::Run()
@ 0x2a59fd8 testing::TestInfo::Run()
@ 0x2a5a0b5 testing::TestCase::Run()
@ 0x2a609c8 testing::internal::UnitTestImpl::RunAllTests()
@ 0x2a6bed8
testing::internal::HandleExceptionsInMethodIfSupported<>()
@ 0x2a5a18d testing::UnitTest::Run()
@ 0xbad3ba main
@ 0x7f44b859eb45 __libc_start_main
@ 0xc0eb38 (unknown)
Aborted (core dumped)
{code}
> Failed to start the mini_cluster while the pid is greater than 262144
> -
>
> Key: KUDU-3044
> URL: https://issues.apache.org/jira/browse/KUDU-3044
> Project: Kudu
> Issue Type: Bug
>Reporter: LiFu He
>Priority: Minor
>
> I met this issue while doing some tests and below is the output:
> {code:java}
> // code placeholder
> WARNING: Logging before InitGoogleLogging() is written to STDERR
> F0120 15:16:07.215484 573686 net_util.cc:485] Check failed: pid < 1 <<
> kPidBits (573686 vs. 262144) PID 573686 is more than 18 bits wide
> *** Check failure stack trace: ***
> *** Aborted at 1579504567 (unix time) try "date -d @1579504567" if you are
> using GNU date ***
> PC: @ 0x7f44b85b2067 gsignal
>
[jira] [Created] (KUDU-3044) Failed to start the mini_cluster while the pid is greater than 262144
LiFu He created KUDU-3044: - Summary: Failed to start the mini_cluster while the pid is greater than 262144 Key: KUDU-3044 URL: https://issues.apache.org/jira/browse/KUDU-3044 Project: Kudu Issue Type: Bug Reporter: LiFu He I hit this issue while doing some tests and below is the output: WARNING: Logging before InitGoogleLogging() is written to STDERR F0120 15:16:07.215484 573686 net_util.cc:485] Check failed: pid < 1 << kPidBits (573686 vs. 262144) PID 573686 is more than 18 bits wide * ** *** Check failure stack trace: *** *** Aborted at 1579504567 (unix time) try "date -d @1579504567" if you are using GNU date *** PC: @ 0x7f44b85b2067 gsignal *** SIGABRT (@0x7d60008c0f6) received by PID 573686 (TID 0x7f44badae880) from PID 573686; stack trace: *** @ 0x7f44ba0e2890 (unknown) @ 0x7f44b85b2067 gsignal @ 0x7f44b85b3448 abort @ 0xc2bdf9 google::logging_fail() @ 0xc2d6ed google::LogMessage::Fail() @ 0xc2f5bd google::LogMessage::SendToLog() @ 0xc2d229 google::LogMessage::Flush() @ 0xc3005f google::LogMessageFatal::~LogMessageFatal() @ 0x2b1477b kudu::GetBindIpForDaemon() @ 0xcbc26d kudu::cluster::MiniCluster::ReserveDaemonSocket() @ 0xcb94f6 kudu::cluster::InternalMiniCluster::StartMasters() @ 0xcbace0 kudu::cluster::InternalMiniCluster::Start() @ 0xc20d01 kudu::tserver::TsTabletManagerITest::StartCluster() @ 0xc15462 kudu::tserver::TsTabletManagerITest_TestTableStats_Test::TestBody() @ 0x2a6b9c8 testing::internal::HandleExceptionsInMethodIfSupported<>() @ 0x2a59e92 testing::Test::Run() @ 0x2a59fd8 testing::TestInfo::Run() @ 0x2a5a0b5 testing::TestCase::Run() @ 0x2a609c8 testing::internal::UnitTestImpl::RunAllTests() @ 0x2a6bed8 testing::internal::HandleExceptionsInMethodIfSupported<>() @ 0x2a5a18d testing::UnitTest::Run() @ 0xbad3ba main @ 0x7f44b859eb45 __libc_start_main @ 0xc0eb38 (unknown) Aborted (core dumped) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KUDU-2986) Incorrect value for the 'live_row_count' metric with pre-1.11.0 tables
[ https://issues.apache.org/jira/browse/KUDU-2986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] LiFu He reassigned KUDU-2986: - Assignee: LiFu He > Incorrect value for the 'live_row_count' metric with pre-1.11.0 tables > -- > > Key: KUDU-2986 > URL: https://issues.apache.org/jira/browse/KUDU-2986 > Project: Kudu > Issue Type: Bug > Components: CLI, client, master, metrics >Affects Versions: 1.11.0 >Reporter: YifanZhang >Assignee: LiFu He >Priority: Major > > When we upgraded the cluster with pre-1.11.0 tables, we got inconsistent > values for the 'live_row_count' metric of these tables: > When visiting masterURL:port/metrics, we got 0 for old tables, and got a > positive integer for a old table with a newly added partition, which is the > count of rows in the newly added partition. > When getting table statistics via `kudu table statistics` CLI tool, we got 0 > for old tables and the old table with a new parition. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KUDU-3033) Add min/max values for the non-primary key columns in the metadata of rowsets/datablocks
LiFu He created KUDU-3033: - Summary: Add min/max values for the non-primary key columns in the metadata of rowsets/datablocks Key: KUDU-3033 URL: https://issues.apache.org/jira/browse/KUDU-3033 Project: Kudu Issue Type: New Feature Components: cfile, tablet Reporter: LiFu He It's possible to add min/max values for the non-primary key columns in the metadata of diskrowset/datablock, and then we can skip decoding/evaluating the unnecessary diskrowset/datablock while scanning. Just like the "compute stats" feature on impala, and the only difference is that kudu supports updates. So, the min/max values should be invalid if the columns that have deltas while scanning. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-3013) Race in StopTabletITest.TestStoppedTabletsDontWrite
[
https://issues.apache.org/jira/browse/KUDU-3013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
LiFu He updated KUDU-3013:
--
Description:
I met this issue on Jenkins this morning, and it seems there is a race in
StopTabletITest.TestStoppedTabletsDontWrite.
{code:java}
// code placeholder
TransactionDriver::ApplyTask()Tablet::Stop()
| |
transaction_->Apply() |
| |
tablet->ApplyRowOperations(state()) |
(RESERVED -> APPLYING) |
| |
StartApplying(tx_state); |
|
set_state_unlocked(kStopped);
ApplyRowOperation() |
| |
CheckHasNotBeenStoppedUnlocked()|
(return error since the tablet has been stopped)|
| |
HandleFailure(s)|
| |
transaction_->Finish(Transaction::ABORTED); |
| |
state()->CommitOrAbort(result); |
| |
ReleaseMvccTxn(result); |
| |
mvcc_tx_->Abort(); |
| |
manager_->AbortTransaction(timestamp_); |
| |
if (PREDICT_FALSE(!is_open())) |
| mvcc_.Close();
| |
| open_.store(false);
CHECK_EQ(old_state, RESERVED) |
(ASSERT failed)
{code}
was:
I met this issue on jekins this morning, and it seems there is a race in
StopTabletITest.TestStoppedTabletsDontWrite.
{code:java}
// code placeholder
TransactionDriver::ApplyTask()Tablet::Stop()
| |
transaction_->Apply() |
| |
tablet->ApplyRowOperations(state()) |
(RESERVED -> APPLYING) |
| |
StartApplying(tx_state); |
|
set_state_unlocked(kStopped);
ApplyRowOperation() |
| |
CheckHasNotBeenStoppedUnlocked()|
(return error since the tablet has been stopped)|
| |
HandleFailure(s)|
| |
transaction_->Finish(Transaction::ABORTED); |
| |
state()->CommitOrAbort(result); |
| |
ReleaseMvccTxn(result); |
| |
mvcc_tx_->Abort(); |
| |
manager_->AbortTransaction(timestamp_); |
| |
if (PREDICT_FALSE(!is_open())) |
| mvcc_.Close();
| |
| open_.store(false);
CHECK_EQ(old_state, RESERVED) |
(ASSERT failed)
{code}
> Race in StopTabletITest.TestStoppedTabletsDontWrite
> ---
>
> Key: KUDU-3013
> URL: https://issues.apache.org/jira/browse/KUDU-3013
> Project: Kudu
>
[jira] [Updated] (KUDU-3013) Race in StopTabletITest.TestStoppedTabletsDontWrite
[
https://issues.apache.org/jira/browse/KUDU-3013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
LiFu He updated KUDU-3013:
--
Description:
I met this issue on jekins this morning, and it seems there is a race in
StopTabletITest.TestStoppedTabletsDontWrite.
{code:java}
// code placeholder
TransactionDriver::ApplyTask()Tablet::Stop()
| |
transaction_->Apply() |
| |
tablet->ApplyRowOperations(state()) |
(RESERVED -> APPLYING) |
| |
StartApplying(tx_state); |
|
set_state_unlocked(kStopped);
ApplyRowOperation() |
| |
CheckHasNotBeenStoppedUnlocked()|
(return error since the tablet has been stopped)|
| |
HandleFailure(s)|
| |
transaction_->Finish(Transaction::ABORTED); |
| |
state()->CommitOrAbort(result); |
| |
ReleaseMvccTxn(result); |
| |
mvcc_tx_->Abort(); |
| |
manager_->AbortTransaction(timestamp_); |
| |
if (PREDICT_FALSE(!is_open())) |
| mvcc_.Close();
| |
| open_.store(false);
CHECK_EQ(old_state, RESERVED) |
(ASSERT failed)
{code}
was:
I met this issue this morning, and it seems there is a race in
StopTabletITest.TestStoppedTabletsDontWrite.
{code:java}
// code placeholder
TransactionDriver::ApplyTask()Tablet::Stop()
| |
transaction_->Apply() |
| |
tablet->ApplyRowOperations(state()) |
(RESERVED -> APPLYING) |
| |
StartApplying(tx_state); |
|
set_state_unlocked(kStopped);
ApplyRowOperation() |
| |
CheckHasNotBeenStoppedUnlocked()|
(return error since the tablet has been stopped)|
| |
HandleFailure(s)|
| |
transaction_->Finish(Transaction::ABORTED); |
| |
state()->CommitOrAbort(result); |
| |
ReleaseMvccTxn(result); |
| |
mvcc_tx_->Abort(); |
| |
manager_->AbortTransaction(timestamp_); |
| |
if (PREDICT_FALSE(!is_open())) |
| mvcc_.Close();
| |
| open_.store(false);
CHECK_EQ(old_state, RESERVED) |
(ASSERT failed)
{code}
> Race in StopTabletITest.TestStoppedTabletsDontWrite
> ---
>
> Key: KUDU-3013
> URL: https://issues.apache.org/jira/browse/KUDU-3013
> Project: Kudu
> Issue
[jira] [Created] (KUDU-3013) Race in StopTabletITest.TestStoppedTabletsDontWrite
LiFu He created KUDU-3013:
-
Summary: Race in StopTabletITest.TestStoppedTabletsDontWrite
Key: KUDU-3013
URL: https://issues.apache.org/jira/browse/KUDU-3013
Project: Kudu
Issue Type: Bug
Reporter: LiFu He
Attachments:
jenkins-slave.1575252039.26703.311237e4f4a39e5fea3b175fbf12d3e4aa8674dc.81.0-artifacts.zip
I met this issue this morning, and it seems there is a race in
StopTabletITest.TestStoppedTabletsDontWrite.
{code:java}
// code placeholder
TransactionDriver::ApplyTask()Tablet::Stop()
| |
transaction_->Apply() |
| |
tablet->ApplyRowOperations(state()) |
(RESERVED -> APPLYING) |
| |
StartApplying(tx_state); |
|
set_state_unlocked(kStopped);
ApplyRowOperation() |
| |
CheckHasNotBeenStoppedUnlocked()|
(return error since the tablet has been stopped)|
| |
HandleFailure(s)|
| |
transaction_->Finish(Transaction::ABORTED); |
| |
state()->CommitOrAbort(result); |
| |
ReleaseMvccTxn(result); |
| |
mvcc_tx_->Abort(); |
| |
manager_->AbortTransaction(timestamp_); |
| |
if (PREDICT_FALSE(!is_open())) |
| mvcc_.Close();
| |
| open_.store(false);
CHECK_EQ(old_state, RESERVED) |
(ASSERT failed)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KUDU-3011) Support for smooth maintenance window
LiFu He created KUDU-3011: - Summary: Support for smooth maintenance window Key: KUDU-3011 URL: https://issues.apache.org/jira/browse/KUDU-3011 Project: Kudu Issue Type: New Feature Reporter: LiFu He A scan corresponding to a tablet failure causes the entire SQL to fail on the common query engines, such as Impala. Though we have the fault-tolerant feature by "SetFaultTolerant()", Impala doesn't use it right now since that will make lower throughput. Thus, lots of SQL that are running will fail when we shutdown/reboot/upgrade the tserver. That can be scary. Maybe we can do some improvement in this area, for example, the tablets are not allowed to be scanned after the tserver is in maintenance mode (KUDU-2069). And for the LEADER_ONLY mode scanning, the leader role needs to be shifted from the maintenance tserver. Then we can shutdown the tserver smoothly after all the existing SQL are completed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KUDU-2986) Incorrect value for the 'live_row_count' metric with pre-1.11.0 tables
[ https://issues.apache.org/jira/browse/KUDU-2986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16966486#comment-16966486 ] LiFu He commented on KUDU-2986: --- [~aserbin] any thoughts on the kudu versions we need to support in the mixed deployment? 1.10.x, 1.9.x, 1.8.x ... > Incorrect value for the 'live_row_count' metric with pre-1.11.0 tables > -- > > Key: KUDU-2986 > URL: https://issues.apache.org/jira/browse/KUDU-2986 > Project: Kudu > Issue Type: Bug > Components: CLI, client, master, metrics >Affects Versions: 1.11.0 >Reporter: YifanZhang >Priority: Major > > When we upgraded the cluster with pre-1.11.0 tables, we got inconsistent > values for the 'live_row_count' metric of these tables: > When visiting masterURL:port/metrics, we got 0 for old tables, and got a > positive integer for a old table with a newly added partition, which is the > count of rows in the newly added partition. > When getting table statistics via `kudu table statistics` CLI tool, we got 0 > for old tables and the old table with a new parition. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KUDU-2038) Add b-tree or inverted index on value field
[ https://issues.apache.org/jira/browse/KUDU-2038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16963597#comment-16963597 ] LiFu He commented on KUDU-2038: --- It looks like the Procella from the YouTuBe supports secondary index, and takes the example of Roaring. [Procella|[http://www.vldb.org/pvldb/vol12/p2022-chattopadhyay.pdf]] * 2.2.2 Metadata Storage * 3.2 Data format -> Supports storing inverted indexes. > Add b-tree or inverted index on value field > --- > > Key: KUDU-2038 > URL: https://issues.apache.org/jira/browse/KUDU-2038 > Project: Kudu > Issue Type: Task >Reporter: Yi Guolei >Priority: Major > Labels: roadmap-candidate > > Do we have a plan to add index on any column [not primary column] ? Currently > kudu does not have btree or inverted index on columns. In this case if a > query wants to filter a column then kudu has to scan all datas in all > rowsets. > For example, select * from table where salary > 1 and age < 40, the bloom > filter or min max index will have no effect, kudu has to scan all datas in > all row sets. But if kudu has inverted index, then it will be much faster. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KUDU-2483) Scan tablets with bloom filter
[
https://issues.apache.org/jira/browse/KUDU-2483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955772#comment-16955772
]
LiFu He commented on KUDU-2483:
---
Ah, that was 2 years ago.
No, the launch of this feature was delayed though we had implemented it,
because [~twmarshall] said that the min-max filter was on the way at that time.
Anyway, the test result showed this feature is attractive in some cases.
> Scan tablets with bloom filter
> --
>
> Key: KUDU-2483
> URL: https://issues.apache.org/jira/browse/KUDU-2483
> Project: Kudu
> Issue Type: New Feature
> Components: client
>Reporter: jin xing
>Assignee: Xu Yao
>Priority: Major
> Labels: roadmap-candidate
> Attachments: BloomFilter+Design+Doc.pdf, KUDU-2483,
> image-2018-07-01-23-29-05-517.png
>
>
> Join is really common/popular in Spark SQL, in this JIRA I take broadcast
> join as an example and describe how Kudu's bloom filter can help accelerate
> distributed computing.
> Spark runs broadcast join with below steps:
> 1. When do broadcast join, we have a small table and a big table; Spark will
> read all data from small table to one worker and build a hash table;
> 2. The generated hash table from step 1 is broadcasted to all the workers,
> which will read the splits from big table;
> 3. Workers start fetching and iterating all the splits of big table and see
> if the joining keys exists in the hash table; Only matched joining keys is
> retained.
> From above, step 3 is the heaviest, especially when the worker and split
> storage is not on the same host and bandwith is limited. Actually the cost
> brought by step 3 is not always necessary. Think about below scenario:
> {code:none}
> Small table A
> id name
> 1 Jin
> 6 Xing
> Big table B
> id age
> 1 10
> 2 21
> 3 33
> 4 65
> 5 32
> 6 23
> 7 18
> 8 20
> 9 22
> {code}
> Run query with SQL: *select * from A inner join B on A.id=B.id*
> It's pretty straight that we don't need to fetch all the data from Table B,
> because the number of matched keys is really small;
> I propose to use small table to build a bloom filter(BF) and use the
> generated BF as a predicate/filter to fetch data from big table, thus:
> 1. Much traffic/bandwith is saved.
> 2. Less data to processe by worker
> Broadcast join is just an example, other types of join will also benefit if
> we scan with a BF
> In a nutshell, I think Kudu can provide an iterface, by which user can scan
> data with bloom filters
>
> Here I want add some statistics for Spark-Kudu integration with/without
> BloomBloomFilter.
> In our product environment the bandwidth of each executor is 50M bps.
> We do inner join with two tables – – one is large and another one is
> comparatively small.
> In Spark, inner join can be implemented as SortMergeJoin or
> BroadcastHashJoin, we implemented the corresponding operators with
> BloomFilter as SortMergeBloomFilterJoin and BroadcastBloomFilterJoin.
> The hash table of BloomFilter is configured as 32M.
> I show statistics as below:
> ||Records of Table A||Records of Table B||Join Operator||Executor Time||
> |400 thousand|14 billion|SortMergeJoin|760 seconds|
> |400 thousand|14 billion|BroadcastHashJoin|376s|
> |400 thousand|14 billion|BroadcastBloomFilterJoin|21s|
> |2 million|14 billion|SortMergeJoin|707s|
> |2 million|14 billion|BroadcastHashJoin|329s|
> |2 million|14 billion|SortMergeBloomFilterJoin|75s|
> |2 million|14 billion|BroadcastBloomFilterJoin|35s|
> As we can see, it benefit a lot from BloomFilter-PushDown.
> I want to take this jira as a umbrella and my workmates will submit
> following sub-task/pr.
> It will be great if some can take more look at this and share some comments.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KUDU-2975) Spread WAL across multiple data directories
LiFu He created KUDU-2975: - Summary: Spread WAL across multiple data directories Key: KUDU-2975 URL: https://issues.apache.org/jira/browse/KUDU-2975 Project: Kudu Issue Type: New Feature Components: fs, tablet, tserver Reporter: LiFu He Attachments: network.png, tserver-WARNING.png, util.png Recently, we deployed a new kudu cluster and every node has 12 SSD. Then, we created a big table and loaded data to it through flink. We noticed that the util of one SSD which is used to store WAL is 100% but others are free. So, we suggest to spread WAL across multiple data directories. -- This message was sent by Atlassian Jira (v8.3.4#803005)
