[jira] [Resolved] (KUDU-2995) Tuning kudu write performance
[
https://issues.apache.org/jira/browse/KUDU-2995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adar Dembo resolved KUDU-2995.
--
Fix Version/s: n/a
Resolution: Information Provided
For problems like these, please e-mail [the Kudu user mailing
list|mailto:u...@kudu.apache.org] or post in the [Kudu upstream
Slack|https://getkudu-slack.herokuapp.com/]. Jiras should be opened for
specific defects or feature requests in the Kudu product.
> Tuning kudu write performance
> -
>
> Key: KUDU-2995
> URL: https://issues.apache.org/jira/browse/KUDU-2995
> Project: Kudu
> Issue Type: Test
> Components: client, java
>Affects Versions: 1.7.0
> Environment: kudu 1.7.0-cdh5.16.2
>Reporter: wxmimperio
>Priority: Major
> Labels: performance
> Fix For: n/a
>
>
> Kudu version： kudu 1.7.0-cdh5.16.2
> System memary pre node：256G
> 4 SSDs per machine：512G
> Three Master nodes and three Tserver nodes.
> {code:java}
> // Master config
> --fs_wal_dir=/mnt/disk1/kudu/var/wal
> --fs_data_dirs=/mnt/disk1/kudu/var/data,/mnt/disk2/kudu/var/data,/mnt/disk3/kudu/var/data,/mnt/disk4/kudu/var/data
> --fs_metadata_dir=/mnt/disk1/kudu/var/metadata
> --log_dir=/mnt/disk1/kudu/var/logs
> --master_addresses=
> --maintenance_manager_num_threads=2
> --block_cache_capacity_mb=6144
> --memory_limit_hard_bytes=34359738368
> --max_log_size=40
> // Tserver config
> --fs_wal_dir=/mnt/disk1/kudu/var/wal
> --fs_data_dirs=/mnt/disk1/kudu/var/data,/mnt/disk2/kudu/var/data,/mnt/disk3/kudu/var/data,/mnt/disk4/kudu/var/data
> --fs_metadata_dir=/mnt/disk1/kudu/var/metadata
> --log_dir=/mnt/disk1/kudu/var/logs
> --tserver_master_addrs=
> --block_cache_capacity_mb=6144
> --memory_limit_hard_bytes=34359738368
> --max_log_size=40
> // Table schema
> // _key is UUID for each msg
> // event_time is data time
> // Schema has only 15 columns
> // Single message does not exceed 100Bytes
> HASH (_key) PARTITIONS 3,
> RANGE (event_time) (
> PARTITION 2019-10-31T16:00:00.00Z <= VALUES <
> 2019-11-30T16:00:00.00Z
> )
> {code}
> I write a project to write data to kudu.
> Whether manual or automatic flush mode write speed is only 6MB/s.
> I think SSD should be more than this speed, and the network and memory have
> not reached the bottleneck.
> Is this the normal level of kudu writing? How to tuning?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KUDU-2989) SASL server fails when FQDN is greater than 63 characters long
[ https://issues.apache.org/jira/browse/KUDU-2989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972090#comment-16972090 ] ASF subversion and git services commented on KUDU-2989: --- Commit 3b84cf781aa5e0f7c2c8c4f60a21d82ad7fffb3e in kudu's branch refs/heads/branch-1.11.x from Todd Lipcon [ https://gitbox.apache.org/repos/asf?p=kudu.git;h=3b84cf7 ] KUDU-2989. Work around SASL bug when FQDN is >=64 characters This adds a workaround for an upstream SASL bug which is triggered when the FQDN has more than 64 characters. In this case, SASL would truncate the FQDN and not be able to find the relevant keytab. The workaround simply uses our own code to determine the FQDN. Change-Id: I4898814f2f7ab87151798336414dde7078d28a4a Reviewed-on: http://gerrit.cloudera.org:8080/14609 Reviewed-by: Anurag Mantripragada Reviewed-by: Adar Dembo Tested-by: Kudu Jenkins (cherry picked from commit 111b13775193820b3e3551368fe00a8f00387007) Reviewed-on: http://gerrit.cloudera.org:8080/14687 Reviewed-by: Grant Henke Tested-by: Alexey Serbin > SASL server fails when FQDN is greater than 63 characters long > -- > > Key: KUDU-2989 > URL: https://issues.apache.org/jira/browse/KUDU-2989 > Project: Kudu > Issue Type: Bug > Components: rpc, security >Affects Versions: 1.10.0 >Reporter: Todd Lipcon >Priority: Critical > > Currently, on the server side, Kudu doesn't explicitly pass the host's FQDN > into the SASL library. Due to an upstream SASL bug > (https://github.com/cyrusimap/cyrus-sasl/issues/583) the FQDN gets truncated > when trying to determine the server's principal, in the case that the > server's fQDN is longer than 64 characters. > This results in startup failures where the preflight checks fail due to not > finding the appropriate keytab entry (after searching for a truncated host > name) > To work around this, we should use our own code to compute the FQDN. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KUDU-2987) Intra location rebalance will crash in special case
[
https://issues.apache.org/jira/browse/KUDU-2987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972089#comment-16972089
]
ASF subversion and git services commented on KUDU-2987:
---
Commit a8733419fd43c488586172f71cab0892581146e8 in kudu's branch
refs/heads/branch-1.11.x from triplesheep
[ https://gitbox.apache.org/repos/asf?p=kudu.git;h=a873341 ]
KUDU-2987 Intra location rebalance crashes in special case.
The crash manifested itself in cases where a Kudu cluster
had a location that didn't host even a single replica of
a tablet.
Change-Id: Iea39472e55178fca688b390249432a0f7fefaaba
Reviewed-on: http://gerrit.cloudera.org:8080/14608
Tested-by: Alexey Serbin
Reviewed-by: Alexey Serbin
(cherry picked from commit 0fea1cb8ede852a87efc422b394ffe8d1e89bc6c)
Reviewed-on: http://gerrit.cloudera.org:8080/14686
Tested-by: Kudu Jenkins
Reviewed-by: Grant Henke
> Intra location rebalance will crash in special case
> ---
>
> Key: KUDU-2987
> URL: https://issues.apache.org/jira/browse/KUDU-2987
> Project: Kudu
> Issue Type: Bug
>Reporter: ZhangYao
>Assignee: ZhangYao
>Priority: Major
>
> Recently I am doing POC about rebalance and I get core when running intra
> location rebalance.
> Here is the log:
> {code:java}
> I2019-10-30 20:02:17.843044 40915 rebalancer_tool.cc:225] running rebalancer
> within location '/location/2044'
> F2019-10-30 20:02:17.884591 40915 map-util.h:109] Check failed: it !=
> collection.end() Map key not found: a9119004b2d24f42a1acf09d142565fb
> *** Check failure stack trace: ***
> @ 0x111a75d google::LogMessage::Fail()
> @ 0x111c6d3 google::LogMessage::SendToLog()
> @ 0x111a2b9 google::LogMessage::Flush()
> @ 0x111d0ef google::LogMessageFatal::~LogMessageFatal()
> @ 0xe26da7 FindOrDie<>()
> @ 0xe1f204
> kudu::tools::RebalancerTool::AlgoBasedRunner::GetNextMovesImpl()
> @ 0xe162e0
> kudu::tools::RebalancerTool::BaseRunner::GetNextMoves()
> @ 0xe15bf5 kudu::tools::RebalancerTool::RunWith()
> @ 0xe1db0e kudu::tools::RebalancerTool::Run()
> @ 0xb6fea1 kudu::tools::(anonymous namespace)::RunRebalance()
> @ 0xb70e14 std::_Function_handler<>::_M_invoke()
> @ 0x11714a2 kudu::tools::Action::Run()
> @ 0xc00587 kudu::tools::DispatchCommand()
> @ 0xc00f4b kudu::tools::RunTool()
> @ 0xb0fd6d main
> @ 0x7f37086a4b15 __libc_start_main
> @ 0xb6b399 (unknown)
> {code}
> I found it may be the problem in
> {{RebalancerTool::AlgoBasedRunner::GetNextMovesImpl}} when building
> extra_info_by_tablet_id, it check that the table id in tablet must occur in
> table info. But when we build ClusterRawInfo in
> {{RebalancerTool::KsckResultsToClusterRawInfo}} we only collect the table
> occurs in location but all tablets in cluster.
> This problem will occur when the location doesn't have replica for all
> table. When location is far more than table's replica it will happen.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KUDU-2162) Expose stats about scan filters
[ https://issues.apache.org/jira/browse/KUDU-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas D'Silva updated KUDU-2162: - Code Review: https://gerrit.cloudera.org/c/8375/ (was: http://gerrit.cloudera.org:8080/14652) > Expose stats about scan filters > --- > > Key: KUDU-2162 > URL: https://issues.apache.org/jira/browse/KUDU-2162 > Project: Kudu > Issue Type: Improvement > Components: client >Reporter: Thomas Tauber-Marshall >Assignee: Thomas D'Silva >Priority: Major > > Impala is working on implementing runtime filters that get pushed down into > Kudu using KuduScanner::AddConjunctPredicate() > It would be useful for perf analysis and debugging to be able to include info > in Impala's runtime profile about the effectiveness of the filters, eg. > number of rows that are filtered. > This would probably require at least two counters: > - # of blocks that are entirely skipped > - # of rows that are filtered from blocks that aren't entirely skipped -- This message was sent by Atlassian Jira (v8.3.4#803005)
