Re: Any plans for "Aggregation Push down" or integrating Impala + Kudu more tightly?

our service but we've made Java Program to serve our service. I wanted to know same idea can be applied to our client program. Thanks. Regards, Jason 2017-06-30 3:53 GMT+09:00 Todd Lipcon : > Hey Jason, > > Answers inline below > > On Thu, Jun 29, 2017 at 2:52 AM, Jas

Any plans for "Aggregation Push down" or integrating Impala + Kudu more tightly?

Hi, Q1. After reading Druid vs Kudu , I wondered Druid has aggregation push down. *Druid includes its own query layer that allows it to push down > aggregations and computations directly to data nodes for faster query > processing. *

Re: What does "Failed RPC negotiation" in kudu-tserver.WARNING

Hi Jean-Daniel, Todd, and Alexey Thank your for the replies. Recently, I've experienced many issues but successfully resolved them with your helps. I really appreciate it. Regards, Jason

What does "Failed RPC negotiation" in kudu-tserver.WARNING

Hi. I'm using Apache Kudu 1.4.0 and Impala 2.8.0 I'm getting an error message like this in kudu-tserver.WARNING W0617 08:10:20.838203 57169 negotiation.cc:303] Failed RPC negotiation. Trace: 0617 08:10:20.698376 (+ 0us) reactor.cc:449] Submitting negotiation task for server connection from

Re: tserver died by clock unsync.

se your > internal NTP server(s) as a source. Also, it would make sense to take a > look at some 'NTP best practice' guides you could find elsewhere on the > Internet -- hopefully, you could find some ideas how to tailor those for > you case. > > Hope this helps. > >

Re: tserver died by clock unsync.

hub.com/apache/kudu/blob/1.4.0/src/kudu/server/hybrid_clock.cc#L90> returns TIME_ERROR Could anyone can help me? Regards, Jason 2017-06-15 12:40 GMT+09:00 Jason Heo : > Hi, > > I'm using Apache Kudu 1.4.0 > > Yesterday, 6 tservers die at the same time. Following

Re: I got an "authentication token expired" error.

secure Kudu cluster. In this case you don't need to restart your Java > clients once you restarted Kudu server-side components. > > Hope this helps. > > > Kind regards, > > Alexey > > > > On 6/14/17 2:44 AM, Jason Heo wrote: > >> Hi. >> >> I&

tserver died by clock unsync.

Hi, I'm using Apache Kudu 1.4.0 Yesterday, 6 tservers die at the same time. Following message is logged for each tserver. F0614 14:58:32.868551 111454 hybrid_clock.cc:227] Couldn't get the current time: Clock unsynchronized. Status: Service unavailable: Error reading clock. Clock considered

I got an "authentication token expired" error.

Hi. I'm using Apache Kudu 1.4.0. And I have a long running Java Daemon which is a kudu client at the same time. Today (7 days has been past since the Java Daemon has been started) I suddenly got an following error. W0614 15:29:11.934401 62459 negotiation.cc:310] Unauthorized connection attempt

How to manage yearly range partition efficiently

Hi. This is a partition strategy of my table. PARTITION BY HASH (...) PARTITIONS 40, RANGE (ymd) ( PARTITION VALUES < "2015", PARTITION "2015" <= VALUES < "2016", PARTITION "2016" <= VALUES < "2017", PARTITION "2017" <= VALUES ) My concern is that how to manage RANGE(ymd) partiti

Re: kudu-tserver died suddenly

g all Data. Thanks. 2017-06-06 15:13 GMT+09:00 Jason Heo : > Hi Todd, > > Thank you for your reply. > > Ok, I got it. I should have googled it before mailing ;) > > Regards, > > Jason > > 2017-06-06 15:03 GMT+09:00 Todd Lipcon : > >> Hi Jason, >>

Re: kudu-tserver died suddenly

x27;t gotten around to fixing yet. I hadn't seen > it "in the wild" before, but I'll add a note to the JIRA that you hit it, > and try to prioritize a fix soon (eg for 1.4.1) > > -Todd > > On Mon, Jun 5, 2017 at 6:38 PM, Jason Heo wrote: > >> Hello. >&

Re: kudu-tserver died suddenly

election timeout plus an additional 1.566s I0605 15:04:35.121601 75055 raft_consensus.cc:2061] T 87e5a254f3aa4070bf138a6d08ed611d P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0605 15:04:35.445930 138806 raft_consensus.cc

kudu-tserver died suddenly

Hello. I'm using this patch https://gerrit.cloudera.org/#/c/6925/ One of tservers died suddenly. Here is ERROR and FATAL log. E0605 15:04:33.376554 138642 tablet.cc:1219] T 3cca831acf744e1daee72582b8e16dc4 P 125dbd2ffb8a401bb7e4fd982995ccf8: Rowset selected for compaction but not available anymo

Re: Question about redistributing tablets on failure of a tserver.

; today, and confirmed that the patch is working as intended. I've updated >> the commit message with more info about the failure that was occurring, in >> case you were interested. I expect this fix will land in time for 1.5. >> >> - Dan >> >> On Sat, May

Re: Question about redistributing tablets on failure of a tserver.

in to it. Any feedback you have on > it would be appreciated. > > - Dan > > On Fri, May 19, 2017 at 10:07 PM, Jason Heo > wrote: > >> Thanks, @dan @Todd >> >> This issue has been resolved via https://gerrit.cloudera.org/#/c/6925/ >> >> Regards

Re: Question about redistributing tablets on failure of a tserver.

gt; > FWIW Will B is working on adding more informative output to ksck to help > diagnose cases like this: > https://gerrit.cloudera.org/#/c/6772/ > > -Todd > > On Thu, Apr 13, 2017 at 11:35 PM, Jason Heo > wrote: > >> @Dan >> >> I monitored with `kudu ksc

Re: Kudu Table Design Question

unt adjusted (but not necessarily equal to) to the core count > gives you a good performance tradeoff. > Of course this is not a hard limit, the tablet server should be able > anything up to 100 reasonably well depending on your hardware. > > HTH > -david > > > On

Kudu Table Design Question

Hi. This email (and many other resources) suggests that tserver should have small number of tablets. In the above mail, Dan says that: >> something more

Re: Some bulk requests are missing when a tserver stopped

David Alves : > Hi Jason > > Were there errors, like timeouts, when you were writing the rows? > > -david > > On Mon, Apr 24, 2017 at 7:38 PM, Jason Heo > wrote: > >> Hi David, Todd. >> >> @David >> >> >> *What do you mean that

Re: Number of data files and opened file descriptors are not decreasing after DROP TABLE.

7;ve answered all your questions. If you have specific > concerns, please let us know what you are worried about. > > Mike > > On Sun, Apr 23, 2017 at 11:43 PM, Jason Heo > wrote: > >> Hi. >> >> Before dropping, there were about 30 tables, 27,000 files in tablet_d

Re: Some bulk requests are missing when a tserver stopped

gt;> Currently some throttling might be required to make sure that the >> clients don't overload the server with writes which causes writes to start >> timing out. More efficient bulk loads is something we're working on right >> now. >> >> Best >>

Re: tserver died during bulk indexing and dies again after restarting

d out of most if not all consensus config, at which point, > if all you data is available you could just delete the data and re-add it > to the cluster. > > Best > David > > > On Mon, Apr 24, 2017 at 4:33 AM, Jason Heo > wrote: > >> Hi David. >> >> Th

Question about memory_limit_hard_bytes and

Hello. I'm using Apache Kudu 1.3.0. I found some explanation in manual and cdh about memory_limit_hard and block_cache_capacity But I couldn't understand what are they exactly used for, what is stored at block_cache, and what's the meaning of hard limit of memory? In Kudu Configuration Reference

Re: tserver died during bulk indexing and dies again after restarting

> A wal file has a name like: "wal-00001" > > Best > David > > > On Sat, Apr 22, 2017 at 7:46 PM, Jason Heo > wrote: > >> Hi David. >> >> Sorry for the insufficient information. >> >> There are 14 nodes in my test kudu clus

Number of data files and opened file descriptors are not decreasing after DROP TABLE.

Hi. Before dropping, there were about 30 tables, 27,000 files in tablet_data directory. I dropped most tables and there is ONLY one table which has 400 tablets in my test Kudu cluster. After dropping, there are still 27,000 files in tablet_data directory, and output of /sbin/lsof is the same befo

Re: tserver died during bulk indexing and dies again after restarting

Hi David. Sorry for the insufficient information. There are 14 nodes in my test kudu cluster. Only one tserver has been dead. It has only above two logs. Other 13 nodes has "Error trying to read ahead of the log while preparing peer request: Incomplete: Op with" error 7~10 times. >> *Would it b

Re: Table size is not decreasing after large amount of rows deleted.

Hi David. Thank you for your reply. I'll try to upgrade to 1.3 this week. Regards, Jason 2017-04-23 2:06 GMT+09:00 : > Hi Jason > > In Kudu 1.2 if there are compactions happening, they will reclaim > space. Unfortunately the conditions for this to happen don't always > occur (if the portion

Some bulk requests are missing when a tserver stopped

Hi. I'm using Apache Kudu 1.2. I'm currently testing high availability of Kudu. During bulk loading, one tserver is stopped via CDH Manager intentionally and 2% of rows are missing. I use Spark 1.6 and package org.apache.kudu:kudu-spark_2.10:1.1.0 for bulk loading. I got a error several times d

tserver died during bulk indexing and dies again after restarting

Hi. I'm using Apache Kudu 1.2. One of tservers has been dead during bulk indexing. Here is the log of the dead tserver. Can I know that's the problem and how to start tserver? == $ tail -f kudu-tserver.WARNING Log file created at: 2017/04/22 14:44:59 Running on machi

Table size is not decreasing after large amount of rows deleted.

Hello. I'm using Apache Kudu 1.2. I've deleted 30% of rows from a 20TB table. What I expected is that the size of table was decreased by almost 30%. But unfortunately, its size was increased by 4.4%. (I noticed that there was no Major Delta compaction during deletion or after deletion) Is there

Re: Building from Source fails on my CentOS 7.2

012b k5crypto_3_MIT > krb5_enctype_to_name > > Do you have the MIT krb5 dev libraries installed, or is it possible you > have heimdal or some other krb5 implementation? > > -Todd > > On Thu, Apr 13, 2017 at 10:00 PM, Jason Heo > wrote: > >> Hello. >>

Re: How to flush `block_cache_capacity_mb` easily?

Hi, Todd. I've temporarily pushed this patch to my repository. https://github.com/jason-heo/kudu/commit/aff1fe181541671d2dc192ad9cb4ed2172a51826 Could you please check I'm on right track? It will take more time until pushing to cloudera's gerrit because I have yet to test if

Re: Question about redistributing tablets on failure of a tserver.

=== table consistency check error: Corruption: 52 table(s) are bad FAILED Runtime error: ksck discovered errors ``` 2017-04-13 3:47 GMT+09:00 Dan Burkert : > Hi Jason, answers inline: > > On Wed, Apr 12, 2017 at 5:53 AM, Jason Heo > wrote: > >> >> Q1. Can

Building from Source fails on my CentOS 7.2

Hello. I'm using CentOS 7.2 To build from Source Code, I followed the manual (except for Re Hat Developer Toolset because I use CentOS 7.2) Though I failed to compile :( ``` ... [ 31%] Building CXX object src/kudu/master/CMakeFi

Re: Question about redistributing tablets on failure of a tserver.

Thanks, Jason 2017-04-13 3:47 GMT+09:00 Dan Burkert : > Hi Jason, answers inline: > > On Wed, Apr 12, 2017 at 5:53 AM, Jason Heo > wrote: > >> >> Q1. Can I disable redistributing tablets on failure of a tserver? The >> reason why I need this is described in

Re: Physical Tablet Data size is larger than size in Chart Library.

#x27;s the >> resident expert on the subject. >> >> KUDU-1755 <https://issues.apache.org/jira/browse/KUDU-1755> >> KUDU-1853 <https://issues.apache.org/jira/browse/KUDU-1853> >> KUDU-1856 <https://issues.apache.org/jira/browse/KUDU-1856> >

Question about redistributing tablets on failure of a tserver.

Hello. I'm using Apache Kudu 1.2 on CDH 5.10. Background --- I'm currently using Elasticsearch to serve web analytic service. Elasticsearch is very easy to manage cluster. One nice feature of ES is that I can disable allocation of shard (shard is similar to tablet of Kudu) intentiona

Physical Tablet Data size is larger than size in Chart Library.

Hello. I'm using Apache Kudu 1.2 on CDH 1.2. I'm estimating how many servers needed to store my data. After loading my test data sets, total_kudu_on_disk_size_across_kudu_replicas in chart library at CDH is 27.9TB whereas sum of `du -sh /path/to/tablet_data/data` on each node is 39.9TB which is

Re: How to flush `block_cache_capacity_mb` easily?

Great. I would be appreciated it if you guide me how can I contribute it. Then I'll try in my spare time. 2017-04-11 7:46 GMT+09:00 Todd Lipcon : > On Sun, Apr 9, 2017 at 6:38 PM, Jason Heo wrote: > >> Hi Todd. >> >> I hope you had a good weekend. >> >>

Re: How to flush `block_cache_capacity_mb` easily?

t I typically look at the 'MemRowSet' >> memory usage for the tablet on the /mem-trackers HTTP endpoint; it should >> show something minimal like 256B if it's fully flushed and empty. You can >> also see details about how much memory is in the block cache on

How to flush `block_cache_capacity_mb` easily?

Hi. I'm using Apache Kudu 1.2 on CDH 5.10. Currently, I'm doing a performance test of Kudu. Flushing OS Page Cache is easy, but I don't know how to flush `block_cache_capacity_mb` easily. I currently execute SELECT statement over a unnecessarily table to evict cached block of testing table. It

How to calculate the optimal value of `maintenance_manager_num_threads`

Hi, I'm using Apache Kudu 1.2 on CDH 5.10. Recently, after reading "Bulk write performance improvements for Kudu 1.4 " I've noticed that `maintenance_manager_num_threads` is 4 for the 5 spinning disks. In my cl

Re: What does RowSet Compaction Duration means?

ifferent > tables run independently. So, if you have some other activity doing > inserts/updates on tableB, then it's natural to see compaction happen on > tabletB as well. > > > Best regards, > > Alexey > > On Tue, Mar 14, 2017 at 12:50 AM, Jason Heo > wrote: >

Re: Load is high on the Kudu dedicated node.

isn't the issue, do you think you could use 'perf record > -g -a' and generate a flame graph? http://www. > brendangregg.com/FlameGraphs/cpuflamegraphs.html > > -Todd > > On Tue, Mar 14, 2017 at 6:14 AM, Jason Heo > wrote: > >> Hi. I'm experiencing

Re: AUTO_FLUSH_BACKGROUND is supported in Impala 2.7?

ext.scala#L212 > > J-D > > On Tue, Mar 14, 2017 at 3:38 AM, Jason Heo > wrote: > >> @Harsh >> >> Can I ask one more question? >> >> I wanted to know if AUTO_FLUSH_BACKGROUND can be enabled or already >> enabled by default with Apache Spark. >

Load is high on the Kudu dedicated node.

Hi. I'm experiencing high load and high cpu usage. Kudu is running on 5 kudu dedicated nodes. 2 nodes' load is 40, while 3 nodes' load is 15. Here is the output of `perf record -a & perf report` during bulk indexing only operation. http://imgur.com/8lz1CRk I'm wondering this is a reasonable situ

Re: AUTO_FLUSH_BACKGROUND is supported in Impala 2.7?

@Harsh Can I ask one more question? I wanted to know if AUTO_FLUSH_BACKGROUND can be enabled or already enabled by default with Apache Spark. Thanks. 2017-03-14 19:24 GMT+09:00 Jason Heo : > @Harsh > > Ok. I got it. > > Thanks. > > 2017-03-14 17:41 GMT+09:00 Harsh J : &

Re: AUTO_FLUSH_BACKGROUND is supported in Impala 2.7?

62. > > On Tue, 14 Mar 2017 at 12:34 Jason Heo wrote: > >> Sorry. >> >> I've noticed that `v2.7.0` is just Impala Shell's. >> >> It seems CDH 5.10.x includes Impala 2.8. But I can't find if >> AUTO_FLUSH_BACKGROUND is added at the &

What does RowSet Compaction Duration means?

Hi. I'm stuck with performance degradation on compaction happens. My Duration is "4956.71 milliseconds / second" What does this mean? I can't figure it out. Here is the captured image: http://imgur.com/WU9sRRq When I'm doing bulk indexing on tableA, sometimes compaction happens over tableB. Is

Re: AUTO_FLUSH_BACKGROUND is supported in Impala 2.7?

mpala_new_features.html#new_features_280> " 2017-03-14 15:47 GMT+09:00 Jason Heo : > Hi. > > I'm using (Impala + Kudu) on CDH 5.10 > > > > version; > Shell version: Impala Shell v2.7.0-cdh5.10.0 (785a073) built on Fri Jan 20 > 12:03:56 PST 2017 &

AUTO_FLUSH_BACKGROUND is supported in Impala 2.7?

Hi. I'm using (Impala + Kudu) on CDH 5.10 > version; Shell version: Impala Shell v2.7.0-cdh5.10.0 (785a073) built on Fri Jan 20 12:03:56 PST 2017 Server version: impalad version 2.7.0-cdh5.10.0 RELEASE (build 785a073cd07e2540d521ecebb8b38161ccbd2aa2) I've read IMPALA-4134

Re: Apache Kudu Table is 6.6 times larger than Parquet File.

Hi, Janne As I mentioned, I'm using CDH 5.10. I checked it using Cloudera Manager at "Kudu -> Chart Library" I'm not sure there is another way. Thanks. 2017-03-13 17:46 GMT+09:00 Janne Keskitalo : > Hi > > How do you check the physical size of a kudu table? > > ​ >

Re: Apache Kudu Table is 6.6 times larger than Parquet File.

t; > That should get you close to Parquet sizes (and those are the new defaults > in the upcoming 1.3 release). If you still see a 6x blowup after making > these changes please report back. > > -Todd > > On Fri, Mar 10, 2017 at 7:16 PM, Jason Heo > wrote: > >> Hello, I

Apache Kudu Table is 6.6 times larger than Parquet File.

Hello, I'm new to Apache Kudu. I was really impressed by the concept of Kudu and benchmark results. I'm considering using (Impala + Kudu) on my team project. One of the issues I have is that Kudu Table is too big compared to Parquet File - Parquet File: 1.3TB - Kudu Table: 8.6TB (both tables con