Hi Sumudu, Could you share the DDL of the table, i.e. "show create table default.table_abc"? Just want to check the primary keys.
> We have achieved the performance as we expected in Kudu level, however, in > Impala we haven’t reached to the performance we expected. The TopN operator in the query also takes a significant amount of time. How do you test the performance in Kudu level? It'd be helpful if you could share the test codes. Another question is about the versions. The profile doesn't show the exact Impala version. What's the kudu client version used in your Impala? What's the version of your kudu cluster? Regards, Quanlong On Mon, May 2, 2022 at 5:31 PM Sumudu Madushanka <ngsmadushanka...@cse.mrt.ac.lk> wrote: > > Hi, > Here is the profile of the query > > > > Thank you! > > > > Best Regards > > Sumudu Madushanka > > > > From: Tim Armstrong > Sent: Friday, April 29, 2022 8:55 PM > To: user@impala.apache.org > Subject: Re: Impala Performance Issue with huge Kudu Data load > > > > Hi Sumudu, > > We might be able to help more if you attach a query profile. You can get it > from impala-shell with the "profile" command or from the Impala debug web UI > "queries" tab. > > > > - Tim > > > > On Fri, 29 Apr 2022 at 07:07, Sumudu Madushanka > <ngsmadushanka...@cse.mrt.ac.lk> wrote: > > Hi Team, > > This is regarding a performance issue me and my team have on a huge data load > in Kudu. We are hoping you can guide us on a solution to the below mentioned > concerns. > > We have 212 million data loads in Kudu. Currently for such a data load, when > loading through impala, 47 seconds are spent for query processing and loading > overall. We have used default configurations in Kudu and Impala with 6 node > clusters to get these numbers. We have achieved the performance as we > expected in Kudu level, however, in Impala we haven’t reached to the > performance we expected. > > > What can we do to reduce the time spent for loading 212 million data loads > from 47 seconds to 10 seconds through impala? > > We would be much obliged if you can provide us with some solutions. > > Thank You! > > > Best Regards > > Sumudu Madushanka > >
