Hi all. Thanks for kindly reply. I can bring up a partial list of problems that we faced in production: * metadata refreshing become more expensive when HDFS centralized cache is enabled (cached location is also stored, hence file meta tend to change from time to time). * memory consumption of both impalad, statestored and catalogd grow as number of partitions and files increases. which makes scaling out expensive (total reserved memory is increasing). also, the issue about size <https://issues.cloudera.org/browse/IMPALA-3499>. * we took the simplest approach that when user finished their 'INSERT' with Hive or other tools, they should then execute 'REFRESH' on Impala. this may randomly fail (didn't investigate yet) and leaving meta data stale, causing correctness issues on following queries (empty result, for example). * rolling restart of impalad may exhaust the bandwidth of node running statestored (given that size of metadata is large). on the other hand, increasing sleeping period between restart operations may brings down availability (running queries will fail when any impalad restarts).
I also don't have a theory to solve these problems yet. But I wonder if it's worth to make the system simpler as tradeoff for availability. (As you know, it's too early to talk about performance when availability does not meet the needs) Would someone share any analysis on query performance about hive metadata cache and HDFS meta cache? On Thu, Jul 14, 2016 at 9:13 AM, Henry Robinson <[email protected]> wrote: > Hi Tianyi - > > The idea behind the metadata cache is to try to ensure that no RPCs to the > metastore are on the critical path of query planning. However, the way we > populate and invalidate the cache does cause us some problems; the > statestore was never designed to carry this kind of payload and this can > cause difficulties in large clusters. > > We are aware of the problems, and hope to do something about them, but > don't have concrete plans just yet. If you have thoughts, please bring them > up on this dev@ mailing list and we can discuss them! > > Thanks, > Henry > > On 12 July 2016 at 13:34, Huaisi Xu <[email protected]> wrote: > >> Hi Tianyi, thanks for contacting us! >> >> >> >> Could you elaborate the biggest problems you are facing with this design? >> >> >> >> As we are moving to ASF, you can ask questions here >> [email protected]. >> >> >> >> I think for questions regarding design decisions and future improvement, >> +Dimitris and +Henry knows better. >> >> >> >> >> >> Huaisi >> >> >> >> *From: *何天一 <[email protected]> >> *Date: *Monday, July 11, 2016 at 10:42 PM >> *To: *Huaisi Xu <[email protected]> >> *Subject: *Looking for OLAP suggestion >> >> >> >> Hi, Huaisi. >> >> >> >> We communicated before in cloudera JIRA (IMPALA-3499 >> <https://issues.cloudera.org/browse/IMPALA-3499>). I am currently >> working on distributed storage and computing, including OLAP engines, for >> 今日头条. >> >> >> >> I am looking for technical suggestions and hope you could help. >> >> >> >> I see that Impala Catalogd caches metadata from Hive Metastore and HDFS >> (or other storage). >> >> IMHO This can be considered as a good optimization for performance. >> >> However, in our production environment, this mechanism tend to cause >> problem. >> >> Could you help to explain the design choice behind this? Why did Impala >> cache meta in the first place? And, is there any optimization in progress >> to make the mechanism better? >> >> >> >> Thanks. >> >> >> >> >> -- >> >> Cheers, >> >> Tianyi HE >> >> (+86) 185 0042 4096 >> > > > > -- > Henry Robinson > Software Engineer > Cloudera > 415-994-6679 > -- Cheers, Tianyi HE (+86) 185 0042 4096
