Hi,

I have bumped into the 200MB limit when calculating incremental stats (
https://issues.apache.org/jira/browse/IMPALA-3552).

I don't understand which data catalogd sends to the impalad each time the
incremental stats are calculated. Does it send only the new information
calculated for new partitions or all the statistics data?

In my case I have 387 tables with 2550 columns. I am creating a new
partition for each table every hour and calculating incremental stats for
these new partitions. If catalogd is sending serialized data for the new
partitions and columns, it shouldn't amount to 200MB.

I would appreciate if someone can help me understand this concept or point
me to some documentation.

Best regards,
Miguel

-- 
Miguel Figueiredo
Software Developer

"I'm a pretty lazy person and am prepared to work quite hard in order to
avoid work."
-- Martin Fowler

Reply via email to