I see; Thanks for the sharing! When the metadata changing frequently, the HBase will optimize the metadata table automatically.
Some users encounter performance downgrade when the job history is very long. User needs to run the metadata cleanup job to ensure it won't be too large. Best regards, Shaofeng Shi 史少锋 Apache Kylin PMC Email: [email protected] Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html Join Kylin user mail group: [email protected] Join Kylin dev mail group: [email protected] Iñigo Martínez <[email protected]> 于2019年3月12日周二 下午5:54写道： > Well, we see regular compaction tasks in our hbase cluster. And all of > them should be related to kylin_metadata table since regions used for > segment tables are not modified. We also launch a daily cleanup task to > deal with kylin_metadata size in order to maintain size inside reasonable > limits. > Anyway, our kylin usage is unusual in terms of segment rebuilds. We have 8 > cubes and we have to rebuild segments in a regular way in order to deal > with changes in data. Our data modification window is one week wide and > sometimes months. And we also require "near real time" updates. So, our > kylin deployment is constantly rebuilding segments (for current day) and > rebuilding daily for current week and so on.... and kylin_medatada grows > and grows since kylin stores logs inside. > > I was talking with Yu Li and Luke last time they were in Spain and they > told me that eventually we could update already built segments via kafka (I > think this feature is not available yet)... > > El mar., 12 mar. 2019 a las 10:39, ShaoFeng Shi (<[email protected]>) > escribió: > >> As I know, we seldom run major compaction. >> >> Do you see the need for compaction? Is the metadata changing very often? >> Welcome to share your experience. Thanks! >> >> Best regards, >> >> Shaofeng Shi 史少锋 >> Apache Kylin PMC >> Email: [email protected] >> >> Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html >> Join Kylin user mail group: [email protected] >> Join Kylin dev mail group: [email protected] >> >> >> >> >> Iñigo Martínez <[email protected]> 于2019年3月7日周四 下午8:31写道： >> >>> Good morning. >>> >>> A simple maintenance question. Since kylin_metadata hbase table is >>> constantly being modified (not like regular segment tables), do you >>> recommended to perform a manual major compaction on this table on a regular >>> manner? I'm thinking about a nightly major compaction. >>> >>> Thanks in advance. >>> >>> >>> > > -- > > > > > Iñigo Martínez > Systems Manager > [email protected] > > > > > > > [image: Telecoming - Make it digital] > [image: 5000_empresas] > [image: 1000_empresas] > > Paseo de la Castellana, 95. Torre Europa, pl 16. 28046 Madrid, Spain | > telecoming.com <http://www.telecoming.com/> > > > > Este correo electrónico y sus archivos adjuntos están dirigidos > únicamente a la(s) dirección(es) indicada(s) anteriormente. El carácter > confidencial, personal e intransferible del mismo está protegido > legalmente. Cualquier publicación, reproducción, distribución o > retransmisión no autorizada, ya sea completa o en parte, se encuentra > prohibida. Si ha recibido este mensaje por equivocación, notifíquelo > inmediatamente a la persona que lo ha enviado y borre el mensaje original > junto con sus ficheros anexos sin leerlo ni grabarlo en modo alguno. > >
