Well, we see regular compaction tasks in our hbase cluster. And all of them should be related to kylin_metadata table since regions used for segment tables are not modified. We also launch a daily cleanup task to deal with kylin_metadata size in order to maintain size inside reasonable limits. Anyway, our kylin usage is unusual in terms of segment rebuilds. We have 8 cubes and we have to rebuild segments in a regular way in order to deal with changes in data. Our data modification window is one week wide and sometimes months. And we also require "near real time" updates. So, our kylin deployment is constantly rebuilding segments (for current day) and rebuilding daily for current week and so on.... and kylin_medatada grows and grows since kylin stores logs inside.
I was talking with Yu Li and Luke last time they were in Spain and they told me that eventually we could update already built segments via kafka (I think this feature is not available yet)... El mar., 12 mar. 2019 a las 10:39, ShaoFeng Shi (<[email protected]>) escribió: > As I know, we seldom run major compaction. > > Do you see the need for compaction? Is the metadata changing very often? > Welcome to share your experience. Thanks! > > Best regards, > > Shaofeng Shi 史少锋 > Apache Kylin PMC > Email: [email protected] > > Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html > Join Kylin user mail group: [email protected] > Join Kylin dev mail group: [email protected] > > > > > Iñigo Martínez <[email protected]> 于2019年3月7日周四 下午8:31写道： > >> Good morning. >> >> A simple maintenance question. Since kylin_metadata hbase table is >> constantly being modified (not like regular segment tables), do you >> recommended to perform a manual major compaction on this table on a regular >> manner? I'm thinking about a nightly major compaction. >> >> Thanks in advance. >> >> >> -- Iñigo Martínez Systems Manager [email protected] [image: Telecoming - Make it digital] [image: 5000_empresas] [image: 1000_empresas] Paseo de la Castellana, 95. Torre Europa, pl 16. 28046 Madrid, Spain | telecoming.com <http://www.telecoming.com/> Este correo electrónico y sus archivos adjuntos están dirigidos únicamente a la(s) dirección(es) indicada(s) anteriormente. El carácter confidencial, personal e intransferible del mismo está protegido legalmente. Cualquier publicación, reproducción, distribución o retransmisión no autorizada, ya sea completa o en parte, se encuentra prohibida. Si ha recibido este mensaje por equivocación, notifíquelo inmediatamente a la persona que lo ha enviado y borre el mensaje original junto con sus ficheros anexos sin leerlo ni grabarlo en modo alguno.
