Glad to hear this; Yes there are a couple of enhancements in the metadata accessing from v2.4 to v2.6; Thank you for sharing this information, it is a good encouragement for the contributors!
Best regards, Shaofeng Shi 史少锋 Apache Kylin PMC Email: [email protected] Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html Join Kylin user mail group: [email protected] Join Kylin dev mail group: [email protected] Billy Liu <[email protected]> 于2019年3月20日周三 下午1:48写道： > Good to know. As you mentioned, the querying jost list has reduced the > time from 10 seconds to 1, the improvement is not only from MySQL backend > database, but also the querying job function itself. There are some > enhancements about loading huge metadata. > > With Warm regards > > Billy Liu > > > Iñigo Martínez <[email protected]> 于2019年3月19日周二 下午2:01写道： > >> Good afternoon. >> >> I want to share my experience on tests run this morning. Our current >> production environment is kyin 2.4.1. >> >> - First, we configured kylin 2.6.1 in order to check that everything was >> running properly and we launched several cube build processes (with example >> learn_cube). As metadata backend we started using mysql instead of hbase. >> - Second, we created a metadata backup from Kylin 2.4.1 and restored on >> kylin 2.6.1. Process was really fast, and when I say fast is VERY VERY >> fast. In the past, the restore procedure took around 15 minutes with HBase >> as backend. With mysql only a couple of minutes. No problem at all. >> - Finally, we started kylin 2.6.1. All cube metadata definitions were in >> place. We launched several queries and all of them worked without issues. >> In this moment we had a new mysql metadata from one side and an old base >> metadata in our production cluster. Kylin tables were, of course, located >> at hbase and were shared by both enviroments. Of course, beginning this >> point we have two different Kylin installations and we have to be carefull >> with build process in order to not delete shared segments, but this is >> another story. >> >> We can say that metadata performance is much better with mysql as >> backend. For example, querying job list takes around 10 seconds when >> metadata backend is HBase and only one second when mysql. Remember that we >> have migrated all metadata stuff from hbase, so both metadata databases are >> equal in terms of size and number of entries. >> >> For the next days we are going to evaluate another aspects of 2.6.1 >> deployment, mainly focused on performance, bugs fixed and stability. >> >> Thank you Shao Feng for your tips. >> >> >> El mar., 19 mar. 2019 a las 11:01, Iñigo Martínez (< >> [email protected]>) escribió: >> >>> Hi Shao Feng. >>> >>> Yesterday I deployed a new 2.6.1 instance with both mysql and hbase as >>> metastore. Since I've only built test cube, everything runs smoothly. >>> I'm going to test today restoring metadata from 2.4.1 to see if we can >>> proceed with an in place migration. Using mysql as backend will probably >>> help is in order to get more stability with HBase. We have made huge >>> improvements tuning hbase, but still some issues as present and having >>> thousands of builds per month dumping logs in kylin_metadata table is not >>> very good. >>> >>> El mar., 19 mar. 2019 a las 8:48, ShaoFeng Shi (<[email protected]>) >>> escribió: >>> >>>> Hello Inigo, >>>> >>>> This is a good question. >>>> >>>> The MySQL metadata store was introduced in Kylin 2.5, as preparation >>>> for the no-HBase deployment. There is no evidence to say the MySQL meta >>>> store will have better performance, or be more stable. But at least, when >>>> HBase has a problem, Kylin service won't be impacted. >>>> >>>> To migrate to MySQL meta store, you can just 1) dump all metadata to >>>> local disk; 2) change Kylin configuration to use MySQL meta store; 3) >>>> restore metadata from local disk. MySQL will use two tables to persist the >>>> metadata, one for static resources (project, cube, etc), the other for job >>>> outputs. But this is transparent to end user. >>>> >>>> >>>> Best regards, >>>> >>>> Shaofeng Shi 史少锋 >>>> Apache Kylin PMC >>>> Email: [email protected] >>>> >>>> Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html >>>> Join Kylin user mail group: [email protected] >>>> Join Kylin dev mail group: [email protected] >>>> >>>> >>>> >>>> >>>> Na Zhai <[email protected]> 于2019年3月17日周日 下午9:28写道： >>>> >>>>> Hi, Iñigo Martínez. >>>>> >>>>> >>>>> >>>>> If you meet too many problems with HBase, you can try to use MySQL >>>>> instead. Hope this can help you. >>>>> http://kylin.apache.org/docs/tutorial/mysql_metastore.html. In HBase, >>>>> there is one metadata_table. In MySQL, there are two metadata_tables. So I >>>>> think you can not migrate metadata from HBase to MySQL directly. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> 发送自 Windows 10 版邮件 <https://go.microsoft.com/fwlink/?LinkId=550986>应用 >>>>> >>>>> >>>>> ------------------------------ >>>>> *发件人:* Iñigo Martínez <[email protected]> >>>>> *发送时间:* Thursday, March 14, 2019 10:40:27 PM >>>>> *收件人:* [email protected] >>>>> *主题:* Upgrade from 2.4.1 to 2.6.1 >>>>> >>>>> Now that 2.6.1 is launched, we are planning an upgrade (with a >>>>> transition period and an extensive test plan) from 2.4.1. I have a few >>>>> questions for developers and / or system administrators. >>>>> >>>>> - Do you recommend use mysql as kylin metadata storage? Until now, >>>>> most of our problems are related with hbase since we rebuild cubes very >>>>> frequently and kylin_metadata table grows a lot. Do you notice performance >>>>> loss or gain? Is it stable enough? >>>>> >>>>> - If using mysql as metadata backend, can we export / import metadata >>>>> in order to migrate from hbase to mysql without having rebuild existing >>>>> cubes? >>>>> >>>>> Thanks in advance. >>>>> >>>>> >>> >>> -- >>> >>> >>> >>> >>> Iñigo Martínez >>> Systems Manager >>> [email protected] >>> >>> >>> >>> >>> >>> >>> [image: Telecoming - Make it digital] >>> [image: 5000_empresas] >>> [image: 1000_empresas] >>> >>> Paseo de la Castellana, 95. Torre Europa, pl 16. 28046 Madrid, Spain | >>> telecoming.com <http://www.telecoming.com/> >>> >>> >>> >>> Este correo electrónico y sus archivos adjuntos están dirigidos >>> únicamente a la(s) dirección(es) indicada(s) anteriormente. El carácter >>> confidencial, personal e intransferible del mismo está protegido >>> legalmente. Cualquier publicación, reproducción, distribución o >>> retransmisión no autorizada, ya sea completa o en parte, se encuentra >>> prohibida. Si ha recibido este mensaje por equivocación, notifíquelo >>> inmediatamente a la persona que lo ha enviado y borre el mensaje original >>> junto con sus ficheros anexos sin leerlo ni grabarlo en modo alguno. >>> >>> >> >> -- >> >> >> >> >> Iñigo Martínez >> Systems Manager >> [email protected] >> >> >> >> >> >> >> [image: Telecoming - Make it digital] >> [image: 5000_empresas] >> [image: 1000_empresas] >> >> Paseo de la Castellana, 95. Torre Europa, pl 16. 28046 Madrid, Spain | >> telecoming.com <http://www.telecoming.com/> >> >> >> >> Este correo electrónico y sus archivos adjuntos están dirigidos >> únicamente a la(s) dirección(es) indicada(s) anteriormente. El carácter >> confidencial, personal e intransferible del mismo está protegido >> legalmente. Cualquier publicación, reproducción, distribución o >> retransmisión no autorizada, ya sea completa o en parte, se encuentra >> prohibida. Si ha recibido este mensaje por equivocación, notifíquelo >> inmediatamente a la persona que lo ha enviado y borre el mensaje original >> junto con sus ficheros anexos sin leerlo ni grabarlo en modo alguno. >> >>
