Thanks for taking time to reply ShaoFeng, 1) "Using Kylin's rest API to trigger a cube rebuild/refresh when the data is changed in Hive;" Rebuild / refresh builds the whole cube. I was looking at a way to somehow present only the changed (updates / inserts) facts records.
2) "No change I think; JIRA is welcomed" Okay added to Jira. https://issues.apache.org/jira/browse/KYLIN-1843 https://issues.apache.org/jira/browse/KYLIN-1844 3) "Do these dimension snapshots get reloaded on a cube refresh / data appended ?" Wanted to know if data is appended or a segment is refreshed, will the dimension snapshots be reloaded. Pointers to the code also will help. 4) "The cap on snapshot is still there, you can customize the max size in kylin.properties" Okay 5) "When deploy into cluster, the query servers are identical in role, that means all servers can server all cubes" Okay. Can you please let us know where to look to figure out how kylin decides which server to pass the request to. Also could you let us know how it handles one of the query servers going down. 6) " You can play the sample cube shipped with kylin binary, which is partitioned by a date column, then you can try the incremental build function: https://kylin.apache.org/docs15/tutorial/kylin_sample.html " Okay, we will try it out. But, I feel it will help someone evaluating the tool, if we list down the limitations / assumptions for an incremental/segment build. Eg: How does pure distinct count (for int) work when only of the data is refreshed. Also updates on dimension keys in the fact table. Regards, Abhilash On Fri, Jul 1, 2016 at 9:00 AM, ShaoFeng Shi <[email protected]> wrote: > Hi Abhilash, welcome back; > > Kylin 1.5 delivers a new plugable architecture, with many enhancements > especially on performance, like fast cubing algorithm, sharded storage > engine, etc. The facts 1) to 4) are unchanged in between I think; For the > issues like memory usage, you can open JIRA to us or submit a patch if you > have solved it; > > 1) Using Kylin's rest API to trigger a cube rebuild/refresh when the data > is changed in Hive; > 2) No change I think; JIRA is welcomed; > 3) What do you mean by "reload" here? If new segment is ready and query > comes, its dim snapshots will be loaded; as the it is immutable, a > "reload"/"refresh" is not needed; > 4) The cap on snapshot is still there, you can customize the max size in > kylin.properties; > 5) When deploy into cluster, the query servers are identical in role, that > means all servers can server all cubes; > 6) You can play the sample cube shipped with kylin binary, which is > partitioned by a date column, then you can try the incremental build > function: https://kylin.apache.org/docs15/tutorial/kylin_sample.html > > > > 2016-06-29 13:56 GMT+08:00 Abhilash L L <[email protected]>: > > > Hello, > > > > We plan to upgrade to Kylin 1.5.x series from 1.2 series. We want to > ask > > the team a few questions / clarification before proceeding. Since we are > > looking at upgrading, please do let us know the behaviour for the latest > > 1.5.x release. > > > > > > 1) Updating facts > > Eg: > > Lets say im building a cube for jira tickets. List of tickets is my fact > > table. List of ticket status is my dimension table. Now if a ticket is > > updated from 'open' to 'in progress' how to tell kylin this change ? > > > > 2) Multiple dimensions in memory > > When we were on 1.2 versions, all the dimensions were kept in memory. So > > memory always keeps increasing. The in memory dimensions were not swapped > > in / out, causing tomcat OOM issues. Is this still the same in 1.5.x ? > > > > 3) Do these dimension snapshots get reloaded on a cube refresh / data > > appended ? > > > > 4) Maximum size for dimension > > There was an older thread talking about 300mb max snapshot size. Does > this > > limitation still hold? Do very high cardinality dimension still build the > > trees on one node ? > > > > > > 5) Multiple query servers > > When there are multiple query servers, will one query server serve only > one > > cube or all servers serve all cubes. If its one cube only in one server, > > how does kylin handle a server going down. > > > > > > 6) Incremental build > > Is there a document on how incremental build works ? We want to > understand > > limitations / assumptions for this. > > > > Regards, > > Abhilash > > > > > > -- > Best regards, > > Shaofeng Shi >
