[jira] [Created] (KYLIN-2957) Fix copy ACL issue when migrate cube
Zhong Yanghong created KYLIN-2957: - Summary: Fix copy ACL issue when migrate cube Key: KYLIN-2957 URL: https://issues.apache.org/jira/browse/KYLIN-2957 Project: Kylin Issue Type: Bug Components: Tools, Build and Test Reporter: Zhong Yanghong Assignee: Zhong Yanghong -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2956) building trie dictionary blocked on value of length over 4095
[ https://issues.apache.org/jira/browse/KYLIN-2956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wang, Gang updated KYLIN-2956: -- Description: In the new release, Kylin will check the value length when building trie dictionary, in class TrieDictionaryBuilder method buildTrieBytes, through method: private void positiveShortPreCheck(int i, String fieldName) { if (!BytesUtil.isPositiveShort(i)) { throw new IllegalStateException(fieldName + " is not positive short, usually caused by too long dict value."); } } public static boolean isPositiveShort(int i) { return (i & 0x7000) == 0; } And 0x7000 in binary: 0111 , so the value length should be less than 0001 0001 , values 4095 in decimalism. I wonder why is 0x7000, should 0x8000 ( 1000 ), support max length: 0111 (32767) be what you want? Or 32767 may be too large, I prefer use 0xE000, 0xE000 ( 1110 ), support max length: 0001 (8191) was: In the new release, Kylin will check the value length when building trie dictionary, in class TrieDictionaryBuilder method buildTrieBytes, through method: private void positiveShortPreCheck(int i, String fieldName) { if (!BytesUtil.isPositiveShort(i)) { throw new IllegalStateException(fieldName + " is not positive short, usually caused by too long dict value."); } } public static boolean isPositiveShort(int i) { return (i & 0x7000) == 0; } And 0x7000 in binary: 0111 , so the value length should be less than 0001 0001 , values 4095 in decimalism. I wonder why is 0x7000, should 0x8000: 1000 support max length: 0111 (32767) be what you want? And 32767 may be too lagrge, I prefer use 0xE000, 0xE000: 1110 , support max length: 0001 (8191) > building trie dictionary blocked on value of length over 4095 > -- > > Key: KYLIN-2956 > URL: https://issues.apache.org/jira/browse/KYLIN-2956 > Project: Kylin > Issue Type: Bug > Components: General >Reporter: Wang, Gang >Assignee: Shaofeng SHI > > In the new release, Kylin will check the value length when building trie > dictionary, in class TrieDictionaryBuilder method buildTrieBytes, through > method: > private void positiveShortPreCheck(int i, String fieldName) { > if (!BytesUtil.isPositiveShort(i)) { > throw new IllegalStateException(fieldName + " is not positive short, > usually caused by too long dict value."); > } > } > public static boolean isPositiveShort(int i) { > return (i & 0x7000) == 0; > } > And 0x7000 in binary: 0111 , so the > value length should be less than 0001 0001 , > values 4095 in decimalism. > I wonder why is 0x7000, should 0x8000 ( 1000 > ), support max length: 0111 > (32767) > be what you want? > Or 32767 may be too large, I prefer use 0xE000, 0xE000 ( > 1110 ), support max length: 0001 > (8191) > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (KYLIN-2956) building trie dictionary blocked on value of length over 4095
[ https://issues.apache.org/jira/browse/KYLIN-2956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wang, Gang reassigned KYLIN-2956: - Assignee: Shaofeng SHI (was: Wang, Gang) > building trie dictionary blocked on value of length over 4095 > -- > > Key: KYLIN-2956 > URL: https://issues.apache.org/jira/browse/KYLIN-2956 > Project: Kylin > Issue Type: Bug > Components: General >Reporter: Wang, Gang >Assignee: Shaofeng SHI > > In the new release, Kylin will check the value length when building trie > dictionary, in class TrieDictionaryBuilder method buildTrieBytes, through > method: > private void positiveShortPreCheck(int i, String fieldName) { > if (!BytesUtil.isPositiveShort(i)) { > throw new IllegalStateException(fieldName + " is not positive short, > usually caused by too long dict value."); > } > } > public static boolean isPositiveShort(int i) { > return (i & 0x7000) == 0; > } > And 0x7000 in binary: 0111 , so the > value length should be less than 0001 0001 , > values 4095 in decimalism. > I wonder why is 0x7000, should > 0x8000: 1000 > support max length: 0111 (32767) > be what you want? And 32767 may be too lagrge, I prefer use 0xE000, > 0xE000: 1110 , > support max length: 0001 (8191) > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2956) building trie dictionary blocked on value of length over 4095
[ https://issues.apache.org/jira/browse/KYLIN-2956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wang, Gang updated KYLIN-2956: -- Description: In the new release, Kylin will check the value length when building trie dictionary, in class TrieDictionaryBuilder method buildTrieBytes, through method: private void positiveShortPreCheck(int i, String fieldName) { if (!BytesUtil.isPositiveShort(i)) { throw new IllegalStateException(fieldName + " is not positive short, usually caused by too long dict value."); } } public static boolean isPositiveShort(int i) { return (i & 0x7000) == 0; } And 0x7000 in binary: 0111 , so the value length should be less than 0001 0001 , values 4095 in decimalism. I wonder why is 0x7000, should 0x8000: 1000 support max length: 0111 (32767) be what you want? And 32767 may be too lagrge, I prefer use 0xE000, 0xE000: 1110 , support max length: 0001 (8191) was: In the new release, Kylin will check the value length when building trie dictionary, in class TrieDictionaryBuilder method buildTrieBytes, through method: _private void positiveShortPreCheck(int i, String fieldName) { if (!BytesUtil.isPositiveShort(i)) { throw new IllegalStateException(fieldName + " is not positive short, usually caused by too long dict value."); } }_ _public static boolean isPositiveShort(int i) { return (i & 0x7000) == 0; } _ And 0x7000 in binary: 0111 , so the value length should be less than 0001 0001 , values 4095 in decimalism. I wonder why is 0x7000, should 0x8000: 1000 support max length: 0111 (32767) be what you want? And 32767 may be too lagrge, I prefer use 0xE000, 0xE000: 1110 , support max length: 0001 (8191) > building trie dictionary blocked on value of length over 4095 > -- > > Key: KYLIN-2956 > URL: https://issues.apache.org/jira/browse/KYLIN-2956 > Project: Kylin > Issue Type: Bug > Components: General >Reporter: Wang, Gang >Assignee: Wang, Gang > > In the new release, Kylin will check the value length when building trie > dictionary, in class TrieDictionaryBuilder method buildTrieBytes, through > method: > private void positiveShortPreCheck(int i, String fieldName) { > if (!BytesUtil.isPositiveShort(i)) { > throw new IllegalStateException(fieldName + " is not positive short, > usually caused by too long dict value."); > } > } > public static boolean isPositiveShort(int i) { > return (i & 0x7000) == 0; > } > And 0x7000 in binary: 0111 , so the > value length should be less than 0001 0001 , > values 4095 in decimalism. > I wonder why is 0x7000, should > 0x8000: 1000 > support max length: 0111 (32767) > be what you want? And 32767 may be too lagrge, I prefer use 0xE000, > 0xE000: 1110 , > support max length: 0001 (8191) > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2956) building trie dictionary blocked on value of length over 4095
[ https://issues.apache.org/jira/browse/KYLIN-2956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wang, Gang updated KYLIN-2956: -- Description: In the new release, Kylin will check the value length when building trie dictionary, in class TrieDictionaryBuilder method buildTrieBytes, through method: _private void positiveShortPreCheck(int i, String fieldName) { if (!BytesUtil.isPositiveShort(i)) { throw new IllegalStateException(fieldName + " is not positive short, usually caused by too long dict value."); } }_ _public static boolean isPositiveShort(int i) { return (i & 0x7000) == 0; } _ And 0x7000 in binary: 0111 , so the value length should be less than 0001 0001 , values 4095 in decimalism. I wonder why is 0x7000, should 0x8000: 1000 support max length: 0111 (32767) be what you want? And 32767 may be too lagrge, I prefer use 0xE000, 0xE000: 1110 , support max length: 0001 (8191) was: In the new release, Kylin will check the value length when building trie dictionary, in class _TrieDictionaryBuilder_ method _buildTrieBytes_ , through method: _private void positiveShortPreCheck(int i, String fieldName) { if (!BytesUtil.isPositiveShort(i)) { throw new IllegalStateException(fieldName + " is not positive short, usually caused by too long dict value."); } } _ _public static boolean isPositiveShort(int i) { return (i & 0x7000) == 0; }_ And 0x7000 in binary: 0111 , so the value length should be less than 0001 0001 , values 4095 in decimalism. I wonder why is 0x7000, should 0x8000: 1000 support max length: 0111 (32767) be what you want? And 32767 may be too lagrge, I prefer use 0xE000, 0xE000: 1110 , support max length: 0001 (8191) > building trie dictionary blocked on value of length over 4095 > -- > > Key: KYLIN-2956 > URL: https://issues.apache.org/jira/browse/KYLIN-2956 > Project: Kylin > Issue Type: Bug > Components: General >Reporter: Wang, Gang >Assignee: Wang, Gang > > In the new release, Kylin will check the value length when building trie > dictionary, in class TrieDictionaryBuilder method buildTrieBytes, through > method: > _private void positiveShortPreCheck(int i, String fieldName) { > if (!BytesUtil.isPositiveShort(i)) { > throw new IllegalStateException(fieldName + " is not positive short, > usually caused by too long dict value."); > } > }_ > _public static boolean isPositiveShort(int i) { > return (i & 0x7000) == 0; > } > _ > And 0x7000 in binary: 0111 , so the > value length should be less than 0001 0001 , > values 4095 in decimalism. > I wonder why is 0x7000, should > 0x8000: 1000 > support max length: 0111 (32767) > be what you want? And 32767 may be too lagrge, I prefer use 0xE000, > 0xE000: 1110 , > support max length: 0001 (8191) > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (KYLIN-2956) building trie dictionary blocked on value of length over 4095
[ https://issues.apache.org/jira/browse/KYLIN-2956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wang, Gang reassigned KYLIN-2956: - Assignee: Wang, Gang > building trie dictionary blocked on value of length over 4095 > -- > > Key: KYLIN-2956 > URL: https://issues.apache.org/jira/browse/KYLIN-2956 > Project: Kylin > Issue Type: Bug > Components: General >Reporter: Wang, Gang >Assignee: Wang, Gang > > In the new release, Kylin will check the value length when building trie > dictionary, in class _TrieDictionaryBuilder_ method _buildTrieBytes_ , > through method: > _private void positiveShortPreCheck(int i, String fieldName) { > if (!BytesUtil.isPositiveShort(i)) { > throw new IllegalStateException(fieldName + " is not positive short, > usually caused by too long dict value."); > } > } _ > _public static boolean isPositiveShort(int i) { > return (i & 0x7000) == 0; > }_ > And 0x7000 in binary: 0111 , so the > value length should be less than 0001 0001 , > values 4095 in decimalism. > I wonder why is 0x7000, should > 0x8000: 1000 > support max length: 0111 (32767) > be what you want? And 32767 may be too lagrge, I prefer use 0xE000, > 0xE000: 1110 , > support max length: 0001 (8191) > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2903) support cardinality calculation for Hive view
[ https://issues.apache.org/jira/browse/KYLIN-2903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16214668#comment-16214668 ] Wang, Gang commented on KYLIN-2903: --- Will use HQL statement: count(distinct _column_ ) to calculate _column_ cardinality. > support cardinality calculation for Hive view > - > > Key: KYLIN-2903 > URL: https://issues.apache.org/jira/browse/KYLIN-2903 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Reporter: Wang, Gang >Assignee: Wang, Gang >Priority: Minor > > Currently, Kylin leverage HCatlog to calculate column cardinality for Hive > tables. While, HCatlog does not support Hive view actually. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KYLIN-2956) building trie dictionary blocked on value of length over 4095
Wang, Gang created KYLIN-2956: - Summary: building trie dictionary blocked on value of length over 4095 Key: KYLIN-2956 URL: https://issues.apache.org/jira/browse/KYLIN-2956 Project: Kylin Issue Type: Bug Components: General Reporter: Wang, Gang In the new release, Kylin will check the value length when building trie dictionary, in class _TrieDictionaryBuilder_ method _buildTrieBytes_ , through method: _private void positiveShortPreCheck(int i, String fieldName) { if (!BytesUtil.isPositiveShort(i)) { throw new IllegalStateException(fieldName + " is not positive short, usually caused by too long dict value."); } } _ _public static boolean isPositiveShort(int i) { return (i & 0x7000) == 0; }_ And 0x7000 in binary: 0111 , so the value length should be less than 0001 0001 , values 4095 in decimalism. I wonder why is 0x7000, should 0x8000: 1000 support max length: 0111 (32767) be what you want? And 32767 may be too lagrge, I prefer use 0xE000, 0xE000: 1110 , support max length: 0001 (8191) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2764) Build the dict for UHC column with MR
[ https://issues.apache.org/jira/browse/KYLIN-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16214609#comment-16214609 ] kangkaisen commented on KYLIN-2764: --- OK. Thanks you, shaofeng! > Build the dict for UHC column with MR > - > > Key: KYLIN-2764 > URL: https://issues.apache.org/jira/browse/KYLIN-2764 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Affects Versions: v2.0.0 >Reporter: kangkaisen >Assignee: kangkaisen > Fix For: v2.3.0 > > Attachments: job-memory-after.png, job-memory-before.png > > > KYLIN-2217 has built dict for normal column with MR, but the UHC column > still build dict in JobServer. Like KYLIN-2217, we also could use MR build > dict for UHC column. which could thoroughly release the memory pressure and > improve job concurrent for JobServer as well as speed up multi UHC columns > procedure. > The MR input is the output of "Extract Fact Table Distinct Columns", the MR > output is the UHC column dict. Because it is very hard build global dict with > multi reducers, I use one reducer handle one UHC column and allocate enough > memory to the reducer. According to my test, 8G memory is enough. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2764) Build the dict for UHC column with MR
[ https://issues.apache.org/jira/browse/KYLIN-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16214604#comment-16214604 ] Shaofeng SHI commented on KYLIN-2764: - The changes in 2764 branch (together with your commit for KYLIN-2800) has passed the integration test. I have merged it to master branch. Thanks kaisen! > Build the dict for UHC column with MR > - > > Key: KYLIN-2764 > URL: https://issues.apache.org/jira/browse/KYLIN-2764 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Affects Versions: v2.0.0 >Reporter: kangkaisen >Assignee: kangkaisen > Fix For: v2.3.0 > > Attachments: job-memory-after.png, job-memory-before.png > > > KYLIN-2217 has built dict for normal column with MR, but the UHC column > still build dict in JobServer. Like KYLIN-2217, we also could use MR build > dict for UHC column. which could thoroughly release the memory pressure and > improve job concurrent for JobServer as well as speed up multi UHC columns > procedure. > The MR input is the output of "Extract Fact Table Distinct Columns", the MR > output is the UHC column dict. Because it is very hard build global dict with > multi reducers, I use one reducer handle one UHC column and allocate enough > memory to the reducer. According to my test, 8G memory is enough. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (KYLIN-2764) Build the dict for UHC column with MR
[ https://issues.apache.org/jira/browse/KYLIN-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI resolved KYLIN-2764. - Resolution: Fixed > Build the dict for UHC column with MR > - > > Key: KYLIN-2764 > URL: https://issues.apache.org/jira/browse/KYLIN-2764 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Affects Versions: v2.0.0 >Reporter: kangkaisen >Assignee: kangkaisen > Fix For: v2.3.0 > > Attachments: job-memory-after.png, job-memory-before.png > > > KYLIN-2217 has built dict for normal column with MR, but the UHC column > still build dict in JobServer. Like KYLIN-2217, we also could use MR build > dict for UHC column. which could thoroughly release the memory pressure and > improve job concurrent for JobServer as well as speed up multi UHC columns > procedure. > The MR input is the output of "Extract Fact Table Distinct Columns", the MR > output is the UHC column dict. Because it is very hard build global dict with > multi reducers, I use one reducer handle one UHC column and allocate enough > memory to the reducer. According to my test, 8G memory is enough. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2565) Upgrade Kylin to Hadoop3.0
[ https://issues.apache.org/jira/browse/KYLIN-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16214568#comment-16214568 ] Wang Cheng commented on KYLIN-2565: --- [~Sammi], Kylin on Hadoop3.0-alpha is ready to test when this issue is created. I would know if this patch is helpful for you? > Upgrade Kylin to Hadoop3.0 > -- > > Key: KYLIN-2565 > URL: https://issues.apache.org/jira/browse/KYLIN-2565 > Project: Kylin > Issue Type: New Feature >Reporter: Wang Cheng > > Hadoop3.0-alpha is released, Kylin should also keep compatible with it. Below > is the Hadoop3.0 components requirements: > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0.0+release -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-1892) Support setting a volatile range for auto merge
[ https://issues.apache.org/jira/browse/KYLIN-1892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16214547#comment-16214547 ] Yang Hao commented on KYLIN-1892: - [~Shaofengshi] Okay, I'll follow your advice of 1、3、4, and try to change the web > Support setting a volatile range for auto merge > --- > > Key: KYLIN-1892 > URL: https://issues.apache.org/jira/browse/KYLIN-1892 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Affects Versions: v1.5.2 >Reporter: fengYu >Assignee: Yang Hao > Attachments: KYLIN-1892.patch > > > We always has some data need to be amended some days later > in current kylin, once I set Auto Merge Thresholds, the segment newly build > will merge if reach Thresholds, the next day refresh will refresh merged > segemnt, which is unnecessary. > So I want to add a interval configuration means auto merge will merge > segments outside of the interval. > for example, if interval = 2, Auto Merge Thresholds=7, if 07-01 to 07-07 is > built, auto merge will not trigger, when 07-09 built success, auto merge will > trigger and merge segments from 07-01 to 07-07. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2952) dynamic cube build for time(statTime and endTime)
[ https://issues.apache.org/jira/browse/KYLIN-2952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16214522#comment-16214522 ] wenxue lin commented on KYLIN-2952: --- 3q > dynamic cube build for time(statTime and endTime) > - > > Key: KYLIN-2952 > URL: https://issues.apache.org/jira/browse/KYLIN-2952 > Project: Kylin > Issue Type: Improvement > Components: REST Service >Affects Versions: v1.6.0 > Environment: linux >Reporter: wenxue lin >Assignee: Zhong,Jason >Priority: Minor > > ex => curl -X PUT -u "ADMIN:KYLIN" -H > "Content-Type:application/json;charset=utf-8" -d > '{"startTime":150860160,"endTime":150868800,"buildType":"BUILD"}' > http://host:port/kylin/api/cubes/bi_dispatch_waiting_service_cube/rebuild > desc : > rest api param is startTime:150860160(2017-10-22) and > endTime:150868800(2017-10-23), but the actual time of building the cube > is 1 day ahead of schedule (actually 8 hours ahead of schedule) > =》【2017-10-21~2017-10-22】,But using kylinUI build without question, the > actual view kylin code, found that is because the kylin on the server side > source code for configuration of GMT rather than use the timezone of fixed > GMT + 8, and front-end UI will according to the configuration of the timezone > is transformed to the time of the page to add GMT + 8 time, then the back-end > to GMT + 8 time in into GMT time, so the kylinUI cube build time without > error, and using restAPI build cube time not making timezone 8 hours is not > accurate time difference problem。 > *for code:* > kylinProperties.js > this.getTimeZone = function () { > if (!this.timezone) { > this.timezone = this.getProperty("kylin.rest.timezone").trim(); > } > return this.timezone; > } > org.apache.kylin.cube.CubeSegment > public static String makeSegmentName(long startDate, long endDate, long > startOffset, long endOffset) { > if (startOffset != 0 || endOffset != 0) { > if (startOffset == 0 && (endOffset == 0 || endOffset == > Long.MAX_VALUE)) { > return "FULL_BUILD"; > } > return startOffset + "_" + endOffset; > } > // using time > SimpleDateFormat dateFormat = new SimpleDateFormat("MMddHHmmss"); > dateFormat.setTimeZone(TimeZone.getTimeZone("GMT")); > return dateFormat.format(startDate) + "_" + dateFormat.format(endDate); > } > org.apache.kylin.common.util.DateFormat: > public static FastDateFormat getDateFormat(String datePattern) { > FastDateFormat r = formatMap.get(datePattern); > if (r == null) { > r = FastDateFormat.getInstance(datePattern, > TimeZone.getTimeZone("GMT")); // NOTE: this must be GMT to calculate epoch > date correctly > formatMap.put(datePattern, r); > } > return r; > } -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KYLIN-2955) Fix assign a wrong table for find column method in DataModelDesc
Zhong Yanghong created KYLIN-2955: - Summary: Fix assign a wrong table for find column method in DataModelDesc Key: KYLIN-2955 URL: https://issues.apache.org/jira/browse/KYLIN-2955 Project: Kylin Issue Type: Bug Reporter: Zhong Yanghong Assignee: Zhong Yanghong -- This message was sent by Atlassian JIRA (v6.4.14#64029)