[jira] [Resolved] (KYLIN-3277) Kylin should override hiveconf settings when connecting to hive using jdbc

2018-04-03 Thread Billy Liu (JIRA) 

 [ 
https://issues.apache.org/jira/browse/KYLIN-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Billy Liu resolved KYLIN-3277.
--
Resolution: Fixed

> Kylin should override hiveconf settings when connecting to hive using jdbc
> --
>
> Key: KYLIN-3277
> URL: https://issues.apache.org/jira/browse/KYLIN-3277
> Project: Kylin
>  Issue Type: Bug
>Reporter: Chuqian Yu
>Assignee: Chuqian Yu
>Priority: Major
>  Labels: patch
> Fix For: v2.4.0
>
> Attachments: 0001-KYLIN-3277.patch
>
>
> Hi, kylin developers. My cube building procedure failing at Step 2 
> "Redistribute Flat Hive Table" because Kylin always trying to submit a mr job 
> to the default yarn queue.
>  
> I have overrided the mapred.job.queue.name property in both 
> kylin_hive_conf.xml and kylin.properties but it doesn't work. 
>  
> kylin.properties
> ```
> kylin.source.hive.beeline-params=-n hive -p hive --hiveconf 
> mapred.job.queue.name=myQueue  -u 
> "jdbc:hive2://myZk:2181/;serviceDiscoveryMode=zooKeeper;" 
> ```
>  
> kylin_hive_conf.xml
> ```
>    
>    mapred.job.queue.name
>    myQueue
>    
> ```
>  
> After digging into the source code ,I found that kylin try to get the row 
> count of hive table before redistributing it. But it dose not override the 
> hive configuration when using jdbc to connect to hive server.
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KYLIN-3311) Segments overlap error (refactor write conflict exception)

2018-04-03 Thread Billy Liu (JIRA) 

 [ 
https://issues.apache.org/jira/browse/KYLIN-3311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Billy Liu resolved KYLIN-3311.
--
   Resolution: Fixed
 Assignee: liyang
Fix Version/s: v2.4.0

> Segments overlap error (refactor write conflict exception)
> --
>
> Key: KYLIN-3311
> URL: https://issues.apache.org/jira/browse/KYLIN-3311
> Project: Kylin
>  Issue Type: Bug
>  Components: Metadata
>Affects Versions: v2.3.0
>Reporter: xujing
>Assignee: liyang
>Priority: Major
>  Labels: build
> Fix For: v2.4.0
>
> Attachments: Segments_Overlap_ErrorLog.txt
>
>
> when "updateCubeWithRetry" method be called at first time ,
> line newSegs.validate();was passed .
> then 
> cube = crud.save(cube);seem with error throw exception
> write conflict to update cube  at try 0 ,will retry...
> while retry "updateCubeWithRetry" start
> line newSegs.validate();was not passed .
> throw exception 
> Segments overlap: [2018031800_2018031900] and 
> sales_order_channel[2018031800_201803
> 1900]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KYLIN-3315) allow each project to set its own source in project level override configuration

2018-04-03 Thread Billy Liu (JIRA) 

 [ 
https://issues.apache.org/jira/browse/KYLIN-3315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Billy Liu resolved KYLIN-3315.
--
   Resolution: Fixed
 Assignee: Dong Li
Fix Version/s: v2.4.0

> allow each project to set its own source in project level override 
> configuration
> 
>
> Key: KYLIN-3315
> URL: https://issues.apache.org/jira/browse/KYLIN-3315
> Project: Kylin
>  Issue Type: Improvement
>  Components: Metadata
>Reporter: Dong Li
>Assignee: Dong Li
>Priority: Major
> Fix For: v2.4.0
>
>
> Currently, all projects connect to the same source which is set in 
> kylin.properties with kylin.source.default property.
> It's better to allow each project to set its own source in project level 
> override configuration.
> As the result, we can have project A connects to JDBC, and project B connects 
> to Hive.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KYLIN-3324) NegativeArraySizeException in CreateDictionaryJob$2.getDictionary()

2018-04-03 Thread Billy Liu (JIRA) 

 [ 
https://issues.apache.org/jira/browse/KYLIN-3324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Billy Liu resolved KYLIN-3324.
--
   Resolution: Fixed
Fix Version/s: v2.4.0

> NegativeArraySizeException in CreateDictionaryJob$2.getDictionary()
> ---
>
> Key: KYLIN-3324
> URL: https://issues.apache.org/jira/browse/KYLIN-3324
> Project: Kylin
>  Issue Type: Bug
>Reporter: liyang
>Assignee: liyang
>Priority: Major
> Fix For: v2.4.0
>
>
> During cube build, got following exception:
> {noformat}
> 2018-01-31 09:58:14,982 ERROR [Scheduler 311988476 Job 
> b931b081-3b44-462b-9bce-e15bf245f1b9-862] common.HadoopShellExecutable : 
> error execute 
> HadoopShellExecutable{id=b931b081-3b44-462b-9bce-e15bf245f1b9-03, name=Build 
> Dimension Dictionary, state=RUNNING}
> java.lang.NegativeArraySizeException
>   at 
> org.apache.hadoop.io.BytesWritable.setCapacity(BytesWritable.java:144)
>   at org.apache.hadoop.io.BytesWritable.setSize(BytesWritable.java:123)
>   at org.apache.hadoop.io.BytesWritable.readFields(BytesWritable.java:179)
>   at 
> org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:2259)
>   at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2387)
>   at 
> org.apache.kylin.engine.mr.steps.CreateDictionaryJob$2.getDictionary(CreateDictionaryJob.java:87)
>   at 
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:62)
>   at 
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:49)
>   at 
> org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:66)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>   at 
> org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:62)
>   at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125)
>   at 
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:64)
>   at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125)
>   at 
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:156)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745){noformat}
> Root cause seems to be 
> https://stackoverflow.com/questions/24127304/negativearraysizeexception-when-creating-a-sequencefile-with-large-1gb-bytesw



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3277) Kylin should override hiveconf settings when connecting to hive using jdbc

2018-04-03 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/KYLIN-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424971#comment-16424971
 ] 

ASF subversion and git services commented on KYLIN-3277:


Commit 417c4b646eb28c3f5dd51378e5cac1546e65c7ff in kylin's branch 
refs/heads/master from [~nichunen]
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=417c4b6 ]

KYLIN-3277 Code review


> Kylin should override hiveconf settings when connecting to hive using jdbc
> --
>
> Key: KYLIN-3277
> URL: https://issues.apache.org/jira/browse/KYLIN-3277
> Project: Kylin
>  Issue Type: Bug
>Reporter: Chuqian Yu
>Assignee: Chuqian Yu
>Priority: Major
>  Labels: patch
> Fix For: v2.4.0
>
> Attachments: 0001-KYLIN-3277.patch
>
>
> Hi, kylin developers. My cube building procedure failing at Step 2 
> "Redistribute Flat Hive Table" because Kylin always trying to submit a mr job 
> to the default yarn queue.
>  
> I have overrided the mapred.job.queue.name property in both 
> kylin_hive_conf.xml and kylin.properties but it doesn't work. 
>  
> kylin.properties
> ```
> kylin.source.hive.beeline-params=-n hive -p hive --hiveconf 
> mapred.job.queue.name=myQueue  -u 
> "jdbc:hive2://myZk:2181/;serviceDiscoveryMode=zooKeeper;" 
> ```
>  
> kylin_hive_conf.xml
> ```
>    
>    mapred.job.queue.name
>    myQueue
>    
> ```
>  
> After digging into the source code ,I found that kylin try to get the row 
> count of hive table before redistributing it. But it dose not override the 
> hive configuration when using jdbc to connect to hive server.
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3315) allow each project to set its own source in project level override configuration

2018-04-03 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/KYLIN-3315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424974#comment-16424974
 ] 

ASF subversion and git services commented on KYLIN-3315:


Commit 5552164ba09eba989b9ddccdf3f1e4f83ed0b799 in kylin's branch 
refs/heads/master from [~lidong_sjtu]
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=5552164 ]

KYLIN-3315 allow each project to set its own source


> allow each project to set its own source in project level override 
> configuration
> 
>
> Key: KYLIN-3315
> URL: https://issues.apache.org/jira/browse/KYLIN-3315
> Project: Kylin
>  Issue Type: Improvement
>  Components: Metadata
>Reporter: Dong Li
>Priority: Major
>
> Currently, all projects connect to the same source which is set in 
> kylin.properties with kylin.source.default property.
> It's better to allow each project to set its own source in project level 
> override configuration.
> As the result, we can have project A connects to JDBC, and project B connects 
> to Hive.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3277) Kylin should override hiveconf settings when connecting to hive using jdbc

2018-04-03 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/KYLIN-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424970#comment-16424970
 ] 

ASF subversion and git services commented on KYLIN-3277:


Commit cfe84a18f3f54d405ceb394ed3b40bd4b09a692b in kylin's branch 
refs/heads/master from [~yuchuqian]
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=cfe84a1 ]

KYLIN-3277 Kylin should override hiveconf settings when connecting to hive 
using jdbc

Signed-off-by: nichunen 


> Kylin should override hiveconf settings when connecting to hive using jdbc
> --
>
> Key: KYLIN-3277
> URL: https://issues.apache.org/jira/browse/KYLIN-3277
> Project: Kylin
>  Issue Type: Bug
>Reporter: Chuqian Yu
>Assignee: Chuqian Yu
>Priority: Major
>  Labels: patch
> Fix For: v2.4.0
>
> Attachments: 0001-KYLIN-3277.patch
>
>
> Hi, kylin developers. My cube building procedure failing at Step 2 
> "Redistribute Flat Hive Table" because Kylin always trying to submit a mr job 
> to the default yarn queue.
>  
> I have overrided the mapred.job.queue.name property in both 
> kylin_hive_conf.xml and kylin.properties but it doesn't work. 
>  
> kylin.properties
> ```
> kylin.source.hive.beeline-params=-n hive -p hive --hiveconf 
> mapred.job.queue.name=myQueue  -u 
> "jdbc:hive2://myZk:2181/;serviceDiscoveryMode=zooKeeper;" 
> ```
>  
> kylin_hive_conf.xml
> ```
>    
>    mapred.job.queue.name
>    myQueue
>    
> ```
>  
> After digging into the source code ,I found that kylin try to get the row 
> count of hive table before redistributing it. But it dose not override the 
> hive configuration when using jdbc to connect to hive server.
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3324) NegativeArraySizeException in CreateDictionaryJob$2.getDictionary()

2018-04-03 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/KYLIN-3324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424972#comment-16424972
 ] 

ASF subversion and git services commented on KYLIN-3324:


Commit 343d63d19d281905c2900b250c11d5375d68503d in kylin's branch 
refs/heads/master from Li Yang
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=343d63d ]

KYLIN-3324 fix NegativeArraySizeException in 
CreateDictionaryJob$2.getDictionary()


> NegativeArraySizeException in CreateDictionaryJob$2.getDictionary()
> ---
>
> Key: KYLIN-3324
> URL: https://issues.apache.org/jira/browse/KYLIN-3324
> Project: Kylin
>  Issue Type: Bug
>Reporter: liyang
>Assignee: liyang
>Priority: Major
>
> During cube build, got following exception:
> {noformat}
> 2018-01-31 09:58:14,982 ERROR [Scheduler 311988476 Job 
> b931b081-3b44-462b-9bce-e15bf245f1b9-862] common.HadoopShellExecutable : 
> error execute 
> HadoopShellExecutable{id=b931b081-3b44-462b-9bce-e15bf245f1b9-03, name=Build 
> Dimension Dictionary, state=RUNNING}
> java.lang.NegativeArraySizeException
>   at 
> org.apache.hadoop.io.BytesWritable.setCapacity(BytesWritable.java:144)
>   at org.apache.hadoop.io.BytesWritable.setSize(BytesWritable.java:123)
>   at org.apache.hadoop.io.BytesWritable.readFields(BytesWritable.java:179)
>   at 
> org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:2259)
>   at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2387)
>   at 
> org.apache.kylin.engine.mr.steps.CreateDictionaryJob$2.getDictionary(CreateDictionaryJob.java:87)
>   at 
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:62)
>   at 
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:49)
>   at 
> org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:66)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>   at 
> org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:62)
>   at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125)
>   at 
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:64)
>   at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125)
>   at 
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:156)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745){noformat}
> Root cause seems to be 
> https://stackoverflow.com/questions/24127304/negativearraysizeexception-when-creating-a-sequencefile-with-large-1gb-bytesw



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3311) Segments overlap error (refactor write conflict exception)

2018-04-03 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/KYLIN-3311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424969#comment-16424969
 ] 

ASF subversion and git services commented on KYLIN-3311:


Commit a6bfc9d019b7df4d96d46db2b8e010323fe4fddf in kylin's branch 
refs/heads/master from Li Yang
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=a6bfc9d ]

KYLIN-3311 refactor Resource.checkAndPutResourceImpl() to throw 
WriteConflictException


> Segments overlap error (refactor write conflict exception)
> --
>
> Key: KYLIN-3311
> URL: https://issues.apache.org/jira/browse/KYLIN-3311
> Project: Kylin
>  Issue Type: Bug
>  Components: Metadata
>Affects Versions: v2.3.0
>Reporter: xujing
>Priority: Major
>  Labels: build
> Attachments: Segments_Overlap_ErrorLog.txt
>
>
> when "updateCubeWithRetry" method be called at first time ,
> line newSegs.validate();was passed .
> then 
> cube = crud.save(cube);seem with error throw exception
> write conflict to update cube  at try 0 ,will retry...
> while retry "updateCubeWithRetry" start
> line newSegs.validate();was not passed .
> throw exception 
> Segments overlap: [2018031800_2018031900] and 
> sales_order_channel[2018031800_201803
> 1900]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3315) allow each project to set its own source in project level override configuration

2018-04-03 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/KYLIN-3315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424973#comment-16424973
 ] 

ASF subversion and git services commented on KYLIN-3315:


Commit 709e5640bc1f59bbb7753951fde5f22ac941aee4 in kylin's branch 
refs/heads/master from [~lidong_sjtu]
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=709e564 ]

KYLIN-3315 allow each project to set its own source


> allow each project to set its own source in project level override 
> configuration
> 
>
> Key: KYLIN-3315
> URL: https://issues.apache.org/jira/browse/KYLIN-3315
> Project: Kylin
>  Issue Type: Improvement
>  Components: Metadata
>Reporter: Dong Li
>Priority: Major
>
> Currently, all projects connect to the same source which is set in 
> kylin.properties with kylin.source.default property.
> It's better to allow each project to set its own source in project level 
> override configuration.
> As the result, we can have project A connects to JDBC, and project B connects 
> to Hive.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (KYLIN-3308) Improvement exception in REST

2018-04-03 Thread Pan, Julian (JIRA) 

[ 
https://issues.apache.org/jira/browse/KYLIN-3308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407559#comment-16407559
 ] 

Pan, Julian edited comment on KYLIN-3308 at 4/3/18 11:08 AM:
-

I'm not sure whether the controller2 will replace controller and I should 
enhance this issue in controller. 


was (Author: julianpan):
I can lead the exception defined if needed.

BTW, I'm not sure whether the controller2 will cover it or not. 

> Improvement exception in REST
> -
>
> Key: KYLIN-3308
> URL: https://issues.apache.org/jira/browse/KYLIN-3308
> Project: Kylin
>  Issue Type: Improvement
>  Components: REST Service
>Reporter: Pan, Julian
>Assignee: Pan, Julian
>Priority: Major
>
> There are kinds of exception throw by rest call, and we need better defined 
> the exception. 
> For example:
> 1> There are three types of exception for cube equals null in CubeController:
> NotFoundException (deleteCube)
> InternalErrorException(getCube)
> BadRequestException (cloneCube)
> 2> And sometimes InternalErrorException will warp the BadRequestException in 
> CubeController.saveCubeDesc
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3001) Fix the wrong Cache key issue

2018-04-03 Thread Pan, Julian (JIRA) 

[ 
https://issues.apache.org/jira/browse/KYLIN-3001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423810#comment-16423810
 ] 

Pan, Julian commented on KYLIN-3001:


It is replace for multiple space to one space, which will reduce the sql length.

> Fix the wrong Cache key issue 
> --
>
> Key: KYLIN-3001
> URL: https://issues.apache.org/jira/browse/KYLIN-3001
> Project: Kylin
>  Issue Type: Bug
>  Components: Query Engine
>Affects Versions: v2.1.0
>Reporter: Pan, Julian
>Assignee: Pan, Julian
>Priority: Major
> Fix For: v2.3.0
>
> Attachments: KYLIN_3001_Cache_key_issue.patch
>
>
> Currently the cacheKey generate by SQLRequest getCacheKey:
> cacheKey = Lists.newArrayList(sql.replaceAll("\\s+", ""), project, offset, 
> limit, acceptPartial,
> backdoorToggles);
> Which will remove all the space and break line.
> If I have a sql:
> select a, sum(c) from table 
> --where a > 0 group by a;
> and another sql:
> select a, sum(c) from table 
> --where a > 0 
> group by a;
> The two sql will share same cache key.
> But the first sql should be break. If you query second sql, and then query 
> the first again. It will return result.
> And another issue is if remove all space, which will mix a as b and aasb.
> I think the better way is just replace multiple space to one space
> Should I create a patch? Or just provide the fix:
> cacheKey = Lists.newArrayList(sql.replaceAll("[ ]", " "), project, offset, 
> limit, acceptPartial,
> backdoorToggles);



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3311) Segments overlap error (refactor write conflict exception)

2018-04-03 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/KYLIN-3311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423568#comment-16423568
 ] 

ASF subversion and git services commented on KYLIN-3311:


Commit 3222de27c508ed3e7576330dc8c40206fe6cd84d in kylin's branch 
refs/heads/sync from Li Yang
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=3222de2 ]

KYLIN-3311 refactor Resource.checkAndPutResourceImpl() to throw 
WriteConflictException


> Segments overlap error (refactor write conflict exception)
> --
>
> Key: KYLIN-3311
> URL: https://issues.apache.org/jira/browse/KYLIN-3311
> Project: Kylin
>  Issue Type: Bug
>  Components: Metadata
>Affects Versions: v2.3.0
>Reporter: xujing
>Priority: Major
>  Labels: build
> Attachments: Segments_Overlap_ErrorLog.txt
>
>
> when "updateCubeWithRetry" method be called at first time ,
> line newSegs.validate();was passed .
> then 
> cube = crud.save(cube);seem with error throw exception
> write conflict to update cube  at try 0 ,will retry...
> while retry "updateCubeWithRetry" start
> line newSegs.validate();was not passed .
> throw exception 
> Segments overlap: [2018031800_2018031900] and 
> sales_order_channel[2018031800_201803
> 1900]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3324) NegativeArraySizeException in CreateDictionaryJob$2.getDictionary()

2018-04-03 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/KYLIN-3324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423571#comment-16423571
 ] 

ASF subversion and git services commented on KYLIN-3324:


Commit 35009d465b9c628e2be8ab9bceff7852e20a4d42 in kylin's branch 
refs/heads/sync from Li Yang
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=35009d4 ]

KYLIN-3324 fix NegativeArraySizeException in 
CreateDictionaryJob$2.getDictionary()


> NegativeArraySizeException in CreateDictionaryJob$2.getDictionary()
> ---
>
> Key: KYLIN-3324
> URL: https://issues.apache.org/jira/browse/KYLIN-3324
> Project: Kylin
>  Issue Type: Bug
>Reporter: liyang
>Assignee: liyang
>Priority: Major
>
> During cube build, got following exception:
> {noformat}
> 2018-01-31 09:58:14,982 ERROR [Scheduler 311988476 Job 
> b931b081-3b44-462b-9bce-e15bf245f1b9-862] common.HadoopShellExecutable : 
> error execute 
> HadoopShellExecutable{id=b931b081-3b44-462b-9bce-e15bf245f1b9-03, name=Build 
> Dimension Dictionary, state=RUNNING}
> java.lang.NegativeArraySizeException
>   at 
> org.apache.hadoop.io.BytesWritable.setCapacity(BytesWritable.java:144)
>   at org.apache.hadoop.io.BytesWritable.setSize(BytesWritable.java:123)
>   at org.apache.hadoop.io.BytesWritable.readFields(BytesWritable.java:179)
>   at 
> org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:2259)
>   at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2387)
>   at 
> org.apache.kylin.engine.mr.steps.CreateDictionaryJob$2.getDictionary(CreateDictionaryJob.java:87)
>   at 
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:62)
>   at 
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:49)
>   at 
> org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:66)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>   at 
> org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:62)
>   at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125)
>   at 
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:64)
>   at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125)
>   at 
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:156)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745){noformat}
> Root cause seems to be 
> https://stackoverflow.com/questions/24127304/negativearraysizeexception-when-creating-a-sequencefile-with-large-1gb-bytesw



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3315) allow each project to set its own source in project level override configuration

2018-04-03 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/KYLIN-3315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423572#comment-16423572
 ] 

ASF subversion and git services commented on KYLIN-3315:


Commit 54bdae63cfca137c02cdf8749f93d152a9ac2010 in kylin's branch 
refs/heads/sync from [~lidong_sjtu]
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=54bdae6 ]

KYLIN-3315 allow each project to set its own source


> allow each project to set its own source in project level override 
> configuration
> 
>
> Key: KYLIN-3315
> URL: https://issues.apache.org/jira/browse/KYLIN-3315
> Project: Kylin
>  Issue Type: Improvement
>  Components: Metadata
>Reporter: Dong Li
>Priority: Major
>
> Currently, all projects connect to the same source which is set in 
> kylin.properties with kylin.source.default property.
> It's better to allow each project to set its own source in project level 
> override configuration.
> As the result, we can have project A connects to JDBC, and project B connects 
> to Hive.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3277) Kylin should override hiveconf settings when connecting to hive using jdbc

2018-04-03 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/KYLIN-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423570#comment-16423570
 ] 

ASF subversion and git services commented on KYLIN-3277:


Commit 2e58857d8d6903e5a0e5dc92efdaf93e7fe1f479 in kylin's branch 
refs/heads/sync from [~nichunen]
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=2e58857 ]

KYLIN-3277 Code review


> Kylin should override hiveconf settings when connecting to hive using jdbc
> --
>
> Key: KYLIN-3277
> URL: https://issues.apache.org/jira/browse/KYLIN-3277
> Project: Kylin
>  Issue Type: Bug
>Reporter: Chuqian Yu
>Assignee: Chuqian Yu
>Priority: Major
>  Labels: patch
> Fix For: v2.4.0
>
> Attachments: 0001-KYLIN-3277.patch
>
>
> Hi, kylin developers. My cube building procedure failing at Step 2 
> "Redistribute Flat Hive Table" because Kylin always trying to submit a mr job 
> to the default yarn queue.
>  
> I have overrided the mapred.job.queue.name property in both 
> kylin_hive_conf.xml and kylin.properties but it doesn't work. 
>  
> kylin.properties
> ```
> kylin.source.hive.beeline-params=-n hive -p hive --hiveconf 
> mapred.job.queue.name=myQueue  -u 
> "jdbc:hive2://myZk:2181/;serviceDiscoveryMode=zooKeeper;" 
> ```
>  
> kylin_hive_conf.xml
> ```
>    
>    mapred.job.queue.name
>    myQueue
>    
> ```
>  
> After digging into the source code ,I found that kylin try to get the row 
> count of hive table before redistributing it. But it dose not override the 
> hive configuration when using jdbc to connect to hive server.
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3315) allow each project to set its own source in project level override configuration

2018-04-03 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/KYLIN-3315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423573#comment-16423573
 ] 

ASF subversion and git services commented on KYLIN-3315:


Commit d3ca7ef4a69cb5c957754606140df0566e41b202 in kylin's branch 
refs/heads/sync from [~lidong_sjtu]
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=d3ca7ef ]

KYLIN-3315 allow each project to set its own source


> allow each project to set its own source in project level override 
> configuration
> 
>
> Key: KYLIN-3315
> URL: https://issues.apache.org/jira/browse/KYLIN-3315
> Project: Kylin
>  Issue Type: Improvement
>  Components: Metadata
>Reporter: Dong Li
>Priority: Major
>
> Currently, all projects connect to the same source which is set in 
> kylin.properties with kylin.source.default property.
> It's better to allow each project to set its own source in project level 
> override configuration.
> As the result, we can have project A connects to JDBC, and project B connects 
> to Hive.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3277) Kylin should override hiveconf settings when connecting to hive using jdbc

2018-04-03 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/KYLIN-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423569#comment-16423569
 ] 

ASF subversion and git services commented on KYLIN-3277:


Commit 25b4b3bc06b30a4030f4949d97a05566bcbddca4 in kylin's branch 
refs/heads/sync from [~yuchuqian]
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=25b4b3b ]

KYLIN-3277 Kylin should override hiveconf settings when connecting to hive 
using jdbc

Signed-off-by: nichunen 


> Kylin should override hiveconf settings when connecting to hive using jdbc
> --
>
> Key: KYLIN-3277
> URL: https://issues.apache.org/jira/browse/KYLIN-3277
> Project: Kylin
>  Issue Type: Bug
>Reporter: Chuqian Yu
>Assignee: Chuqian Yu
>Priority: Major
>  Labels: patch
> Fix For: v2.4.0
>
> Attachments: 0001-KYLIN-3277.patch
>
>
> Hi, kylin developers. My cube building procedure failing at Step 2 
> "Redistribute Flat Hive Table" because Kylin always trying to submit a mr job 
> to the default yarn queue.
>  
> I have overrided the mapred.job.queue.name property in both 
> kylin_hive_conf.xml and kylin.properties but it doesn't work. 
>  
> kylin.properties
> ```
> kylin.source.hive.beeline-params=-n hive -p hive --hiveconf 
> mapred.job.queue.name=myQueue  -u 
> "jdbc:hive2://myZk:2181/;serviceDiscoveryMode=zooKeeper;" 
> ```
>  
> kylin_hive_conf.xml
> ```
>    
>    mapred.job.queue.name
>    myQueue
>    
> ```
>  
> After digging into the source code ,I found that kylin try to get the row 
> count of hive table before redistributing it. But it dose not override the 
> hive configuration when using jdbc to connect to hive server.
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3277) Kylin should override hiveconf settings when connecting to hive using jdbc

2018-04-03 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/KYLIN-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423559#comment-16423559
 ] 

ASF subversion and git services commented on KYLIN-3277:


Commit a2d3b5f50b4c4c49dfe7b4ead488713f1b1654b8 in kylin's branch 
refs/heads/sync from [~nichunen]
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=a2d3b5f ]

KYLIN-3277 Code review


> Kylin should override hiveconf settings when connecting to hive using jdbc
> --
>
> Key: KYLIN-3277
> URL: https://issues.apache.org/jira/browse/KYLIN-3277
> Project: Kylin
>  Issue Type: Bug
>Reporter: Chuqian Yu
>Assignee: Chuqian Yu
>Priority: Major
>  Labels: patch
> Fix For: v2.4.0
>
> Attachments: 0001-KYLIN-3277.patch
>
>
> Hi, kylin developers. My cube building procedure failing at Step 2 
> "Redistribute Flat Hive Table" because Kylin always trying to submit a mr job 
> to the default yarn queue.
>  
> I have overrided the mapred.job.queue.name property in both 
> kylin_hive_conf.xml and kylin.properties but it doesn't work. 
>  
> kylin.properties
> ```
> kylin.source.hive.beeline-params=-n hive -p hive --hiveconf 
> mapred.job.queue.name=myQueue  -u 
> "jdbc:hive2://myZk:2181/;serviceDiscoveryMode=zooKeeper;" 
> ```
>  
> kylin_hive_conf.xml
> ```
>    
>    mapred.job.queue.name
>    myQueue
>    
> ```
>  
> After digging into the source code ,I found that kylin try to get the row 
> count of hive table before redistributing it. But it dose not override the 
> hive configuration when using jdbc to connect to hive server.
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3277) Kylin should override hiveconf settings when connecting to hive using jdbc

2018-04-03 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/KYLIN-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423558#comment-16423558
 ] 

ASF subversion and git services commented on KYLIN-3277:


Commit df597a7f625ea2bb3ea103706f56934608015936 in kylin's branch 
refs/heads/sync from [~yuchuqian]
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=df597a7 ]

KYLIN-3277 Kylin should override hiveconf settings when connecting to hive 
using jdbc

Signed-off-by: nichunen 


> Kylin should override hiveconf settings when connecting to hive using jdbc
> --
>
> Key: KYLIN-3277
> URL: https://issues.apache.org/jira/browse/KYLIN-3277
> Project: Kylin
>  Issue Type: Bug
>Reporter: Chuqian Yu
>Assignee: Chuqian Yu
>Priority: Major
>  Labels: patch
> Fix For: v2.4.0
>
> Attachments: 0001-KYLIN-3277.patch
>
>
> Hi, kylin developers. My cube building procedure failing at Step 2 
> "Redistribute Flat Hive Table" because Kylin always trying to submit a mr job 
> to the default yarn queue.
>  
> I have overrided the mapred.job.queue.name property in both 
> kylin_hive_conf.xml and kylin.properties but it doesn't work. 
>  
> kylin.properties
> ```
> kylin.source.hive.beeline-params=-n hive -p hive --hiveconf 
> mapred.job.queue.name=myQueue  -u 
> "jdbc:hive2://myZk:2181/;serviceDiscoveryMode=zooKeeper;" 
> ```
>  
> kylin_hive_conf.xml
> ```
>    
>    mapred.job.queue.name
>    myQueue
>    
> ```
>  
> After digging into the source code ,I found that kylin try to get the row 
> count of hive table before redistributing it. But it dose not override the 
> hive configuration when using jdbc to connect to hive server.
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3315) allow each project to set its own source in project level override configuration

2018-04-03 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/KYLIN-3315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423561#comment-16423561
 ] 

ASF subversion and git services commented on KYLIN-3315:


Commit 69420aad854a62c137b742f353e35511d5484df7 in kylin's branch 
refs/heads/sync from [~lidong_sjtu]
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=69420aa ]

KYLIN-3315 allow each project to set its own source


> allow each project to set its own source in project level override 
> configuration
> 
>
> Key: KYLIN-3315
> URL: https://issues.apache.org/jira/browse/KYLIN-3315
> Project: Kylin
>  Issue Type: Improvement
>  Components: Metadata
>Reporter: Dong Li
>Priority: Major
>
> Currently, all projects connect to the same source which is set in 
> kylin.properties with kylin.source.default property.
> It's better to allow each project to set its own source in project level 
> override configuration.
> As the result, we can have project A connects to JDBC, and project B connects 
> to Hive.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3324) NegativeArraySizeException in CreateDictionaryJob$2.getDictionary()

2018-04-03 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/KYLIN-3324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423560#comment-16423560
 ] 

ASF subversion and git services commented on KYLIN-3324:


Commit 08a0b9295088a91ac6b34ed3951ed8fb8404ee04 in kylin's branch 
refs/heads/sync from Li Yang
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=08a0b92 ]

KYLIN-3324 fix NegativeArraySizeException in 
CreateDictionaryJob$2.getDictionary()


> NegativeArraySizeException in CreateDictionaryJob$2.getDictionary()
> ---
>
> Key: KYLIN-3324
> URL: https://issues.apache.org/jira/browse/KYLIN-3324
> Project: Kylin
>  Issue Type: Bug
>Reporter: liyang
>Assignee: liyang
>Priority: Major
>
> During cube build, got following exception:
> {noformat}
> 2018-01-31 09:58:14,982 ERROR [Scheduler 311988476 Job 
> b931b081-3b44-462b-9bce-e15bf245f1b9-862] common.HadoopShellExecutable : 
> error execute 
> HadoopShellExecutable{id=b931b081-3b44-462b-9bce-e15bf245f1b9-03, name=Build 
> Dimension Dictionary, state=RUNNING}
> java.lang.NegativeArraySizeException
>   at 
> org.apache.hadoop.io.BytesWritable.setCapacity(BytesWritable.java:144)
>   at org.apache.hadoop.io.BytesWritable.setSize(BytesWritable.java:123)
>   at org.apache.hadoop.io.BytesWritable.readFields(BytesWritable.java:179)
>   at 
> org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:2259)
>   at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2387)
>   at 
> org.apache.kylin.engine.mr.steps.CreateDictionaryJob$2.getDictionary(CreateDictionaryJob.java:87)
>   at 
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:62)
>   at 
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:49)
>   at 
> org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:66)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>   at 
> org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:62)
>   at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125)
>   at 
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:64)
>   at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125)
>   at 
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:156)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745){noformat}
> Root cause seems to be 
> https://stackoverflow.com/questions/24127304/negativearraysizeexception-when-creating-a-sequencefile-with-large-1gb-bytesw



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3311) Segments overlap error (refactor write conflict exception)

2018-04-03 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/KYLIN-3311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423557#comment-16423557
 ] 

ASF subversion and git services commented on KYLIN-3311:


Commit e465a51fab70da301571511adf4ebfb358026aea in kylin's branch 
refs/heads/sync from Li Yang
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=e465a51 ]

KYLIN-3311 refactor Resource.checkAndPutResourceImpl() to throw 
WriteConflictException


> Segments overlap error (refactor write conflict exception)
> --
>
> Key: KYLIN-3311
> URL: https://issues.apache.org/jira/browse/KYLIN-3311
> Project: Kylin
>  Issue Type: Bug
>  Components: Metadata
>Affects Versions: v2.3.0
>Reporter: xujing
>Priority: Major
>  Labels: build
> Attachments: Segments_Overlap_ErrorLog.txt
>
>
> when "updateCubeWithRetry" method be called at first time ,
> line newSegs.validate();was passed .
> then 
> cube = crud.save(cube);seem with error throw exception
> write conflict to update cube  at try 0 ,will retry...
> while retry "updateCubeWithRetry" start
> line newSegs.validate();was not passed .
> throw exception 
> Segments overlap: [2018031800_2018031900] and 
> sales_order_channel[2018031800_201803
> 1900]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3315) allow each project to set its own source in project level override configuration

2018-04-03 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/KYLIN-3315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423562#comment-16423562
 ] 

ASF subversion and git services commented on KYLIN-3315:


Commit a5841b963fa313a73db510f07ebb489fa999e1fe in kylin's branch 
refs/heads/sync from [~lidong_sjtu]
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=a5841b9 ]

KYLIN-3315 allow each project to set its own source


> allow each project to set its own source in project level override 
> configuration
> 
>
> Key: KYLIN-3315
> URL: https://issues.apache.org/jira/browse/KYLIN-3315
> Project: Kylin
>  Issue Type: Improvement
>  Components: Metadata
>Reporter: Dong Li
>Priority: Major
>
> Currently, all projects connect to the same source which is set in 
> kylin.properties with kylin.source.default property.
> It's better to allow each project to set its own source in project level 
> override configuration.
> As the result, we can have project A connects to JDBC, and project B connects 
> to Hive.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)