Hi zeng kong,

Thanks for the feedback to Kylin!

Regarding the JAR conflict of calcite on tomcat 8.5, we're trying to fix it
in KYLIN-3676;

For the question that "why not separate Hadoop 3 upgrade and Hbase 2
upgrade", this is because we don't have many resources to support so many
combinations. As you know, even for Hadoop 2 + HBase 1, we provide two
binaries:
- Hadoop 2 HBase 1.x (community version API)
- Hadoop 2 + CDH 5.7 + (CDH version API)

If we separate Hadoop and HBase upgrades, there will be more combinations:
-  Hadoop 2 + HBase 2
-  Hadoop 2 + HBase 2 CDH API
 - Hadoop 3 + HBase 1.x
 - Hadoop 3 + HBase 2

We can't cover all these combinations, so we will focus on the major ones.
Besides, the pre-built binary packages are just for user's convenience. If
it doesn't match a specific case, the user can compile and built it with
their own versions, this is how open source runs. Just try to make your own
binary package, it is not that difficult to do.

Best regards,

Shaofeng Shi 史少锋
Apache Kylin PMC
Work email: shaofeng....@kyligence.io
Kyligence Inc: https://kyligence.io/

Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
Join Kylin user mail group: user-subscr...@kylin.apache.org
Join Kylin dev mail group: dev-subscr...@kylin.apache.org




zeng.kong <zeng.k...@ucarinc.com> 于2018年10月29日周一 下午1:27写道:

> Hi ShaoFeng Shi、Lijun Cao,
>
>  
> 针对前面提到的几个问题,我再做一下补充吧,我们目前的正在使用的hadoop版本是2.7.6,hive版本是1.2.1,hbase是2.0.1,所以我基于kylin-hadoop3将hive的版本依赖进行降级,然后重新编译了一下,用编译后的kylin-source-hive-2.5.0.jar替换掉原有的,kylin中的hive相关操作可以正常运行,后续有针对spark构建的问题编写了一点代码,使得spark构建成功。我认为这些不应该会引起tomcat类加载问题,不知道两位大神有没有好的见解。
>    另外我在hbase官网上看hbase-2.0.1 on Hadoop3
> 还没有进行很好的测试,为什么不将hadoop和hbase的升级分成两步进行呢?
>
>  Best regards
>    Zeng.kong
>
> -----邮件原件-----
> 发件人: ShaoFeng Shi [mailto:shaofeng...@apache.org]
> 发送时间: 2018年10月29日 11:07
> 收件人: dev <dev@kylin.apache.org>
> 主题: Re: apache-kylin-2.5.0-bin-hadoop3版使用中发现的问题
>
> Hi zeng kong,
>
> Thanks for the reporting; We heard Tomcat has changed the classloading
> policy in 8.5; But when we test it on Hadoop 3, didn't see the conflict,
> maybe we didn't verify enough SQL;  Would you like to report a JIRA on the
> problem you found? We will check it when recovering the Hadoop 3 cluster.
>
> Lijun Cao <641507...@qq.com> 于2018年10月29日周一 上午10:30写道:
>
> > Hi zeng.kong:
> >
> > Kylin for hadoop3 is not support Spark cubing for now because the
> > Spark hasn’t declared that it is support Hadoop 3 explicitly.
> >
> > The time to merge Hadoop 3 support into master branch is not
> > determined due to the issue mentioned above and the users of Hadoop 3
> > is not large so far.
> >
> > Best regards
> >
> > Lijun Cao
> >
> > > 在 2018年10月25日,09:58,zeng.kong <zeng.k...@ucarinc.com> 写道:
> > >
> > >   Hi,
> > >
> > >      我使用的是
> > > <
> > https://dist.apache.org/repos/dist/dev/kylin/apache-kylin-2.5.0-rc2/ap
> > ache-
> > > kylin-2.5.0-bin-hadoop3.tar.gz> apache-kylin-2.5.0-bin-hadoop3版本
> > >
> > >      我的环境搭建方式是hive和hbase使用不同的filesystem,在cube构建过程中如
> > > 果选择spark引擎构建的话出了以下两个问题:
> > >
> > > 1、  在进行到Build Cube with Spark步骤时出现hbase-config.xml查询的filesystem
> > > 与其路径所在的filesystem不一致,因为其在前面步骤放到了hive所在的hdfs,而在这
> > > 步获取的时候用的是hbase所在的filesystem
> > >
> > > 2、  在进行到Convert Cuboid Data to HFile步骤时出现缺少hbase相关jar包的问题
> > >
> > > 另外在构建承购后使用inner进行数据查询时,其sql解析不正确,导致扫描大量hbase
> > > 记录,主要原因是在tomcat调整为8.5.33后其类的加载顺序出现变化,使得calcite重
> > > 写的代码得不到加载。
> > >
> > >
> > >
> > > 以上问题是我在使用过程中发现的,不确定是否与我搭建方式有关,另外此版本会在何
> > > 时合并至master?
> > >
> > >
> > >
> > >
> > > 谢谢!
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> >
> >
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>

Reply via email to