Re: [VOTE] Bigtop 0.7.0 RC0
Since we are going to spin a new Apache Bigtop 0.7.0 RC1, here is my formal -1 for Apache Bigtop 0.7.0 RC0. See inline. On 10/29/2013 03:49 PM, Roman Shaposhnik wrote: On Mon, Oct 28, 2013 at 11:14 PM, Bruno Mahé bm...@apache.org wrote: I would agree with that. The only trouble is I can't repro it :-( Could you please provide additional details on the JIRA? But if it is easy to repro (and I'm just missing something) I'd agree with you. I updated the ticket with some more information. If you still cannot reproduce it, feel free to ping me and I will give you access to the instance. At some point I was wondering if it is because I am mixing the amazon ami with centos6 repos, but given every other service I try do work... Thanks for the details instructions on how to repro. It is indeed easily reproducible and it is also extremely easy to fix. Now, at this point, I'm absolutely in favor of spinning up RC1 with the following fixes in: BIGTOP-1132 and BIGTOP-1129 Both of those are really isolated fixes and here's what I'd like to propose: I respin on Wed and send out a new VOTE thread. This time, however, the voting will only go till noon on Sun 11/3. If anybody objects to that -- please let me know ASAP. Awesome! Sounds like a great plan. I came home too late to verify the patch of BIGTOP-1129, but hopefully I will be able to do so either tomorrow or Thursday. The rest of my comments inline: I can help with that. ;-) At least with the default use case. Let me know if you're still interested. Sure. Any recommendations? Basically, you can simply follow this setup docs: http://www.cloudera.com/content/cloudera-content/cloudera-docs/Search/latest/Cloudera-Search-Installation-Guide/csig_install_search.html and especially this part: http://www.cloudera.com/content/cloudera-content/cloudera-docs/Search/latest/Cloudera-Search-Installation-Guide/csig_deploy_search_solrcloud.html It also covers Hue Search app. It would very nice if somebody can help come up with Bigtop-specific wiki docs, but for now Cloudera Search is close enough. At this point Puppet code is the most reliable way to deploy Hue. Part of it has to do that Hue now depends on way more external services than it uses to -- which means that you have to get configs 'just right'. Our puppet works great for that -- but that's for a fully distributed cluster. I was not using puppet (on purpose). I will give it another shot by looking at what our puppet recipes are doing and see if any of these changes can be baked directly into our packages. Thanks! That would be appreciated. Actually, it was the elasticsearch sink for flume which had me update Apache Lucene jars. I will probably open a ticket against Apache Flume directly sometimes this week to ask for some clarification. Please do. I believe I fixed some of those issues in upcoming Flume 1.5.0 WAT? Really. I'm pretty curious at this point I had the same reaction. But this was reported as blocked by noscript. Also it seems to be a configuration activated by default: https://github.com/cloudera/hue/blob/master/desktop/conf.dist/hue.ini#L63 Do you think we can convince Hue upstream to not spy on its users by default? ;-) I guess worst case scenario -- we can always disable it downstream in Bigtop. I will start with opening a ticket with Hue and Bigtop. And as you said, worst case scenario, that flag can be disabled over here. I opened https://issues.apache.org/jira/browse/BIGTOP-1135 to track that effort. Great! Would you be willing to help with a blog post on the release? ;-) Sure. Do you have anything particular in mind? Just a bit more verbose version of the feedback you've provided on this thread, I guess. Makes sense? Makes sense. Well, there's Hue Search app. Have you had a chance to try it? Not yet. But will do. Give it a try -- its like Google, but on your data ;-) And the docs I quoted above cover it as well. Thanks, Roman. Thanks! Will take a look. Thanks, Bruno
Re: [VOTE] Bigtop 0.7.0 RC0
On 10/18/2013 09:54 PM, Roman Shaposhnik wrote: This is the seventh release for Apache Bigtop, version 0.7.0 It fixes the following issues: http://s.apache.org/Pkp *** Please download, test and vote by Fri 10/25 noon PST Note that we are voting upon the source (tag): release-0.7.0-RC0 Source and binary files: https://repository.apache.org/content/repositories/orgapachebigtop-194/org/apache/bigtop/bigtop/0.7.0/ Binary convenience artifacts: http://bigtop01.cloudera.org:8080/view/Releases/job/Bigtop-0.7.0/ Documentation on how to install (just make sure to adjust the repos for 0.7.0): https://cwiki.apache.org/confluence/display/BIGTOP/How+to+install+Hadoop+distribution+from+Bigtop+0.6.0 Maven staging repo: https://repository.apache.org/content/repositories/orgapachebigtop-194/ The tag to be voted upon: https://git-wip-us.apache.org/repos/asf?p=bigtop.git;a=commit;h=fb628180d289335dcf95641b44482fb680f11573 Bigtop's KEYS file containing PGP keys we use to sign the release: http://svn.apache.org/repos/asf/bigtop/dist/KEYS Thanks, Roman. I am not voting yet since I still have some time, but so far I am leaning toward a -1. I am learning toward a -1 because of https://issues.apache.org/jira/browse/BIGTOP-1129 and my issues with Hue. Other than that, everything I tested either just works out of the box or is nitpick. But BIGTOP-1129 is what I would consider a blocker since it is part of the basic use case of Apache Bigtop. Things I tested: * Apache Hadoop and some basic jobs * Apache HBase and Phoenix. Just basic testing * Apache Flume sending Apache Hadoop and Apache HBase logs to an Elasticsearch instance and visualized through Kibana * Apache Hue smoke tests * Everything running on OpenJDK 6 on ec2 instances Things I still want to test (or rather, things I hope I can test by Tuesday evening): * Apache Pig and datafu * Apache Solr * Load more data into Phoenix Things we could do better: * As described on BIGTOP-1129, I could not stop datanode/namenode through init scripts. * We could provide some templates for Apache Hadoop. I wasted a few hours just to get the pi job running. Thankfully we have the init script for hdfs (which needs some tweaks for the staging directory) and templates for the configuration files in our puppet modules * I enabled short-circuit in Apache HBase. Not sure if I missed something, but I got some org.apache.hadoop.security.AccessControlException: Can't continue with getBlockLocalPathInfo() authorization exceptions. From reading http://www.spaggiari.org/index.php/hbase/how-to-activate-hbase-shortcircuit it seems there are a few things we could do to make it work out of the box * Not sure what I did wrong but although I could access Hue UI, most apps I tried were not working. Ex: all shells give me the error value 222 for UID is less than the minimum UID allowed (500). And the file browser gives me the error Cannot access: /. Note: You are a Hue admin but not a HDFS superuser (which is hdfs).. Note that the first user I created was a user named ec2-user. Although it is not an hdfs super user, I would expect to have a working equivalent of what I can browse with the hdfs -ls command. Also creating a hue user named hdfs yields the same result. Note that I did not have time to dig further. * Phoenix directly embeds Apache Hadoop, Apache HBase and Apache Zookeeper jars. These jars should be symlinks. * Phoenix required me to delete some old Apache lucene jars from Apache Flume installation directory. From the output of the command mvn dependency:tree on the flume project, it appears these jars are only needed for the ElasticSearch and MorphlineSolrSink plugins. but Flume documentation for both of these plugin explicitly ask users to provide jars of Apache Lucene and Apache Solr/ElasticSearch themselves (since they may use a different version of Apache Lucene). So the dependency on Apache Lucene by Apache Flume should probably be marked as provided and we should probably provide some packages to manage these dependencies. * I still need to figure out why my instance of Hue needs access to google-analytics.com Other than that, it was an enjoyable experience to use Apache Bigtop 0.7.0RC0. Doing SQL queries through Phoenix was pretty impressive and did not require much work to setup. Also seeing Apache Hadoop and Apache HBase logs being shipped by flume to ElasticSearch and then being able to query events and create some dynamic charts on kibana was exciting! Also, since I am about to test Apache Solr, is there an equivalent to Kibana I can use for visualizing my indexed logs? Thanks, Bruno
Re: [VOTE] Bigtop 0.7.0 RC0
+1 (Basic testing went well on Centos 6.3) Anatoli On Friday, October 18, 2013 9:54 PM, Roman Shaposhnik r...@apache.org wrote: This is the seventh release for Apache Bigtop, version 0.7.0 It fixes the following issues: http://s.apache.org/Pkp *** Please download, test and vote by Fri 10/25 noon PST Note that we are voting upon the source (tag): release-0.7.0-RC0 Source and binary files: https://repository.apache.org/content/repositories/orgapachebigtop-194/org/apache/bigtop/bigtop/0.7.0/ Binary convenience artifacts: http://bigtop01.cloudera.org:8080/view/Releases/job/Bigtop-0.7.0/ Documentation on how to install (just make sure to adjust the repos for 0.7.0): https://cwiki.apache.org/confluence/display/BIGTOP/How+to+install+Hadoop+distribution+from+Bigtop+0.6.0 Maven staging repo: https://repository.apache.org/content/repositories/orgapachebigtop-194/ The tag to be voted upon: https://git-wip-us.apache.org/repos/asf?p=bigtop.git;a=commit;h=fb628180d289335dcf95641b44482fb680f11573 Bigtop's KEYS file containing PGP keys we use to sign the release: http://svn.apache.org/repos/asf/bigtop/dist/KEYS Thanks, Roman.
Re: [VOTE] Bigtop 0.7.0 RC0
On 10/18/2013 09:54 PM, Roman Shaposhnik wrote: This is the seventh release for Apache Bigtop, version 0.7.0 It fixes the following issues: http://s.apache.org/Pkp *** Please download, test and vote by Fri 10/25 noon PST Note that we are voting upon the source (tag): release-0.7.0-RC0 Source and binary files: https://repository.apache.org/content/repositories/orgapachebigtop-194/org/apache/bigtop/bigtop/0.7.0/ Binary convenience artifacts: http://bigtop01.cloudera.org:8080/view/Releases/job/Bigtop-0.7.0/ Documentation on how to install (just make sure to adjust the repos for 0.7.0): https://cwiki.apache.org/confluence/display/BIGTOP/How+to+install+Hadoop+distribution+from+Bigtop+0.6.0 Maven staging repo: https://repository.apache.org/content/repositories/orgapachebigtop-194/ The tag to be voted upon: https://git-wip-us.apache.org/repos/asf?p=bigtop.git;a=commit;h=fb628180d289335dcf95641b44482fb680f11573 Bigtop's KEYS file containing PGP keys we use to sign the release: http://svn.apache.org/repos/asf/bigtop/dist/KEYS Thanks, Roman. I haven't had time to test it yet, but should be able to get to it this week end. Would it be possible to postpone the end of the vote until Sunday? Also I could not find any jenkins job with all our tests running against this release. It would be great to put such a link in the vote email. Thanks, Bruno
[VOTE] Bigtop 0.7.0 RC0
This is the seventh release for Apache Bigtop, version 0.7.0 It fixes the following issues: http://s.apache.org/Pkp *** Please download, test and vote by Fri 10/25 noon PST Note that we are voting upon the source (tag): release-0.7.0-RC0 Source and binary files: https://repository.apache.org/content/repositories/orgapachebigtop-194/org/apache/bigtop/bigtop/0.7.0/ Binary convenience artifacts: http://bigtop01.cloudera.org:8080/view/Releases/job/Bigtop-0.7.0/ Documentation on how to install (just make sure to adjust the repos for 0.7.0): https://cwiki.apache.org/confluence/display/BIGTOP/How+to+install+Hadoop+distribution+from+Bigtop+0.6.0 Maven staging repo: https://repository.apache.org/content/repositories/orgapachebigtop-194/ The tag to be voted upon: https://git-wip-us.apache.org/repos/asf?p=bigtop.git;a=commit;h=fb628180d289335dcf95641b44482fb680f11573 Bigtop's KEYS file containing PGP keys we use to sign the release: http://svn.apache.org/repos/asf/bigtop/dist/KEYS Thanks, Roman.
