Re: Upgrading Hadoop dependencies and catching potential incompatibilities for HBase

[Enis Söztutar]

> I'm working on this under HADOOP-11804, using HBase as my test
application.
Cool, glad to see progress on this.

Enis

On Wed, Sep 28, 2016 at 12:15 PM, Sean Busbey <bus...@cloudera.com> wrote:

> On Wed, Sep 28, 2016 at 1:55 PM, Enis Söztutar <enis@gmail.com> wrote:
> > Can Hadoop please shade ALL of the dependencies (including PB) in
> Hadoop-3
> > so that we do not have this mess going forward.
> >
> > Enis
>
>
> I'm working on this under HADOOP-11804, using HBase as my test application.
>
> --
> busbey
>

Re: Upgrading Hadoop dependencies and catching potential incompatibilities for HBase

[Enis Söztutar]

Can Hadoop please shade ALL of the dependencies (including PB) in Hadoop-3
so that we do not have this mess going forward.

Enis

On Wed, Sep 28, 2016 at 11:19 AM, Jonathan Hsieh  wrote:

> You can build hbase using the profile by adding this -D setting.
>
> mvn clean test install -Dhadoop.profile=3.0
>
> At the moment the hbase's hadoop 3 profile has fails at mvn compile,
> install, and test phases. There are some issues that are patch available
> addressing these in HBase currently.
>
> compile https://issues.apache.org/jira/browse/HBASE-16711
> test https://issues.apache.org/jira/browse/HBASE-6581
> install https://issues.apache.org/jira/browse/HBASE-16712
>
> IIRC, currently hbase's hadoop3 profile currently tracks the hadoop-3.0
> SNAPSHOT artifacts.  Sean, would these dependencies changes be targeted for
> this branch and land in the next hadoop 3 alpha/beta?
> Jon.
>
> On Wed, Sep 28, 2016 at 9:56 AM, Sean Mackrory 
> wrote:
>
> > HBase folks,
> >
> > I'd like to work on upgrading some of the dependencies in Hadoop that are
> > starting to lag behind:
> >
> > jackson2 (https://issues.apache.org/jira/browse/HADOOP-12705)
> > jaxb-api (https://issues.apache.org/jira/browse/HADOOP-13659)
> > commons-configuration (https://issues.apache.org/
> jira/browse/HADOOP-13660)
> > htrace (https://issues.apache.org/jira/browse/HADOOP-13661)
> >
> > Some of these (chiefly jackson) are known to be high risk upgrades and
> have
> > caused problems for HBase in the past. I wanted to give you a heads up
> > about this work and coordinate with you to make sure I do all the testing
> > you think is necessary to discover any potential problems well in advance
> > of the breakages.
> >
> > I was going to look at just running HBase's unit tests when built against
> > my Hadoop's trunk with my changes (Should I expect any problems here? Is
> > there a branch already focused on Hadoop 3 compatibility or anything?)
> and
> > Bigtop's integration tests too. Anything else you would add to this?
> >
>
>
>
> --
> // Jonathan Hsieh (shay)
> // HBase Tech Lead, Software Engineer, Cloudera
> // j...@cloudera.com // @jmhsieh
>

Re: How to use hadoop to implement discributed lucene?

[Enis Söztutar]

Hi,

There are a lot of different approaches to distributed indexing.

Try to read up on the following (incomplete) list:
 - Elastic search
 - Solr cloud
 - Lucandra / Solandra / lucebase
 - Apache Nutch
 - Facebook's messages papers, and cassandra papers (sections related to
FB's mail search on Cassandra)

Thanks,
Enis

On Tue, Jan 10, 2012 at 9:52 AM, Adam Brown a...@hortonworks.com wrote:

 Hi Tim,

 Have you looked into this?

 http://wiki.apache.org/nutch/NutchHadoopTutorial

 Nutch can be run with Solr and run on Hadoop

 Hope that helps,

 Adam

 On Thu, Jan 5, 2012 at 11:26 PM, TmRobust 670797...@qq.com wrote:

  Hi,team
  I am a student.I used to develop an web application by lucene,that
 provides
  an solution by searching in the we system.
 
  With the index files enlarging,a single file system can't solve this
  problem.
 
  So I want to use hadoop to implement the index distributed and searching
  distributed.I try to create index directly into hdfs by lucene,but I
 fail.
 
  Now I ask for your help,how to use hadoop to implement distributed
 indexing
  and searching?
 
  Thanks
 
  --
  View this message in context:
 
 http://lucene.472066.n3.nabble.com/How-to-use-hadoop-to-implement-discributed-lucene-tp3637253p3637253.html
  Sent from the Hadoop lucene-dev mailing list archive at Nabble.com.
 



 --
 Adam Brown
 Enablement Engineer
 Hortonworks

Re: [VOTE] HADOOP-6671 - To use maven for hadoop common build

[Enis Söztutar]

On 04/01/2010 09:09 PM, Giridharan Kesavan wrote:

I  would like call for a vote to created a development branch of common
trunk to work on mavenizing hadoop common.

-Giri

   
+1 for exploring, but my previous experience with Maven made me switch 
back to ant+ivy. Maven is great, but ant+ivy is far more customizable. 
Do we have any build-specific reason that cannot be done with ivy. One 
issue is pushing releases to maven repo, but again it can be done from ivy.