Time for my two cents...
For the best understanding of Hadoop, I think Google's original papers
on MapReduce and GFS (Google File System) are still the best starting
source. If for no other reason, they were written before the the Hadoop
hype-train left the station, so they don't claim that
Can it be used for example in a web hosting application to process site
requests in the form of load balancing etc
Sent from my iPhone
On 07 Feb 2015, at 09:45, Matt Wallis ma...@madmonks.org wrote:
Hi Jonathan,
On 7 Feb 2015, at 6:20 pm, Jonathan Aquilina jaquil...@eagleeyet.net wrote:
Hi Jonathan,
On 7 Feb 2015, at 6:20 pm, Jonathan Aquilina jaquil...@eagleeyet.net wrote:
Can someone explain to me what exactly the purpose of hadoop is and what we
mean when we say big data? Is this for data storage and retrieval? Number
crunching?
Hadoop can be thought of as HTPC, High
Depends on the nature of the tasks, I'm sure you could use it for back end
processing, load balancing would come as part of the job distribution.
You probably want to check the website for the types of workloads it supports.
Matt
--
Matthew Wallis
ma...@madmonks.org
On 7 Feb 2015, at 7:48
: Jonathan Aquilina jaquil...@eagleeyet.net
To: Douglas Eadline deadl...@eadline.orgCc: Beowulf beowulf@beowulf.org
Subject: Re: [Beowulf] hadoop
Hey Douglas,
Thanks for the information, what has me curious is if it can be used for
example in applications which dont involve large amounts
infrastructure.
Joshua
-- Original Message --
Received: 09:20 AM PST, 02/07/2015
From: Jonathan Aquilina jaquil...@eagleeyet.net
To: Douglas Eadline deadl...@eadline.orgCc: Beowulf
beowulf@beowulf.org
Subject: Re: [Beowulf] hadoop
Hey Douglas,
Thanks for the information, what has me
Hey Douglas,
Thanks for the information, what has me curious is if it can be used for
example in applications which dont involve large amounts of data.
It would be great if you or anyone has any resources like ebooks are
useful websites to read up on it would be great if you could send them
Jonathan
I understand your confusion. Hadoop and Big Data have reached
overused but not well understood status years ago.
First, Hadoop started out at a MapReduce engine. This all
changed with Hadoop V2 and YARN (Yet Another Resource Negotiator)
Hadoop V2 can be considered a platform on which
Can someone explain to me what exactly the purpose of hadoop is and what
we mean when we say big data? Is this for data storage and retrieval?
Number crunching?
--
Regards,
Jonathan Aquilina
Founder Eagle Eye T
___
Beowulf mailing list,
Hey guys I was looking at the hadoop page and it got me wondering. is it
possible to cluster together storage servers? If so how efficient would a
cluster of them be?
--
Jonathan Aquilina
___
Beowulf mailing list, Beowulf@beowulf.org sponsored by
Sadly I have no experience in a cluster environment :( I do have my old
laptop and hopefully have a 2nd one soon. Is it possible to have a mixed
cluster with gluster or hadoop or any other clustering technology which is
a hetrogenous cluster in terms of linux machines and mac machines and if
Sure. With virtualisation by KVM. (although not sure about the windows and
mac) Maybe VirtualBox?
I'm currently designing a cloud computing platform that uses NFS for the
worker virtual machines (databases and webservers etc) and the local
storage can be exported using gluster or whatever.
Say
Interesting indeed. Does LVM span across multiple storage servers?
That is another issue and im not trying to open up a can of worms but what
is the advantage of using KVM over xen or even citrix xen server
On Tue, Nov 27, 2012 at 10:32 AM, Andrew Holway andrew.hol...@gmail.comwrote:
Sure.
2012/11/27 Jonathan Aquilina eagles051...@gmail.com
Interesting indeed. Does LVM span across multiple storage servers?
There is Clustered LVM but I dont think this is what your looking for. CLVM
allows you to have a shared storage target such as an iSCSI box and give
one LV to one box and
Basically my idea is to setup a green self sustaining data center. My hang
up is finding investors. Where I am electricity is super expensive and the
costs are passed onto the clients hosting with the current Local data
centers.
Would be great if there were some investors on this list, but I
On Nov 27, 2012, at 10:19 AM, Jonathan Aquilina wrote:
Sadly I have no experience in a cluster environment :( I do have my
old laptop and hopefully have a 2nd one soon. Is it possible to
have a mixed cluster with gluster or hadoop or any other clustering
technology which is a
Vincent I dont have anything setup just hatching crazy ideas which atm i
dont have time to move forward with anything :(
On Tue, Nov 27, 2012 at 1:18 PM, Vincent Diepeveen d...@xs4all.nl wrote:
On Nov 27, 2012, at 10:19 AM, Jonathan Aquilina wrote:
Sadly I have no experience in a cluster
Laptops are perfectly good for functionality testing and learning. Get back
in your box Vincent.
2012/11/27 Jonathan Aquilina eagles051...@gmail.com
Vincent I dont have anything setup just hatching crazy ideas which atm i
dont have time to move forward with anything :(
On Tue, Nov 27, 2012
All I am going to say is I have tons upon tons of learning to do in regards
to linux. Currently learning to setup a multiple domain setup for emails.
On Tue, Nov 27, 2012 at 1:24 PM, Andrew Holway andrew.hol...@gmail.comwrote:
Laptops are perfectly good for functionality testing and learning.
Don't post something ridicioulous like that.
That 64MB they use of course as with infiniband shipping a message of
64MB wil get you the full bandwidth over the network
and keep number of messages down and infiniband doesn't work at your
laptop.
On Nov 27, 2012, at 1:24 PM, Andrew Holway
Wasnt intending to open up a can of worms here, its just me thinking out
side the box and coming up with crazy and far fetched ideas
On Tue, Nov 27, 2012 at 2:14 PM, Vincent Diepeveen d...@xs4all.nl wrote:
Don't post something ridicioulous like that.
That 64MB they use of course as with
On 27/11/2012 13:14, Vincent Diepeveen wrote:
Don't post something ridicioulous like that.
That 64MB they use of course as with infiniband shipping a message of
64MB wil get you the full bandwidth over the network
and keep number of messages down and infiniband doesn't work at your
On 11/27/2012 08:14 AM, Vincent Diepeveen wrote:
Don't post something ridicioulous like that.
You both are right, so lets stop antagonizing the antagonist here.
Laptops are a reasonable place to toy around with and educate oneself
about Hadoop, but they are also not (obviously, I don't think
Trying to simulate a beowulf cluster at a laptop with everything that
belongs to it, is pretty much nonsense.
No hard feelings towards Jonathan there, he's excused.
I remember he was 10 years ago living at Cyprus (if i remember well)
busy exploring the similar things in linux :)
Speaking of
I remember he was 10 years ago living at Cyprus (if i remember well)
busy exploring the similar things in linux :)
Malta actually Vincent.
Right now I am just toying with some ideas as I have a business concept I
would like to move forward with once i finish my studies in may as
mentioned in
Lo all,
I've pinged Ellis a mail with a couple of specific Q's but I wondered if
anyone here has any experience with disco (http://discoproject.org/).
I'm still in the early days of testing but would be interested to hear
if anyone has an installation running at scale or in anger.
Thanks
Pete
On Tue, Nov 27, 2012 at 09:10:32AM +0100, Jonathan Aquilina wrote:
Hey guys I was looking at the hadoop page and it got me wondering. is it
possible to cluster together storage servers? If so how efficient would a
cluster of them be?
An interesting problem would be to use reasonably powerful
One thing that really caught my eye was seamicros new storage servers up to
5 petabytes of storage
On Tue, Nov 27, 2012 at 2:59 PM, Eugen Leitl eu...@leitl.org wrote:
On Tue, Nov 27, 2012 at 09:10:32AM +0100, Jonathan Aquilina wrote:
Hey guys I was looking at the hadoop page and it got me
On Tue, Nov 27, 2012 at 03:01:57PM +0100, Jonathan Aquilina wrote:
One thing that really caught my eye was seamicros new storage servers up to
5 petabytes of storage
Please do not top-post and please trim your replies. Message
below unchanged for illustration.
The Seamicro is not the same
not as efficient as gluster I would venture.
OrangeFS (PVFS2) has been shown to work as good as or better than
HDFS
--
Doug
2012/11/27 Jonathan Aquilina eagles051...@gmail.com
Hey guys I was looking at the hadoop page and it got me wondering. is it
possible to cluster together storage
On Tue, Nov 27, 2012 at 3:40 PM, Eugen Leitl eu...@leitl.org wrote:
Please do not top-post and please trim your replies.
Thanks for the reminder ! It bugs me too...
I would be
looking at least for an Exynos 5...
... to process code locally ...
I'm still not convinced about computational
I'm still not convinced about computational capabilities of ARM cores.
So how about letting the ARM core deal with I/O and adding a dedicated
computing unit in form of FPGA ? Not suggesting a (GP)GPU to keep in
line with the low power envelope.
What im interested in is seeing how well an
On Tue, Nov 27, 2012 at 04:14:06PM +0100, Bogdan Costescu wrote:
On Tue, Nov 27, 2012 at 3:40 PM, Eugen Leitl eu...@leitl.org wrote:
Please do not top-post and please trim your replies.
Thanks for the reminder ! It bugs me too...
I would be
looking at least for an Exynos 5...
... to
There is a Tegra 3 + GPU development platform, called CARMA.
It has an on-board SATA controller too.
Massimiliano
On Tue, Nov 27, 2012 at 7:28 AM, Eugen Leitl eu...@leitl.org wrote:
On Tue, Nov 27, 2012 at 04:14:06PM +0100, Bogdan Costescu wrote:
On Tue, Nov 27, 2012 at 3:40 PM, Eugen Leitl
On 11/27/2012 09:54 AM, Douglas Eadline wrote:
not as efficient as gluster I would venture.
OrangeFS (PVFS2) has been shown to work as good as or better than
HDFS
I'm not going to take a side on this one since they are in many respects
solving different problems in different domains, but to
On 11/27/2012 08:59 AM, Eugen Leitl wrote:
On Tue, Nov 27, 2012 at 09:10:32AM +0100, Jonathan Aquilina wrote:
Hey guys I was looking at the hadoop page and it got me wondering. is it
possible to cluster together storage servers? If so how efficient would a
cluster of them be?
An interesting
On Tue, Nov 27, 2012 at 11:13:25AM -0500, Ellis H. Wilson III wrote:
Are these problems EP such that they could be entirely Map tasks?
Not at all. This particular application is to derive optimal
feature extraction algorithms from high-resolution volumetric data
(mammal or primate connectome).
On 11/27/2012 11:34 AM, Eugen Leitl wrote:
On Tue, Nov 27, 2012 at 11:13:25AM -0500, Ellis H. Wilson III wrote:
Are these problems EP such that they could be entirely Map tasks?
Not at all. This particular application is to derive optimal
feature extraction algorithms from high-resolution
On Tue, Nov 27, 2012 at 11:50:00AM -0500, Ellis H. Wilson III wrote:
Not at all. This particular application is to derive optimal
feature extraction algorithms from high-resolution volumetric data
(mammal or primate connectome). At ~8 nm, even a mouse will
produce a mountain of structural
On 11/27/2012 10:39 AM, Massimiliano Fatica wrote:
There is a Tegra 3 + GPU development platform, called CARMA.
It has an on-board SATA controller too.
CARMA = Cuda on ARM Architecture. Never heard of it before SC12, but
heard the term plenty once I got there. Potentially useful link below.
Greg Lindahl wrote:
On Fri, Dec 26, 2008 at 05:16:04PM -0600, Gerry Creager wrote:
We've a user who has requested its installation on one of our clusters,
a high-throughput system.
You didn't say anything about what they wanted to do. Hadoop is
designed to store a lot of data, and then
*From:* Joe Landman land...@scalableinformatics.com
*To:* Jeff Layton layto...@att.net
*Cc:* Gerry Creager gerry.crea...@tamu.edu; Beowulf Mailing List
beowulf@beowulf.org
*Sent:* Saturday, December 27, 2008 11:11:20 AM
*Subject:* Re: [Beowulf] Hadoop
N.B. the recent MPI class we gave suggested
[ I can't determine anymore who I'm replying to ... ]
N.B. the recent MPI class we gave suggested that we need to re-tool it
to focus more upon Fortran than C. There was no interest in Java from
the class I polled. Some researchers want to use Matlab for their work,
but most university
*From:* Joe Landman land...@scalableinformatics.com
*To:* Jeff Layton layto...@att.net
*Cc:* Gerry Creager gerry.crea...@tamu.edu; Beowulf Mailing List
beowulf@beowulf.org
*Sent:* Saturday, December 27, 2008 11:11:20 AM
*Subject:* Re: [Beowulf] Hadoop
N.B
27, 2008 11:11:20 AM
*Subject:* Re: [Beowulf] Hadoop
N.B. the recent MPI class we gave suggested that we need to re-tool
it
to focus more upon Fortran than C. There was no interest in Java
from
the class I polled. Some researchers want to use Matlab for their
work,
but most university
On 12/29/08 7:01 AM, Gerry Creager gerry.crea...@tamu.edu wrote:
OUR users are willing to pony up the funds to buy Matlab. We're already
running Octave but they claimed they didn't know how to use it. Even
after we showed them Matlab scripts that just ran on Octave.
I use both on
:-) I didn't check, but may be I just have Fortran-G and H on my PC - as a
part of free Turnkey MVS distribution working w/(free) Hercules emulator for
IBM mainframes.
Ah... Job Control Language. Deep, deep joy.
___
Beowulf mailing list,
@beowulf.org
*Sent:* Saturday, December 27, 2008 11:11:20 AM
*Subject:* Re: [Beowulf] Hadoop
N.B. the recent MPI class we gave suggested that we need to re-tool it
to focus more upon Fortran than C. There was no interest in Java from
the class I polled. Some researchers want to use Matlab for their work
On Fri, Dec 26, 2008 at 05:16:04PM -0600, Gerry Creager wrote:
We've a user who has requested its installation on one of our clusters,
a high-throughput system.
You didn't say anything about what they wanted to do. Hadoop is
designed to store a lot of data, and then enable what we HPC people
On 27 Dec 2008, at 9:04 pm, Jeff Layton wrote:
I hate to tangent (hijack?) this subject, but I'm curious about your
class poll. Did the people who were interested in Matlab consider
Octave?
I can't speak for Joe's class, but when I've asked Matlab users here
about using Octave instead,
- Tim Cutts t...@sanger.ac.uk wrote:
I can't speak for Joe's class, but when I've asked Matlab users here
about using Octave instead, they're generally not interested. Partly
this is a somewhat irrational it doesn't have Matlab on the cover
thing, but largely it's the Matlab
- Joe Landman land...@scalableinformatics.com wrote:
Hi Joe, hope you're feeling better!
This said, I hear of Java's use in HPC every now and then.
We have a few people using Java on the clusters here,
our suspicion is mainly because that's all they've been
taught (or have taught
: Sunday, December 28, 2008 1:28:59 PM
Subject: Re: [Beowulf] Hadoop
On 12/28/08 7:17 AM, Jeff Layton layto...@att.net wrote:
I think I understand why people want the toolboxes - it makes coding easy.
From what I've seen people then stay with the prototype code and never move
to a compiled
Hi Gerry,
- Gerry Creager gerry.crea...@tamu.edu wrote:
I'm a bit concerned that it's not gonna be real compatible
with, say, Torque/Maui and Gluster
There is the Hadoop on Demand (HOD) project to integrate
Hadoop with Torque:
http://hadoop.apache.org/core/docs/current/hod.html
No idea
gerry.crea...@tamu.edu
To: Beowulf Mailing List beowulf@beowulf.org
Sent: Friday, December 26, 2008 6:16:04 PM
Subject: [Beowulf] Hadoop
The subject line says it all: Hadoop: Anyone got any experience with it
on clusters (OK, so Google does, but that really wasn't the question,
was it?).
We've
Karen,
Thanks for the clarifications.
I'm concerned about the software, but it looks like we'll install Hadoop
On Demand, as someone has already promised a user we'd do it... If
there were serious pitfalls, I might be able to slow it down some, but
simply inefficient isn't sufficient... we
are that
it's still early.
Hope this helps.
Jeff
*From:* Gerry Creager gerry.crea...@tamu.edu
*To:* Beowulf Mailing List beowulf@beowulf.org
*Sent:* Friday, December 26, 2008 6:16:04 PM
*Subject:* [Beowulf] Hadoop
Jeff Layton wrote:
BTW - I saw Karen's post about using Java with HadoopFS. Be sure to pay
attention to that since getting a good 64-bit Java implementation for
Linux is not always easy. There are a few out there (Sun has an early
access program to a 64-bit Java) but the reports I've heard
gerry.crea...@tamu.edu; Beowulf Mailing List
beowulf@beowulf.org
Sent: Saturday, December 27, 2008 11:11:20 AM
Subject: Re: [Beowulf] Hadoop
N.B. the recent MPI class we gave suggested that we need to re-tool it
to focus more upon Fortran than C. There was no interest in Java from
the class I
The subject line says it all: Hadoop: Anyone got any experience with it
on clusters (OK, so Google does, but that really wasn't the question,
was it?).
We've a user who has requested its installation on one of our clusters,
a high-throughput system. I'm a bit concerned that it's not gonna
On Fri, Dec 26, 2008 at 05:16:04PM -0600, Gerry Creager wrote:
The subject line says it all: Hadoop: Anyone got any experience with it
on clusters (OK, so Google does, but that really wasn't the question,
was it?).
Hi,
Google doesn't use Hadoop. Google published some papers on their
61 matches
Mail list logo
