Hi Lewis,
Thanks for the reply! That's a lot of info for me to read up on (which is
great!) I'll get into it right away.
About the basic data structures used in OODT - understood. I'll look into the
structs folders and get a sense of what's going on and how to formulate it in
Avro.
I'll start reading up on all the links you've provided. And keep you updated on
my progress.
Thanks again!
--Aditya
_____________________________
From: Lewis John Mcgibbney <[email protected]>
Sent: Tuesday, February 10, 2015 11:31 PM
Subject: Re: GSoC 2015
To: <[email protected]>
Hi Aditya,On Tue, Feb 10, 2015 at 11:05 PM, Aditya Dhulipala wrote:> Hi
Lewis,>> Thanks for the info!>No problems, thanks for looking in to this one.
If this project goes aheadI can guarantee you it will be a MAJOR contribution
to OODT.>> I've been looking at the code you've posted above and trying to get
a sense> of what it does.> I'm looking through the imports and I found some
schema definitions in> XmlStructFactory & XmlRpcStructFactory make this
suitable for transfer> over xml-rpc.>Correct>> My guess is that this project
would be to write .avpr files (Avro files)> that define these file manager
structsThis would be my advice. We first define the Schema and
Protocolimplementations, then we move towards functional implementations.> &
then replace all the xml-rpc> (XmlStructFactory & XmlRpcStructFactory) with
Avro specific implementations> like this>
https://github.com/phunt/avro-rpc-quickstart>Yes, although Patrick's code is a
'little' out of date (running on 1.7.4(from memory), whereas Avro is running on
1.7.7.) Patrick has captured veryconveniently some excellent integration and
definitions for us.>> I understand what Avro is and how its better than xmlrpc
for performance> reasons. I'm interested in reading more about this topic. And
in particular> how it affects various components of OODT.>Well, this is exactly
what needs to be within a proposal for a GSoC projectso I am VERy keen to help
you out here.>> Can you point me in the right direction?>> So, some more higher
level
Javadochttps://cwiki.apache.org/confluence/display/OODT/Apache+OODT+APIs#ApacheOODTAPIs-XML-RPCYou
will also see that the data structrues in the OODT services packagese.g. File
Manager, Workflow Manager and Resource Manager are always locatedin a /structs/
package. These are the fundamental data structures we needto expose when
thinking about the Avro Protocol implementations.The basic user documentation
for File Manager can be
foundhttp://oodt.apache.org/components/maven/filemgr/user/basic.htmlThe
advanced (certainly advanced to a level we currently require)documentation for
Mile Manager can e
foundhttp://oodt.apache.org/components/maven/filemgr/development/developer.htmlI
am sure you will find the latter useful.Please write back here once you've
engaged with this, we will take it fromthere.It may also be a good idea to be
jotting down how you see a potentialworkflow coming together e.g. what tasks do
you think will need implementedin Avro RPC. This will make it much easier when
we actually come to runningand planning against a schedule of work.Thanks
againLewis
