I was able to set both up and use them. And they work like a charm! :)
- The advantage with the C version for me was that the CSV file created, retained the field names for every field. Even though this makes it bulky, as I move my data through different processing steps, this would come in handy for me to eyeball and look for patterns or issues. - With the "avro cat" Python executable, in addition to the "--field" flag, there is this great filtering option in the command line, that allows all kinds of compound expressions. As an example, for someone else reading this thread: $ avro cat test.db --format csv --filter="r['name']>'Person 45' and r['company']>'Company 7'" Company 8,Person 8,"[u'http://myurl0.net', u'http://myurl1.net', u' http://myurl2.net']" Company 9,Person 9,"[u'http://myurl0.net', u'http://myurl1.net', u' http://myurl2.net', u'http://myurl3.net']" (The sample avro file test.db was obtained easily by executing code from here: https://github.com/matteobertozzi/Hadoop/tree/master/avro-examples) One drawback with the Python executable was that fields in the csv aren't in the schema order. Given that I would have records having few tens of fields atleast, this might mean I would have to do some reordering. I do see a FIXME in the source code, from which I understand that it is not in schema order but I don't yet understand (from the code and the comments around it) what other type of ordering has actually been chosen. I am going to choose one of these definitely, just not sure yet which one. I appreciate the help very much! This has saved and will save me a lot of time. Thanks, -Selvi On Tue, Jan 24, 2012 at 3:20 PM, selvi k <[email protected]> wrote: > I found out what the issue was: > I first needed to install snappy downloaded from here: > http://code.google.com/p/snappy/ > > After a simple ./configure, make and make install, 'easy_install avro' > completed successfully. > > I will try out both the CSV conversion options and update this thread in a > bit. > > -Selvi > > > > On Tue, Jan 24, 2012 at 2:37 PM, selvi k <[email protected]> wrote: > >> Douglas and Harsh - Thanks a lot for the immediate and detailed replies! >> Looks like both of these would work well for me. >> >> >> In order to start trying these, I have tried a few things to get started >> with Avro, but this is where I am stuck: >> >> >> 1. I first downloaded the stable version in the form of >> "avro-1.6.1.tar.gz". (I am working out all this on a Ubuntu 10.04 machine). >> >> I don't find a readme file and am not familar with installing a python >> package, so I am not sure if what I am doing is correct. After some basic >> googling, I did: >> >> avro-1.6.1$ ./setup.py build >> >> This appears to complete successfully. Then when I do this: >> >> ...avro-1.6.1$ sudo ./setup.py install >> >> I get an error message. (pasted at the end of this mail [1]) >> >> >> 2. I tried the technique suggested by Harsh, but it ends with a similar >> error as pasted below in [2] >> >> /avro$ sudo easy_install avro >> >> Then I tried to install snappy by itself: >> >> /avro$ sudo easy_install python-snappy >> >> I get the same error. >> >> Also I read that that this might help with this type of error, so I tried: >> >> avro$ sudo apt-get install python2.6-dev >> >> I ensured I have gcc and installed g++ too (because I wasn't sure what >> was needed). >> >> I did see a similar error message reported here for Avro and OS X: >> https://issues.apache.org/jira/browse/AVRO-981 >> >> Before installing g++ and python-dev, the error message I was seeing from >> easy_install python_snappy was different and shorter (attached below) [3]. >> >> >> >> >> Sorry if I should just be reading up on general Python development or >> packages or installs (and/or other things), before I should even be >> attempting to do this. I'll be doing that now to move further. But in >> case anyone might have suggestions for the errors I am seeing, that would >> be great. >> >> >> I did find this Quick Start Guide from the main Avro wiki page, but when >> I look through the Python example it is once again focussed client/server >> and RPC communication between them: >> >> https://github.com/phunt/avro-rpc-quickstart >> >> >> Also my understanding is that I must 'install' or deploy Avro before I >> can try out the C bindings suggested by Douglas. I am stating this since I >> am not exactly clear by what this meant: - "especially since the C >> bindings don't have any library dependencies to install". I am assuming it >> means, I don't need anything *beyond* a basic install of Avro. >> >> >> >> 3. With regards to the two suggested ways, would either of these >> techniques allow me to filter my data records using some sort of a >> condition on a field?(or a few fields) If not it seems like I would have >> to resort to first grepping the log file with the condition I want, and >> then using either of these two techniques to convert to CSV file. This >> would still be much better than what I am doing now, which is through >> not-so-pretty awk invocations to retrieve the fields I need (after the >> initial grep). But if the existing API, allows me to scan through the log >> file and specify conditions for fields, it might be much more efficient. I >> can imagine that I might have to use the low-level API and write a program >> to do this, but I am not sure at this point how to get started on this. >> >> >> Any pointers would be really helpful! >> >> >> Thank you, >> >> Selvi >> >> >> >> >>
