Just added the setter :) Pagerank is sending doubles whereas SSSP is sending integers. I guess we can test both of them.
2012/2/26 Edward J. Yoon <[email protected]> > What is the best example to see the performance improvement by > compression? PageRank? > > On Sun, Feb 26, 2012 at 9:49 PM, Thomas Jungblut > <[email protected]> wrote: > > @Apurv, no because of: if (conf.get(COMPRESSION_CODEC_CLASS) != null). If > > it is not mapped in the configuration, then the returned string is null. > > > > I think we should provide a setter for it so that user can specify > >> compression type and optimize their jobs. > > > > > > +1. > > > > 2012/2/26 Edward J. Yoon <[email protected]> > > > >> I think we should provide a setter for it so that user can specify > >> compression type and optimize their jobs. > >> > >> On Sun, Feb 26, 2012 at 9:29 PM, Thomas Jungblut > >> <[email protected]> wrote: > >> > You can delete the "hama.messenger.compression.class" property in > >> > hama-default.xml. > >> > > >> > 2012/2/26 Edward J. Yoon <[email protected]> > >> > > >> >> Is there currently no way to disable the message compression? > >> >> > >> >> On Fri, Feb 24, 2012 at 4:50 PM, Thomas Jungblut > >> >> <[email protected]> wrote: > >> >> > I have reworked the Avro testcase to test these issues. > >> >> > (org.apache.hama.bsp.message.TestAvroMessageManager). > >> >> > > >> >> > There I am sending 3 types of messages, booleans, doubles and ints. > >> They > >> >> > are packed into a single bundle, compressed and transfered and > back at > >> >> home > >> >> > they get deserialized. > >> >> > The boolean package seem to get to the desitnation correctly since > B > >> is > >> >> the > >> >> > first in the alphabet. > >> >> > However, it seems that Integers and Doubles aren't transferred > >> correctly: > >> >> > EOF Exception while deserializing. > >> >> > > >> >> > Don't waste your time with it, I write him a mail later on. > >> >> > The JNI version works correctly and is just a bit slower on > >> compression, > >> >> > whereas it is faster on decompression. > >> >> > > >> >> > No need to waste the time ;) > >> >> > > >> >> > 2012/2/24 Apurv Verma <[email protected]> > >> >> > > >> >> >> OK, I see it now. You mean to say the JNI-Snappy which is in > >> org.xerial > >> >> >> package works correctly but not Dain's snappy which is in > org.iq80 , > >> >> which > >> >> >> is pure Java. > >> >> >> > >> >> >> I would try to check all of these after my exams, once again.(Day > >> After > >> >> >> tomorrow) I remember that I used 0.3-Incubating version of Dain's > >> snappy > >> >> >> and it seemed to work fine. I was able to run Pi Example with it. > >> >> >> > >> >> >> Now if we are to use the JNI version of snappy it might be slow so > >> >> instead > >> >> >> of using it as a default compressor, we would like to give option > of > >> >> >> choosing between compression and no-compression? > >> >> >> > >> >> >> > >> >> >> > >> >> >> -- > >> >> >> thanks and regards, > >> >> >> > >> >> >> Apurv Verma > >> >> >> B. Tech.(CSE) > >> >> >> IIT- Ropar > >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> >> On Fri, Feb 24, 2012 at 4:36 AM, Apurv Verma <[email protected]> > >> wrote: > >> >> >> > >> >> >> > What is buggy in Dain's snappy, can you let me know? > >> >> >> > > >> >> >> > -- > >> >> >> > thanks and regards, > >> >> >> > > >> >> >> > Apurv Verma > >> >> >> > B. Tech.(CSE) > >> >> >> > IIT- Ropar > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > On Thu, Feb 23, 2012 at 5:06 PM, Thomas Jungblut < > >> >> >> > [email protected]> wrote: > >> >> >> > > >> >> >> >> It seems that Dain's snappy is buggy. I tried the Hadoop codec > as > >> >> well > >> >> >> as > >> >> >> >> directly using streams. However, I just changed it to the > xerial > >> >> version > >> >> >> >> in > >> >> >> >> POM and it now works correctly. > >> >> >> >> > >> >> >> >> I will contact dain and try to sort out things. Please update > the > >> >> trunk > >> >> >> >> and rebuild. > >> >> >> >> Sorry for the inconvenience. > >> >> >> >> > >> >> >> >> 2012/2/23 Thomas Jungblut <[email protected]> > >> >> >> >> > >> >> >> >> > Yep, maybe this is the issue. In the Maven POM is the dain > >> >> version, in > >> >> >> >> > impor statements the xerial version. Sorry for the > >> inconsistency. I > >> >> >> fix > >> >> >> >> it. > >> >> >> >> > > >> >> >> >> > > >> >> >> >> > 2012/2/23 Thomas Jungblut <[email protected]> > >> >> >> >> > > >> >> >> >> >> Wait a minute, I guess this is the wrong library. > >> >> >> >> >> > >> >> >> >> >> My intention was to use the > https://github.com/dain/snappylib, > >> >> >> since > >> >> >> >> >> it is faster than the JNI port. > >> >> >> >> >> I investigate it after launch. > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> 2012/2/23 Edward J. Yoon <[email protected]> > >> >> >> >> >> > >> >> >> >> >>> Hmm, the snappy-0.2.jar is already in lib folder. > >> >> >> >> >>> > >> >> >> >> >>> edward@slave:~/workspace/hama-trunk$ ls lib/ > >> >> >> >> >>> ant-1.7.1.jar commons-configuration-1.6.jar > >> >> >> >> >>> hadoop-test-1.0.0.jar jetty-util-6.1.14.jar > >> >> >> >> >>> netty-3.2.6.Final.jar zookeeper-3.3.3.jar > >> >> >> >> >>> ant-launcher-1.7.1.jar commons-lang-2.6.jar > >> >> >> >> >>> jackson-core-asl-1.9.2.jar jsp-2.1-6.1.14.jar > >> >> >> >> >>> servlet-api-6.0.32.jar > >> >> >> >> >>> avro-1.6.0.jar commons-logging-1.1.1.jar > >> >> >> >> >>> jackson-mapper-asl-1.9.2.jar jsp-api-2.1-6.1.14.jar > >> >> >> >> >>> slf4j-api-1.5.8.jar > >> >> >> >> >>> avro-ipc-1.6.0.jar guava-10.0.1.jar > >> >> >> >> >>> jetty-6.1.14.jar junit-4.8.1.jar > >> >> >> >> >>> slf4j-log4j12-1.5.8.jar > >> >> >> >> >>> commons-cli-1.2.jar hadoop-core-1.0.0.jar > >> >> >> >> >>> jetty-annotations-6.1.14.jar log4j-1.2.16.jar > >> >> snappy-0.2.jar > >> >> >> >> >>> > >> >> >> >> >>> > >> >> >> >> >>> On Thu, Feb 23, 2012 at 4:51 PM, Thomas Jungblut > >> >> >> >> >>> <[email protected]> wrote: > >> >> >> >> >>> > You have to update your jars, Snappy is a new dependency > for > >> >> >> >> >>> compression. > >> >> >> >> >>> > > >> >> >> >> >>> > 2012/2/23 Edward J. Yoon <[email protected]> > >> >> >> >> >>> > > >> >> >> >> >>> >> I received below error message with pi example. > >> >> >> >> >>> >> > >> >> >> >> >>> >> 12/02/23 15:26:16 DEBUG sync.ZooKeeperSyncClientImpl: > >> >> >> >> leaveBarrier() > >> >> >> >> >>> >> at superstep:-1 znode size: (0) znodes:[] > >> >> >> >> >>> >> 12/02/23 15:26:16 WARN bsp.GroomServer: Error running > child > >> >> >> >> >>> >> java.lang.NoClassDefFoundError: > >> >> >> org/xerial/snappy/SnappyInputStream > >> >> >> >> >>> >> at java.lang.Class.forName0(Native Method) > >> >> >> >> >>> >> at java.lang.Class.forName(Class.java:247) > >> >> >> >> >>> >> at > >> >> >> >> >>> >> > >> >> >> >> >>> > >> >> >> >> > >> >> >> > >> >> > >> > org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:819) > >> >> >> >> >>> >> at > >> >> >> >> >>> >> > >> >> >> >> >>> > >> >> >> >> > >> >> >> > >> >> > >> > org.apache.hama.bsp.message.compress.BSPMessageCompressorFactory.getCompressor(BSPMessageCompressorFactory.java:37) > >> >> >> >> >>> >> at > >> >> >> >> >>> >> > >> >> >> >> >>> > >> >> >> >> > >> >> >> > >> >> > >> > org.apache.hama.bsp.message.CompressableMessageManager.initCompression(CompressableMessageManager.java:36) > >> >> >> >> >>> >> at > >> >> >> >> >>> >> > >> >> >> >> >>> > >> >> >> >> > >> >> >> > >> >> > >> > org.apache.hama.bsp.message.AvroMessageManagerImpl.init(AvroMessageManagerImpl.java:60) > >> >> >> >> >>> >> at > >> >> >> >> org.apache.hama.bsp.BSPPeerImpl.<init>(BSPPeerImpl.java:145) > >> >> >> >> >>> >> at > >> >> >> >> >>> >> > >> >> >> >> >>> > >> >> >> >> > >> >> > org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:969) > >> >> >> >> >>> >> Caused by: java.lang.ClassNotFoundException: > >> >> >> >> >>> >> org.xerial.snappy.SnappyInputStream > >> >> >> >> >>> >> at > >> >> java.net.URLClassLoader$1.run(URLClassLoader.java:202) > >> >> >> >> >>> >> at > >> java.security.AccessController.doPrivileged(Native > >> >> >> >> Method) > >> >> >> >> >>> >> at > >> >> >> >> java.net.URLClassLoader.findClass(URLClassLoader.java:190) > >> >> >> >> >>> >> at > >> >> java.lang.ClassLoader.loadClass(ClassLoader.java:306) > >> >> >> >> >>> >> at > >> >> >> >> >>> > sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > >> >> >> >> >>> >> at > >> >> java.lang.ClassLoader.loadClass(ClassLoader.java:247) > >> >> >> >> >>> >> > >> >> >> >> >>> >> > >> >> >> >> >>> >> -- > >> >> >> >> >>> >> Best Regards, Edward J. Yoon > >> >> >> >> >>> >> @eddieyoon > >> >> >> >> >>> >> > >> >> >> >> >>> > > >> >> >> >> >>> > > >> >> >> >> >>> > > >> >> >> >> >>> > -- > >> >> >> >> >>> > Thomas Jungblut > >> >> >> >> >>> > Berlin <[email protected]> > >> >> >> >> >>> > >> >> >> >> >>> > >> >> >> >> >>> > >> >> >> >> >>> -- > >> >> >> >> >>> Best Regards, Edward J. Yoon > >> >> >> >> >>> @eddieyoon > >> >> >> >> >>> > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> -- > >> >> >> >> >> Thomas Jungblut > >> >> >> >> >> Berlin <[email protected]> > >> >> >> >> >> > >> >> >> >> > > >> >> >> >> > > >> >> >> >> > > >> >> >> >> > -- > >> >> >> >> > Thomas Jungblut > >> >> >> >> > Berlin <[email protected]> > >> >> >> >> > > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> -- > >> >> >> >> Thomas Jungblut > >> >> >> >> Berlin <[email protected]> > >> >> >> >> > >> >> >> > > >> >> >> > > >> >> >> > >> >> > > >> >> > > >> >> > > >> >> > -- > >> >> > Thomas Jungblut > >> >> > Berlin <[email protected]> > >> >> > >> >> > >> >> > >> >> -- > >> >> Best Regards, Edward J. Yoon > >> >> @eddieyoon > >> >> > >> > > >> > > >> > > >> > -- > >> > Thomas Jungblut > >> > Berlin <[email protected]> > >> > >> > >> > >> -- > >> Best Regards, Edward J. Yoon > >> @eddieyoon > >> > > > > > > > > -- > > Thomas Jungblut > > Berlin <[email protected]> > > > > -- > Best Regards, Edward J. Yoon > @eddieyoon > -- Thomas Jungblut Berlin <[email protected]>
