Thank you Matei! I can go for Spark 0.3 + pre-protobuf. I understand there might be lots of new updates on the master branch. Is there any significant difference in current master with Spark 0.3 + pre-protobuf? I could also try to keep a separate branch with stable masters if that's the case. Best, Joe On 8/22/11 3:21 PM, "Matei Zaharia" <[email protected]> wrote:
>If you want the most stable versions right now, I recommend Spark 0.3 >(download it from the website or the 0.3-scala-2.9 tag in Git) and >pre-protobuf Mesos. However, the current master Spark + master Mesos >should also work OK; it's just that further commits may break things >temporarily. > >Matei > >On Aug 22, 2011, at 3:17 PM, Joe Xie wrote: > >> Hi Matei, >> Thanks for your feedback. I bet I'm using the master brach of Spark and >> pre-protobuf version of Mesos. Do you mean I should go for Spark >>(master) >> + protobuf (master). Let's me know the most stable combination of >>braches >> from Spark and pre-protobuf, I can try them out. >> >> Best, >> Joe >> >> >> On 8/22/11 3:09 PM, "Matei Zaharia" <[email protected]> wrote: >> >>> Are you using the master branch of Spark, or one of the releases (e.g. >>> 0.3)? The releases work with the pre-protobuf version of Mesos but the >>> master branch of Spark actually requires the master branch of Mesos >>>now. >>> That could cause the kind of error you're seeing. >>> >>> Matei >>> >>> On Aug 22, 2011, at 3:06 PM, Joe Xie wrote: >>> >>>> Sorry this should have been sent to user mail list. >>>> >>>> On 8/22/11 2:19 PM, "Joe Xie" <[email protected]> wrote: >>>> >>>>> Hi there: >>>>> >>>>> I has build and installed mesos (pre-protobuf branch for spark) and >>>>> spark, >>>>> and deployed them to a cluster. However, I have following error when >>>>>I >>>>> run >>>>> spark task on the mesos URL of my master, the spark task runs fine in >>>>> local mode. Any suggestion or help? Thanks a lot! >>>>> Joe >>>>> >>>>> My os: CentOS 5.2 >>>>> Python: 2.6 >>>>> Gcc: 4.2.1 >>>>> Swig:1.3.40 >>>>> ./run spark.examples.SparkPi mesos://[email protected]:5050 1 >>>>> 11/08/22 13:31:11 INFO spark.CacheTrackerActor: Registered actor on >>>>> port >>>>> 50501 >>>>> 11/08/22 13:31:11 INFO spark.MapOutputTrackerActor: Registered actor >>>>>on >>>>> port 50501 >>>>> 11/08/22 13:31:11 INFO spark.SparkContext: Starting job... >>>>> 11/08/22 13:31:11 INFO spark.CacheTracker: Registering RDD ID 0 with >>>>> cache >>>>> 11/08/22 13:31:11 INFO spark.CacheTrackerActor: Registering RDD 0 >>>>>with >>>>> 1 >>>>> partitions >>>>> 11/08/22 13:31:11 INFO spark.CacheTrackerActor: Asked for current >>>>>cache >>>>> locations >>>>> 11/08/22 13:31:12 INFO spark.MesosScheduler: Final stage: Stage 0 >>>>> 11/08/22 13:31:12 INFO spark.MesosScheduler: Parents of final stage: >>>>> List() >>>>> 11/08/22 13:31:12 INFO spark.MesosScheduler: Missing parents: List() >>>>> 11/08/22 13:31:12 INFO spark.MesosScheduler: Submitting Stage 0, >>>>>which >>>>> has >>>>> no missing parents >>>>> 11/08/22 13:31:12 INFO spark.MesosScheduler: Got a job with 1 tasks >>>>> Exception in thread "Spark scheduler" java.lang.UnsatisfiedLinkError: >>>>> org.apache.mesos.MesosSchedulerDriver.initialize()V >>>>> at org.apache.mesos.MesosSchedulerDriver.initialize(Native Method) >>>>> at >>>>> >>>>> >>>>>org.apache.mesos.MesosSchedulerDriver.<init>(MesosSchedulerDriver.java >>>>>:4 >>>>> 2) >>>>> at >>>>> >>>>> >>>>>org.apache.mesos.MesosSchedulerDriver.<init>(MesosSchedulerDriver.java >>>>>:4 >>>>> 6) >>>>> at spark.MesosScheduler$$anon$1.run(MesosScheduler.scala:94) >>>>> >>>>> Sent to wrong address previously >>>>> >>>>> >>>>>> >>>>>> >>>>>> On 8/21/11 4:30 PM, "[email protected]" >>>>>> <[email protected]> wrote: >>>>>> >>>>>>> Hi! This is the ezmlm program. I'm managing the >>>>>>> [email protected] mailing list. >>>>>>> >>>>>>> I'm working for my owner, who can be reached >>>>>>> at [email protected]. >>>>>>> >>>>>>> Acknowledgment: I have added the address >>>>>>> >>>>>>> [email protected] >>>>>>> >>>>>>> to the mesos-dev mailing list. >>>>>>> >>>>>>> Welcome to [email protected]! >>>>>>> >>>>>>> Please save this message so that you know the address you are >>>>>>> subscribed under, in case you later want to unsubscribe or change >>>>>>> your >>>>>>> subscription address. >>>>>>> >>>>>>> >>>>>>> --- Administrative commands for the mesos-dev list --- >>>>>>> >>>>>>> I can handle administrative requests automatically. Please >>>>>>> do not send them to the list address! Instead, send >>>>>>> your message to the correct command address: >>>>>>> >>>>>>> To subscribe to the list, send a message to: >>>>>>> <[email protected]> >>>>>>> >>>>>>> To remove your address from the list, send a message to: >>>>>>> <[email protected]> >>>>>>> >>>>>>> Send mail to the following for info and FAQ for this list: >>>>>>> <[email protected]> >>>>>>> <[email protected]> >>>>>>> >>>>>>> Similar addresses exist for the digest list: >>>>>>> <[email protected]> >>>>>>> <[email protected]> >>>>>>> >>>>>>> To get messages 123 through 145 (a maximum of 100 per request), >>>>>>>mail: >>>>>>> <[email protected]> >>>>>>> >>>>>>> To get an index with subject and author for messages 123-456 , >>>>>>>mail: >>>>>>> <[email protected]> >>>>>>> >>>>>>> They are always returned as sets of 100, max 2000 per request, >>>>>>> so you'll actually get 100-499. >>>>>>> >>>>>>> To receive all messages with the same subject as message 12345, >>>>>>> send a short message to: >>>>>>> <[email protected]> >>>>>>> >>>>>>> The messages should contain one line or word of text to avoid being >>>>>>> treated as sp@m, but I will ignore their content. >>>>>>> Only the ADDRESS you send to is important. >>>>>>> >>>>>>> You can start a subscription for an alternate address, >>>>>>> for example "[email protected]", just add a hyphen and your >>>>>>> address (with '=' instead of '@') after the command word: >>>>>>> <[email protected]> >>>>>>> >>>>>>> To stop subscription for this address, mail: >>>>>>> <[email protected]> >>>>>>> >>>>>>> In both cases, I'll send a confirmation message to that address. >>>>>>>When >>>>>>> you receive it, simply reply to it to complete your subscription. >>>>>>> >>>>>>> If despite following these instructions, you do not get the >>>>>>> desired results, please contact my owner at >>>>>>> [email protected]. Please be patient, my owner >>>>>>>is >>>>>>> a >>>>>>> lot slower than I am ;-) >>>>>>> >>>>>>> --- Enclosed is a copy of the request I received. >>>>>>> >>>>>>> Return-Path: <[email protected]> >>>>>>> Received: (qmail 39962 invoked by uid 99); 21 Aug 2011 23:30:07 >>>>>>>-0000 >>>>>>> Received: from nike.apache.org (HELO nike.apache.org) >>>>>>> (192.87.106.230) >>>>>>> by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 21 Aug 2011 23:30:07 >>>>>>> +0000 >>>>>>> X-ASF-Spam-Status: No, hits=-0.0 required=5.0 >>>>>>> tests=SPF_PASS >>>>>>> X-Spam-Check-By: apache.org >>>>>>> Received-SPF: pass (nike.apache.org: domain of [email protected] >>>>>>> designates >>>>>>> 67.231.145.42 as permitted sender) >>>>>>> Received: from [67.231.145.42] (HELO mx0a-00082601.pphosted.com) >>>>>>> (67.231.145.42) >>>>>>> by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 21 Aug 2011 23:30:00 >>>>>>> +0000 >>>>>>> Received: from pps.filterd (m0004348 [127.0.0.1]) >>>>>>> by m0004348.ppops.net (8.14.4/8.14.4) with SMTP id >>>>>>> p7LNTI6g015638 >>>>>>> for >>>>>>> >>>>>>> >>>>>>><mesos-dev-sc.1313966607.anahdnaofbfbnfgijcem-zhiyong=fb.com@incubat >>>>>>>or >>>>>>> .a >>>>>>> p >>>>>>> a >>>>>>> che.org>; Sun, 21 Aug 2011 16:29:18 -0700 >>>>>>> Received: from mail.thefacebook.com (corpout1.snc1.tfbnw.net >>>>>>> [66.220.144.38]) >>>>>>> by m0004348.ppops.net with ESMTP id yb5cerc88-2 >>>>>>> (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT) >>>>>>> for >>>>>>> >>>>>>> >>>>>>><mesos-dev-sc.1313966607.anahdnaofbfbnfgijcem-zhiyong=fb.com@incubat >>>>>>>or >>>>>>> .a >>>>>>> p >>>>>>> a >>>>>>> che.org>; Sun, 21 Aug 2011 16:29:18 -0700 >>>>>>> Received: from SC-MBX01-5.TheFacebook.com >>>>>>> ([fe80::f05e:c7f1:c7dd:e640]) >>>>>>> by >>>>>>> sc-hub04.TheFacebook.com ([192.168.18.212]) with mapi id >>>>>>> 14.01.0289.001; >>>>>>> Sun, >>>>>>> 21 Aug 2011 16:29:14 -0700 >>>>>>> From: Joe Xie <[email protected]> >>>>>>> To: >>>>>>> >>>>>>> >>>>>>>"mesos-dev-sc.1313966607.anahdnaofbfbnfgijcem-zhiyong=fb.com@incubat >>>>>>>or >>>>>>> .a >>>>>>> p >>>>>>> a >>>>>>> che.org" >>>>>>> >>>>>>> >>>>>>> <mesos-dev-sc.1313966607.anahdnaofbfbnfgijcem-zhiyong=fb.com@incuba >>>>>>>to >>>>>>> r. >>>>>>> a >>>>>>> p >>>>>>> ache.org> >>>>>>> Subject: Re: confirm subscribe to [email protected] >>>>>>> Thread-Topic: confirm subscribe to [email protected] >>>>>>> Thread-Index: AQHMYFO/ABVA64TXDkGEVkIetlsT6JUn85UA >>>>>>> Date: Sun, 21 Aug 2011 23:29:13 +0000 >>>>>>> Message-ID: <ca76e2a4.4259%[email protected]> >>>>>>> In-Reply-To: <[email protected]> >>>>>>> Accept-Language: en-US >>>>>>> Content-Language: en-US >>>>>>> X-MS-Has-Attach: >>>>>>> X-MS-TNEF-Correlator: >>>>>>> x-originating-ip: [192.168.18.252] >>>>>>> Content-Type: text/plain; charset="us-ascii" >>>>>>> Content-ID: <[email protected]> >>>>>>> Content-Transfer-Encoding: quoted-printable >>>>>>> MIME-Version: 1.0 >>>>>>> X-Virus-Checked: Checked by ClamAV on apache.org >>>>>>> >>>>>>> Thanks! >>>>>>> Joe >>>>>>> On 8/21/11 3:43 PM, "[email protected]" >>>>>>> <[email protected]> wrote: >>>>>>> >>>>>>>> Hi! This is the ezmlm program. I'm managing the >>>>>>>> [email protected] mailing list. >>>>>>>> >>>>>>>> I'm working for my owner, who can be reached >>>>>>>> at [email protected]. >>>>>>>> >>>>>>>> To confirm that you would like >>>>>>>> >>>>>>>> [email protected] >>>>>>>> >>>>>>>> added to the mesos-dev mailing list, please send >>>>>>>> a short reply to this address: >>>>>>>> >>>>>>>> =20 >>>>>>>> >>>>>>>> >>>>>>>>mesos-dev-sc.1313966607.anahdnaofbfbnfgijcem-zhiyong=3Dfb.com@incub >>>>>>>>at >>>>>>>> or >>>>>>>> . >>>>>>>> a >>>>>>>> p= >>>>>>> ac >>>>>>>> he.org >>>>>>>> >>>>>>>> Usually, this happens when you just hit the "reply" button. >>>>>>>> If this does not work, simply copy the address and paste it into >>>>>>>> the "To:" field of a new message. >>>>>>>> >>>>>>>> or click here: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> mailto:mesos-dev-sc.1313966607.anahdnaofbfbnfgijcem-zhiyong=3Dfb.c >>>>>>>>om >>>>>>>> @i >>>>>>>> n >>>>>>>> c >>>>>>>> u= >>>>>>> ba >>>>>>>> tor.apache.org >>>>>>>> >>>>>>>> This confirmation serves two purposes. First, it verifies that I >>>>>>>>am >>>>>>>> able >>>>>>>> to get mail through to you. Second, it protects you in case >>>>>>>>someone >>>>>>>> forges a subscription request in your name. >>>>>>>> >>>>>>>> Please note that ALL Apache dev- and user- mailing lists are >>>>>>>> publicly >>>>>>>> archived. Do familiarize yourself with Apache's public archive >>>>>>>> policy >>>>>>>> at >>>>>>>> >>>>>>>> http://www.apache.org/foundation/public-archives.html >>>>>>>> >>>>>>>> prior to subscribing and posting messages to >>>>>>>> [email protected]. >>>>>>>> If you're not sure whether or not the policy applies to this >>>>>>>>mailing >>>>>>>> list, >>>>>>>> assume it does unless the list name contains the word "private" in >>>>>>>> it. >>>>>>>> >>>>>>>> Some mail programs are broken and cannot handle long addresses. If >>>>>>>> you >>>>>>>> cannot reply to this request, instead send a message to >>>>>>>> <[email protected]> and put the >>>>>>>> entire address listed above into the "Subject:" line. >>>>>>>> >>>>>>>> >>>>>>>> --- Administrative commands for the mesos-dev list --- >>>>>>>> >>>>>>>> I can handle administrative requests automatically. Please >>>>>>>> do not send them to the list address! Instead, send >>>>>>>> your message to the correct command address: >>>>>>>> >>>>>>>> To subscribe to the list, send a message to: >>>>>>>> <[email protected]> >>>>>>>> >>>>>>>> To remove your address from the list, send a message to: >>>>>>>> <[email protected]> >>>>>>>> >>>>>>>> Send mail to the following for info and FAQ for this list: >>>>>>>> <[email protected]> >>>>>>>> <[email protected]> >>>>>>>> >>>>>>>> Similar addresses exist for the digest list: >>>>>>>> <[email protected]> >>>>>>>> <[email protected]> >>>>>>>> >>>>>>>> To get messages 123 through 145 (a maximum of 100 per request), >>>>>>>> mail: >>>>>>>> <[email protected]> >>>>>>>> >>>>>>>> To get an index with subject and author for messages 123-456 , >>>>>>>>mail: >>>>>>>> <[email protected]> >>>>>>>> >>>>>>>> They are always returned as sets of 100, max 2000 per request, >>>>>>>> so you'll actually get 100-499. >>>>>>>> >>>>>>>> To receive all messages with the same subject as message 12345, >>>>>>>> send a short message to: >>>>>>>> <[email protected]> >>>>>>>> >>>>>>>> The messages should contain one line or word of text to avoid >>>>>>>>being >>>>>>>> treated as sp@m, but I will ignore their content. >>>>>>>> Only the ADDRESS you send to is important. >>>>>>>> >>>>>>>> You can start a subscription for an alternate address, >>>>>>>> for example "[email protected]", just add a hyphen and your >>>>>>>> address (with '=3D' instead of '@') after the command word: >>>>>>>> <[email protected]> >>>>>>>> >>>>>>>> To stop subscription for this address, mail: >>>>>>>> <[email protected]> >>>>>>>> >>>>>>>> In both cases, I'll send a confirmation message to that address. >>>>>>>> When >>>>>>>> you receive it, simply reply to it to complete your subscription. >>>>>>>> >>>>>>>> If despite following these instructions, you do not get the >>>>>>>> desired results, please contact my owner at >>>>>>>> [email protected]. Please be patient, my owner >>>>>>>> is a >>>>>>>> lot slower than I am ;-) >>>>>>>> >>>>>>>> --- Enclosed is a copy of the request I received. >>>>>>>> >>>>>>>> Return-Path: <[email protected]> >>>>>>>> Received: (qmail 19986 invoked by uid 99); 21 Aug 2011 22:43:26 >>>>>>>> -0000 >>>>>>>> Received: from athena.apache.org (HELO athena.apache.org) >>>>>>>> (140.211.11.136) >>>>>>>> by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 21 Aug 2011 >>>>>>>>22:43:26 >>>>>>>> +0000 >>>>>>>> X-ASF-Spam-Status: No, hits=3D0.2 required=3D10.0 >>>>>>>> tests=3DASF_LIST_OPS,HTML_MESSAGE,SPF_PASS >>>>>>>> X-Spam-Check-By: apache.org >>>>>>>> Received-SPF: pass (athena.apache.org: domain of [email protected] >>>>>>>> designates 67.231.145.42 as permitted sender) >>>>>>>> Received: from [67.231.145.42] (HELO mx0a-00082601.pphosted.com) >>>>>>>> (67.231.145.42) >>>>>>>> by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 21 Aug 2011 >>>>>>>>22:43:19 >>>>>>>> +0000 >>>>>>>> Received: from pps.filterd (m0004347 [127.0.0.1]) >>>>>>>> by m0004347.ppops.net (8.14.4/8.14.4) with SMTP id >>>>>>>> p7LMcbSq017336 >>>>>>>> for <[email protected]>; Sun, 21 Aug 2011 >>>>>>>> 15:42:58 -0700 >>>>>>>> Received: from mail.thefacebook.com (corpout1.snc1.tfbnw.net >>>>>>>> [66.220.144.38]) >>>>>>>> by m0004347.ppops.net with ESMTP id yb0jhrkt9-1 >>>>>>>> (version=3DTLSv1/SSLv3 cipher=3DAES128-SHA bits=3D128 >>>>>>>>verify=3DNOT) >>>>>>>> for <[email protected]>; Sun, 21 Aug 2011 >>>>>>>> 15:42:58 -0700 >>>>>>>> Received: from SC-MBX01-5.TheFacebook.com >>>>>>>> ([fe80::f05e:c7f1:c7dd:e640]) >>>>>>>> by >>>>>>>> sc-hub03.TheFacebook.com ([192.168.18.198]) with mapi id >>>>>>>> 14.01.0289.001; >>>>>>>> Sun, >>>>>>>> 21 Aug 2011 15:42:53 -0700 >>>>>>>> From: Joe Xie <[email protected]> >>>>>>>> To: "[email protected]" >>>>>>>> <[email protected]> >>>>>>>> Subject: can i join this >>>>>>>> Thread-Topic: can i join this >>>>>>>> Thread-Index: AQHMYFOp36obbNljKE2n3mexQg9sQQ=3D=3D >>>>>>>> Date: Sun, 21 Aug 2011 22:42:53 +0000 >>>>>>>> Message-ID: <ca76d7f7.4256%[email protected]> >>>>>>>> Accept-Language: en-US >>>>>>>> Content-Language: en-US >>>>>>>> X-MS-Has-Attach:=20 >>>>>>>> X-MS-TNEF-Correlator: >>>>>>>> x-originating-ip: [192.168.18.252] >>>>>>>> Content-Type: multipart/alternative; >>>>>>>> boundary=3D"_000_CA76D7F74256zhiyongfbcom_" >>>>>>>> MIME-Version: 1.0 >>>>>>>> >>>>>>>> --_000_CA76D7F74256zhiyongfbcom_ >>>>>>>> Content-Type: text/plain; charset=3D"us-ascii" >>>>>>>> Content-Transfer-Encoding: quoted-printable >>>>>>>> >>>>>>>> Best, >>>>>>>> Joe >>>>>>>> >>>>>>>> --_000_CA76D7F74256zhiyongfbcom_ >>>>>>>> Content-Type: text/html; charset=3D"us-ascii" >>>>>>>> Content-ID: <[email protected]> >>>>>>>> Content-Transfer-Encoding: quoted-printable >>>>>>>> >>>>>>>> <html> >>>>>>>> <head> >>>>>>>> <meta http-equiv=3D3D"Content-Type" content=3D3D"text/html; >>>>>>>> charset=3D3Dus-ascii"=3D >>>>>>>>> >>>>>>>> </head> >>>>>>>> <body style=3D3D"word-wrap: break-word; -webkit-nbsp-mode: space; >>>>>>>> -webkit-lin=3D >>>>>>>> e-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; >>>>>>>> font-fami=3D >>>>>>>> ly: Calibri, sans-serif; "> >>>>>>>> <div>Best,</div> >>>>>>>> <div>Joe</div> >>>>>>>> </body> >>>>>>>> </html> >>>>>>>> >>>>>>>> --_000_CA76D7F74256zhiyongfbcom_-- >>>>>>> >>>>>> >>>>> >>>> >>> >> >
