[
https://issues.apache.org/jira/browse/SAMZA-375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14108487#comment-14108487
]
Jon Bringhurst commented on SAMZA-375:
--------------------------------------
Just to clarify the state of this, the job is running, but the tasks are not.
Once the env vars are setup properly, it seems like it might work:
{noformat}
2014-08-24 11:20:57 JobRunner [INFO] job factory:
org.apache.samza.job.mesos.MesosJobFactory
2014-08-24 11:20:57,216:14126(0x7f52c940d700):ZOO_INFO@log_env@712: Client
environment:zookeeper.version=zookeeper C client 3.4.5
2014-08-24 11:20:57,216:14126(0x7f52c940d700):ZOO_INFO@log_env@716: Client
environment:host.name=localhost.localdomain
2014-08-24 11:20:57,216:14126(0x7f52c940d700):ZOO_INFO@log_env@723: Client
environment:os.name=Linux
2014-08-24 11:20:57,216:14126(0x7f52c940d700):ZOO_INFO@log_env@724: Client
environment:os.arch=3.10.0-123.6.3.el7.x86_64
2014-08-24 11:20:57,216:14126(0x7f52c940d700):ZOO_INFO@log_env@725: Client
environment:os.version=#1 SMP Wed Aug 6 21:12:36 UTC 2014
2014-08-24 11:20:57,216:14126(0x7f52c940d700):ZOO_INFO@log_env@733: Client
environment:user.name=jbringhu
2014-08-24 11:20:57,217:14126(0x7f52c940d700):ZOO_INFO@log_env@741: Client
environment:user.home=/home/jbringhu
2014-08-24 11:20:57,217:14126(0x7f52c940d700):ZOO_INFO@log_env@753: Client
environment:user.dir=/home/jbringhu/samza-dev/hello-samza
2014-08-24 11:20:57,217:14126(0x7f52c940d700):ZOO_INFO@zookeeper_init@786:
Initiating client connection, host=127.0.0.1:2181 sessionTimeout=10000
watcher=0x7f52cefa2880 sessionId=0 sessionPasswd=<null> context=0x7f52a8001070
flags=0
2014-08-24 11:20:57,231:14126(0x7f52c7c0a700):ZOO_INFO@check_events@1703:
initiated connection to server [127.0.0.1:2181]
2014-08-24 11:20:57 SamzaScheduler [INFO] No mesos.executor.count specified.
Defaulting to one container.
2014-08-24 11:20:57,251:14126(0x7f52c7c0a700):ZOO_INFO@check_events@1750:
session establishment complete on server [127.0.0.1:2181],
sessionId=0x148089b0c550005, negotiated timeout=10000
2014-08-24 11:20:57 VerifiableProperties [INFO] Verifying properties
2014-08-24 11:20:57 VerifiableProperties [INFO] Property batch.num.messages is
overridden to 1
2014-08-24 11:20:57 VerifiableProperties [INFO] Property client.id is
overridden to samza_admin-wikipedia_feed-1-1408893657274-0
2014-08-24 11:20:57 VerifiableProperties [INFO] Property metadata.broker.list
is overridden to localhost:9092
2014-08-24 11:20:57 VerifiableProperties [INFO] Property producer.type is
overridden to sync
2014-08-24 11:20:57 VerifiableProperties [INFO] Verifying properties
2014-08-24 11:20:57 VerifiableProperties [INFO] Property client.id is
overridden to samza_admin-wikipedia_feed-1-1408893657274-0
2014-08-24 11:20:57 VerifiableProperties [INFO] Property group.id is overridden
to undefined-samza-consumer-group-8ef2ae53-c3ac-42e9-99d9-1d849d59b6a6
2014-08-24 11:20:57 VerifiableProperties [INFO] Property zookeeper.connect is
overridden to localhost:2181/
2014-08-24 11:20:57 Util$ [INFO] Instantiating type
org.apache.samza.container.grouper.stream.GroupByPartitionFactory to build
SystemStreamPartition groupings
2014-08-24 11:20:57 Util$ [INFO] SystemStreamPartitionGrouper
org.apache.samza.container.grouper.stream.GroupByPartition@e1abbbd has grouped
the SystemStreamPartitions into the following taskNames:
2014-08-24 11:20:57 Util$ [INFO] TaskName: Partition 0 =>
[SystemStreamPartition [wikipedia, #en.wiktionary, 0], SystemStreamPartition
[wikipedia, #en.wikinews, 0], SystemStreamPartition [wikipedia, #en.wikipedia,
0]]
2014-08-24 11:20:57 Util$ [INFO] Assigning 1 SystemStreamPartitions taskNames
to 1 containers.
2014-08-24 11:20:57 Util$ [INFO] Grouped SystemStreamPartition TaskNames (size
= 1):
2014-08-24 11:20:57 Util$ [INFO] Container number: 0 => Map(Partition 0 ->
Set(SystemStreamPartition [wikipedia, #en.wiktionary, 0], SystemStreamPartition
[wikipedia, #en.wikinews, 0], SystemStreamPartition [wikipedia, #en.wikipedia,
0]))
2014-08-24 11:20:57 Util$ [INFO] Previous mapping of taskNames to partition:
List()
2014-08-24 11:20:57 Util$ [INFO] Current set of taskNames: List(Partition 0)
2014-08-24 11:20:57 Util$ [WARN] No previous taskName mapping defined. This is
OK if it's the first time the job is being run, otherwise data may have been
lost.
2014-08-24 11:20:57 Util$ [INFO] No taskNames are missing between this run and
previous
2014-08-24 11:20:57 Util$ [WARN] The following new taskNames have been added in
this job run: Set(Partition 0)
2014-08-24 11:20:57 Util$ [INFO] New taskName to partition mapping:
List((Partition 0,0))
2014-08-24 11:20:57 SamzaScheduler [INFO] Awaiting offers for 1 executors
I0824 11:20:57.433486 14158 sched.cpp:126] Version: 0.19.1
2014-08-24 11:20:57,435:14126(0x7f52cac10700):ZOO_INFO@log_env@712: Client
environment:zookeeper.version=zookeeper C client 3.4.5
2014-08-24 11:20:57,435:14126(0x7f52cac10700):ZOO_INFO@log_env@716: Client
environment:host.name=localhost.localdomain
2014-08-24 11:20:57,435:14126(0x7f52cac10700):ZOO_INFO@log_env@723: Client
environment:os.name=Linux
2014-08-24 11:20:57,435:14126(0x7f52cac10700):ZOO_INFO@log_env@724: Client
environment:os.arch=3.10.0-123.6.3.el7.x86_64
2014-08-24 11:20:57,435:14126(0x7f52cac10700):ZOO_INFO@log_env@725: Client
environment:os.version=#1 SMP Wed Aug 6 21:12:36 UTC 2014
2014-08-24 11:20:57,435:14126(0x7f52cac10700):ZOO_INFO@log_env@733: Client
environment:user.name=jbringhu
2014-08-24 11:20:57,435:14126(0x7f52cac10700):ZOO_INFO@log_env@741: Client
environment:user.home=/home/jbringhu
2014-08-24 11:20:57,435:14126(0x7f52cac10700):ZOO_INFO@log_env@753: Client
environment:user.dir=/home/jbringhu/samza-dev/hello-samza
2014-08-24 11:20:57,435:14126(0x7f52cac10700):ZOO_INFO@zookeeper_init@786:
Initiating client connection, host=localhost:2181 sessionTimeout=10000
watcher=0x7f52cefa2880 sessionId=0 sessionPasswd=<null> context=0x7f52d4001040
flags=0
2014-08-24 11:20:57,436:14126(0x7f52c69ff700):ZOO_INFO@check_events@1703:
initiated connection to server [127.0.0.1:2181]
2014-08-24 11:20:57,440:14126(0x7f52c69ff700):ZOO_INFO@check_events@1750:
session establishment complete on server [127.0.0.1:2181],
sessionId=0x148089b0c550006, negotiated timeout=10000
I0824 11:20:57.441314 14173 group.cpp:310] Group process ((5)@127.0.0.1:45052)
connected to ZooKeeper
I0824 11:20:57.441344 14173 group.cpp:784] Syncing group operations: queue size
(joins, cancels, datas) = (0, 0, 0)
I0824 11:20:57.441359 14173 group.cpp:382] Trying to create path '/mesos' in
ZooKeeper
I0824 11:20:57.442654 14173 detector.cpp:135] Detected a new leader: (id='3')
I0824 11:20:57.442818 14173 group.cpp:655] Trying to get
'/mesos/info_0000000003' in ZooKeeper
I0824 11:20:57.443436 14173 detector.cpp:377] A new leading master
([email protected]:5050) is detected
I0824 11:20:57.443491 14173 sched.cpp:222] New master detected at
[email protected]:5050
I0824 11:20:57.443764 14173 sched.cpp:230] No credentials provided. Attempting
to register without authentication
I0824 11:20:57.447118 14173 sched.cpp:397] Framework registered with
wikipedia-feed
2014-08-24 11:20:57 SamzaScheduler [INFO] Framework registered
2014-08-24 11:20:57 SamzaScheduler [INFO] Received offer id {
value: "20140824-112052-16777343-5050-14035-0"
}
framework_id {
value: "wikipedia-feed"
}
slave_id {
value: "20140824-033930-16777343-5050-12649-0"
}
hostname: "localhost"
resources {
name: "cpus"
type: SCALAR
scalar {
value: 2.0
}
role: "*"
}
resources {
name: "mem"
type: SCALAR
scalar {
value: 2761.0
}
role: "*"
}
resources {
name: "disk"
type: SCALAR
scalar {
value: 12798.0
}
role: "*"
}
resources {
name: "ports"
type: RANGES
ranges {
range {
begin: 31000
end: 32000
}
}
role: "*"
}
2014-08-24 11:20:57 SamzaScheduler [INFO] Got available task id (0) for offer:
id {
value: "20140824-112052-16777343-5050-14035-0"
}
framework_id {
value: "wikipedia-feed"
}
slave_id {
value: "20140824-033930-16777343-5050-12649-0"
}
hostname: "localhost"
resources {
name: "cpus"
type: SCALAR
scalar {
value: 2.0
}
role: "*"
}
resources {
name: "mem"
type: SCALAR
scalar {
value: 2761.0
}
role: "*"
}
resources {
name: "disk"
type: SCALAR
scalar {
value: 12798.0
}
role: "*"
}
resources {
name: "ports"
type: RANGES
ranges {
range {
begin: 31000
end: 32000
}
}
role: "*"
}
2014-08-24 11:20:57 SamzaScheduler [INFO] Claimed SSP taskNames Map(Partition 0
-> Set(SystemStreamPartition [wikipedia, #en.wiktionary, 0],
SystemStreamPartition [wikipedia, #en.wikinews, 0], SystemStreamPartition
[wikipedia, #en.wikipedia, 0])) for offer ID 0
2014-08-24 11:20:57 SamzaScheduler [INFO] Task ID 0 using command
bin/run-container.sh
2014-08-24 11:20:57 SamzaScheduler [INFO] Launching task 0
2014-08-24 11:20:57 SamzaScheduler [INFO] Started task ID 0
2014-08-24 11:20:58 SamzaScheduler [INFO] (Status Update for Task %s: %s,value:
"0"
,TASK_RUNNING)
2014-08-24 11:20:59 SamzaScheduler [INFO] (Status Update for Task %s: %s,value:
"0"
,TASK_FAILED)
{noformat}
> Investigate Mesos Job Support
> -----------------------------
>
> Key: SAMZA-375
> URL: https://issues.apache.org/jira/browse/SAMZA-375
> Project: Samza
> Issue Type: Bug
> Reporter: Jon Bringhurst
> Assignee: Jon Bringhurst
> Attachments: Screen Shot 2014-08-23 at 5.51.39 PM.png
>
>
> It would be nice if Samza had support for Mesos (https://mesos.apache.org/).
> The current plan is to create a MesosJob and MesosJobFactory, then look into
> what it would take to allow the AM code to act as a Mesos scheduler.
> The feasibility of this landing in trunk will be better understood after a
> rough prototype has been created.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
