Patrik,
Bingo...you got me.
Alright, then I know what is wrong. You are using cluster sharding on the
worker nodes. That is not done in the activator template.
[Prakhyat]Yes. You are correct.
The WorkerSystem is not part of the cluster, perhaps not part of any
cluster, or do you have a separate cluster for all worker nodes?
[Prakhyat] The WorkerSystem is not part of the cluster. The workers connect
to master by ClusterClient. If you see worker configuration there is no
configuration for cluster or seed nodes. It is bare minimum configuration,
akka {
actor.provider = "akka.cluster.ClusterActorRefProvider"
extensions = ["akka.contrib.pattern.ClusterReceptionistExtension"]
remote.netty.tcp.port=4001
remote.netty.tcp.hostname=127.0.0.1
}
contact-points = [
"akka.tcp://[email protected]:2551"
// Master nodes info
]
As I understand from configuration worker is not part of any cluster.
Correct me if I am wrong.
Do you see the WorkerSystem nodes as Up? Probably not, and then there is no
coordinator for the worker nodes, and thereby no progress.
[Prakhyat] I will check on this have to check the logs.
If my guessing is correct you can solve this by make sure that the worker
nodes join a separate worker cluster.
[Prakhyat] Are you asking us to try including worker node with separate
cluster? I will try this. In any case please guide me which is the right
way.
You should also consider if you have a good reason to not include the
worker nodes in the same cluster as the master/frontend nodes?
[Prakhyat] We have not tried workers in same cluster as master and front.
If worker is in same cluster as master, do you see ClusterClient required?
We thought to use ClusterClient, workers should be in different cluster
then master. Correct me if I am wrong.
Will you have more than 1000 worker nodes? Are they untrusted or unstable?
[Prakhyat] Yes workers can go beyond 1000. They are trusted.
If you answer yes on any of these questions then you might have a good
reason to not include the worker nodes in the same cluster, otherwise I
think you should simplify the architecture by using one single cluster. In
that case you would not use ClusterClient, but instead use
DistributedPubSub for similar things.
[Prakhyat] Let me know from the above points, what steps are recommended.
-Prakhyat M M
On Wednesday, 3 September 2014 18:23:14 UTC+5:30, Patrik Nordwall wrote:
>
> Alright, then I know what is wrong. You are using cluster sharding on the
> worker nodes. That is not done in the activator template.
>
> The WorkerSystem is not part of the cluster, perhaps not part of any
> cluster, or do you have a separate cluster for all worker nodes?
> Do you see the WorkerSystem nodes as Up? Probably not, and then there is
> no coordinator for the worker nodes, and thereby no progress.
>
> If my guessing is correct you can solve this by make sure that the worker
> nodes join a separate worker cluster.
>
> You should also consider if you have a good reason to not include the
> worker nodes in the same cluster as the master/frontend nodes?
> Will you have more than 1000 worker nodes? Are they untrusted or unstable?
> If you answer yes on any of these questions then you might have a good
> reason to not include the worker nodes in the same cluster, otherwise I
> think you should simplify the architecture by using one single cluster. In
> that case you would not use ClusterClient, but instead use
> DistributedPubSub for similar things.
>
> Regards,
> Patrik
>
>
>
>
>
> On Wed, Sep 3, 2014 at 2:26 PM, Prakhyat Mallikarjun <[email protected]
> <javascript:>> wrote:
>
>> Hi Patrik,
>>
>> Thanks.
>>
>> We are working on top of the activator template and later modified it to
>> use cassandra plugin. We see all the nodes in cluster are up and running.
>> Front end is able to communicate web request to master, master is able
>> store work and workers are able to pull work from master.
>>
>> We modified the template locally to integrate with cassandra. The
>> integration works. No exceptions seen.
>>
>> The only one big change we made is, workers will do the processing and
>> persist state via sharded PersistentActors(implemented as Single Writers as
>> we are using DDD/CQRS approach).
>>
>> Request form Browser is coming to Front end . Request flow,
>> Browser -> FrontEnd -> Master -> Worker->Sharded Single
>> Writers(PersistentActor)
>>
>> The worker node is connecting to master using *ClusterClient(this is
>> same as typesafe template)*.
>>
>> Front End, Master, Worker are deployed on different nodes. The actor
>> system of front end and master is same i.e. "*ClusterSystem*". The actor
>> system of worker is different i.e. "*WorkerSystem*".
>>
>> Worker's pull the work from master and on successful processing persists
>> state via Sharded single writers.
>> *Note: Cluster Sharding is initiated with actor system *"*WorkerSystem*".
>> Worker actors and Cluster Sharding have same *actor system *"
>> *WorkerSystem*".
>>
>> *Master node and Worker node share same persistence cassandra.* Front
>> end node doesn't have any persistence.
>>
>> The master and front end configuration are as it is from Typesafe
>> template. Below is the configuration of worker.
>>
>> akka {
>>
>> //actor.provider = "akka.remote.RemoteActorRefProvider"
>> actor.provider = "akka.cluster.ClusterActorRefProvider"
>>
>> extensions = ["akka.contrib.pattern.ClusterReceptionistExtension"]
>> loglevel=DEBUG
>>
>> persistence {
>> journal.plugin = "cassandra-journal"
>> journal.leveldb.native = off
>> }
>>
>>
>> }
>>
>> contact-points = [
>> "akka.tcp://[email protected]:2551"
>> // Master nodes info
>> ]
>>
>> cassandra-journal {
>> class = "akka.persistence.journal.cassandra.CassandraJournal"
>> contact-points = ["172.26.145.251"]
>>
>> # Name of the keyspace to be created/used by the journal
>> keyspace = "akka"
>>
>> # Name of the table to be created/used by the journal
>> table = "messages"
>>
>> # Replication factor to use when creating a keyspace
>> replication-factor = 1
>>
>> # Write consistency level
>> write-consistency = "QUORUM"
>>
>> # Read consistency level
>> read-consistency = "QUORUM"
>>
>> # Maximum number of entries per partition (= columns per row).
>> # Must not be changed after table creation (currently not checked).
>> max-partition-size = 5000000
>>
>> # Maximum size of result set
>> max-result-size = 50001
>>
>> # Dispatcher for fetching and replaying messages
>> replay-dispatcher = "akka.persistence.dispatchers.
>> default-replay-dispatcher"
>> }
>>
>> When worker sends message to shard only below log is displayed,
>> *2014 12:46:46.465] [WorkerSystem-akka.actor.default-dispatcher-16]
>> [akka.tcp://[email protected]:4001/user/sharding/LedgerProcessor
>> <http://[email protected]:4001/user/sharding/LedgerProcessor>]*
>> *Request
>> shard [0] home*
>>
>> The idExtractor block is not called. The persistent actor doesnt receive
>> message. Nor the sent message goes to dead letter queue.
>>
>> -Prakhyat M M
>>
>> On Wednesday, 3 September 2014 13:56:49 UTC+5:30, Patrik Nordwall wrote:
>>
>>>
>>>
>>>
>>> On Wed, Sep 3, 2014 at 9:29 AM, Asha <[email protected]> wrote:
>>>
>>>>
>>>> Hi Patrik,
>>>>
>>>> I didn't get where to put idExtractor.isDefinedAt(msg).
>>>>
>>>
>>> That is something provided by Scala PartialFunction.
>>>
>>>
>>>>
>>>> As u told i have added akka.loglevel=DEBUG in conf file. I am
>>>> getting below line in logs.
>>>> * 2014 12:46:46.465] [WorkerSystem-akka.actor.default-dispatcher-16]
>>>> [akka.tcp://[email protected]:4001/user/sharding/LedgerProcessor
>>>> <http://[email protected]:4001/user/sharding/LedgerProcessor>]*
>>>> *Request
>>>> shard [0] home*
>>>> After that no logs.
>>>>
>>>> Below is my conf file.
>>>>
>>>> akka {
>>>>
>>>> //actor.provider = "akka.remote.RemoteActorRefProvider"
>>>> actor.provider = "akka.cluster.ClusterActorRefProvider"
>>>>
>>>> remote.netty.tcp.port=4001
>>>> remote.netty.tcp.hostname=127.0.0.1
>>>>
>>>> extensions = ["akka.contrib.pattern.ClusterReceptionistExtension"]
>>>> loglevel=DEBUG
>>>>
>>>> persistence {
>>>> journal.plugin = "cassandra-journal"
>>>> journal.leveldb.native = off
>>>> }
>>>>
>>>>
>>>> }
>>>>
>>>> contact-points = [
>>>> "akka.tcp://[email protected]:2551"
>>>> // Master nodes info
>>>> ]
>>>>
>>>> cassandra-journal {
>>>> class = "akka.persistence.journal.cassandra.CassandraJournal"
>>>> contact-points = ["172.26.145.251"]
>>>>
>>>> # Name of the keyspace to be created/used by the journal
>>>> keyspace = "akka"
>>>>
>>>> # Name of the table to be created/used by the journal
>>>> table = "messages"
>>>>
>>>> # Replication factor to use when creating a keyspace
>>>> replication-factor = 1
>>>>
>>>> # Write consistency level
>>>> write-consistency = "QUORUM"
>>>>
>>>> # Read consistency level
>>>> read-consistency = "QUORUM"
>>>>
>>>> # Maximum number of entries per partition (= columns per row).
>>>> # Must not be changed after table creation (currently not checked).
>>>> max-partition-size = 5000000
>>>>
>>>> # Maximum size of result set
>>>> max-result-size = 50001
>>>>
>>>> # Dispatcher for fetching and replaying messages
>>>> replay-dispatcher = "akka.persistence.dispatchers.
>>>> default-replay-dispatcher"
>>>> }
>>>>
>>>> I am not using the akka.cluster.seed-nodes to connect to master. From
>>>> this conf i am getting the master's contact point. and using the
>>>> clusterclient i am able to connect to master and communicate.
>>>>
>>>
>>> Do you join in some other way then? Do you see (in the logs) that the
>>> cluster nodes are Up?
>>>
>>> Have you tried the activator tutorial
>>> <http://typesafe.com/activator/template/akka-distributed-workers> and
>>> got that working? Then change the journal plugin to cassandra, and if it
>>> then fails it is something wrong with your setup of the cassandra journal
>>> plugin.
>>>
>>> /Patrik
>>>
>>>
>>>>
>>>> Please let me know wt did i missed in the conf file.
>>>>
>>>> Thanks
>>>> Asha
>>>>
>>>>
>>>>
>>>>
>>>> On Wednesday, 3 September 2014 12:11:36 UTC+5:30, Patrik Nordwall wrote:
>>>>
>>>>> Hi Asha and Prakhyat (you will not receive help faster by
>>>>> double-posting),
>>>>>
>>>>> First checks that it is a valid message with
>>>>> idExtractor.isDefinedAt(msg).
>>>>> That will not invoke your prints in idExtractor.
>>>>> Then it uses the shardExtractor, which you see in the logs.
>>>>> Before creating the entry, or delegating to the right node, it must
>>>>> get information from the coordinator. It is probably this that is not
>>>>> successful.
>>>>>
>>>>> You will find more information about what is going on by enabling
>>>>> akka.loglevel=DEBUG and look log messages from the "/user/sharding"
>>>>> actors.
>>>>>
>>>>> Regards,
>>>>> Patrik
>>>>>
>>>>>
>>>>> On Tue, Sep 2, 2014 at 8:02 PM, Asha <[email protected]> wrote:
>>>>>
>>>>>>
>>>>>> Hi Martynas,
>>>>>>
>>>>>> I am using* Cassandra plugin * for *persistance*, For the same
>>>>>> below are configuration changes in application.conf for persistance.
>>>>>>
>>>>>> akka{
>>>>>> persistence {
>>>>>> journal.plugin = "cassandra-journal"
>>>>>> }
>>>>>> }
>>>>>>
>>>>>> cassandra-journal {
>>>>>> contact-points = ["xxx.xxxx.xxx.xxx"]
>>>>>> }
>>>>>>
>>>>>> For this i am using below jars
>>>>>> akka-persistence-cassandra_2.11-0.3.3.jar
>>>>>> akka-persistence-experimental_2.10-2.3.4.jar
>>>>>> cassandra-driver-core-2.0.3.jar
>>>>>>
>>>>>> I am not using LevelDB.
>>>>>>
>>>>>> Thanks
>>>>>> Asha
>>>>>>
>>>>>>
>>>>>> On Tuesday, 2 September 2014 19:02:22 UTC+5:30, Martynas Mickevičius
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Asha,
>>>>>>>
>>>>>>> I would guess that persistence system fails to initialize. What
>>>>>>> persistence plugin and settings do you use? If you are prototyping and
>>>>>>> using LevelDB you either need to provide native LevelDB libraries or
>>>>>>> use
>>>>>>> Java LevelDB implementation by adding the following to your
>>>>>>> application.conf file:
>>>>>>>
>>>>>>> akka.persistence.journal.leveldb.native = off
>>>>>>>
>>>>>>> Just remember to select a plugin
>>>>>>> <http://akka.io/community/#journal-plugins> for a distributed
>>>>>>> journal when going to production.
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Sep 1, 2014 at 1:11 PM, Asha <[email protected]> wrote:
>>>>>>>
>>>>>>>> I think, i have given very less information.
>>>>>>>>
>>>>>>>> Here is the detailed description
>>>>>>>>
>>>>>>>> I am using Akka-pull pattern in my project. i am using
>>>>>>>> akka-contrib_2.10-2.3.4.jar and akka-persistence-experimental_
>>>>>>>> 2.10-2.3.4.jar
>>>>>>>> Front end, master, worker all are running different nodes. The
>>>>>>>> actor system of front end and master is "ClusterSystem".
>>>>>>>> The actor system of worker is "WorkerSystem". Request form Browser
>>>>>>>> is coming to Front end .
>>>>>>>> Request flow
>>>>>>>> Browser -> FrontEnd -> Master -> Worker
>>>>>>>>
>>>>>>>> Worker is able to pull the work from master and processing the
>>>>>>>> same. I want to persit the state after processing work. I am using
>>>>>>>> single
>>>>>>>> writer.
>>>>>>>> Shard for the persisted actor is stared when node comes up. For
>>>>>>>> persisting the state, i am hitting the shard region of the Persisted
>>>>>>>> Actor.
>>>>>>>> It is a able resovle the shard,
>>>>>>>> but failing to resolve the persisted actor(not executing the
>>>>>>>> idExtractor ())
>>>>>>>>
>>>>>>>> Below is the code.
>>>>>>>>
>>>>>>>> val idExtractor: ShardRegion.IdExtractor = {
>>>>>>>> case created: LedgerCreated =>{
>>>>>>>> println("%%%%%%%%% IdExtractor : "
>>>>>>>> +created.ledgerId.id.toString)
>>>>>>>> (created.ledgerId.id.toString, created)
>>>>>>>> }
>>>>>>>> case edited: LedgerEdited => (edited.ledgerId.id.toString,
>>>>>>>> edited)
>>>>>>>> case deleted: LedgerDeleted => (deleted.ledgerId.id.toString,
>>>>>>>> deleted)
>>>>>>>> }
>>>>>>>>
>>>>>>>> val shardResolver: ShardRegion.ShardResolver = msg => msg match {
>>>>>>>> case created: LedgerCreated =>{
>>>>>>>> println("%%%%%%%%% ShardResolver : " +created.ledgerId.id %
>>>>>>>> shards)
>>>>>>>> (created.ledgerId.id % shards).toString
>>>>>>>> }
>>>>>>>> case edited: LedgerEdited => (edited.ledgerId.id %
>>>>>>>> shards).toString
>>>>>>>> case deleted: LedgerDeleted => (deleted.ledgerId.id %
>>>>>>>> shards).toString
>>>>>>>> }
>>>>>>>>
>>>>>>>> I am getting the log written in the ShardResolver. But the log
>>>>>>>> written in the IdExtractor is not coming.
>>>>>>>>
>>>>>>>> Please help me to solve this problem.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Monday, 1 September 2014 15:00:16 UTC+5:30, Asha wrote:
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I am using Akka-pull pattern in my project. Request from the
>>>>>>>>> browser will land on spray layer, it will pass the request to master
>>>>>>>>> then
>>>>>>>>> master to worker.
>>>>>>>>> In worker i am hitting the shard region. When i hit the shard
>>>>>>>>> region it is able to resolve shard but failing to execute the
>>>>>>>>> idExtractor.
>>>>>>>>> It is not throwing any error.
>>>>>>>>>
>>>>>>>>> Not getting whether issue with the jar or my code. Please help
>>>>>>>>> me to resolve this problem
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>> Asha
>>>>>>>>>
>>>>>>>>>
>>>>>>>> --
>>>>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>> >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/c
>>>>>>>> urrent/additional/faq.html
>>>>>>>> >>>>>>>>>> Search the archives: https://groups.google.com/grou
>>>>>>>> p/akka-user
>>>>>>>> ---
>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>> Groups "Akka User List" group.
>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>> send an email to [email protected].
>>>>>>>> To post to this group, send email to [email protected].
>>>>>>>>
>>>>>>>> Visit this group at http://groups.google.com/group/akka-user.
>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Martynas Mickevičius
>>>>>>> Typesafe <http://typesafe.com/> – Reactive
>>>>>>> <http://www.reactivemanifesto.org/> Apps on the JVM
>>>>>>>
>>>>>> --
>>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>> >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/c
>>>>>> urrent/additional/faq.html
>>>>>> >>>>>>>>>> Search the archives: https://groups.google.com/grou
>>>>>> p/akka-user
>>>>>> ---
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "Akka User List" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>> send an email to [email protected].
>>>>>> To post to this group, send email to [email protected].
>>>>>> Visit this group at http://groups.google.com/group/akka-user.
>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Patrik Nordwall
>>>>> Typesafe <http://typesafe.com/> - Reactive apps on the JVM
>>>>> Twitter: @patriknw
>>>>>
>>>>> --
>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/
>>>> >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/
>>>> current/additional/faq.html
>>>> >>>>>>>>>> Search the archives: https://groups.google.com/
>>>> group/akka-user
>>>> ---
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Akka User List" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to [email protected].
>>>> To post to this group, send email to [email protected].
>>>> Visit this group at http://groups.google.com/group/akka-user.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>
>>>
>>>
>>> --
>>>
>>> Patrik Nordwall
>>> Typesafe <http://typesafe.com/> - Reactive apps on the JVM
>>> Twitter: @patriknw
>>>
>>> --
>> >>>>>>>>>> Read the docs: http://akka.io/docs/
>> >>>>>>>>>> Check the FAQ:
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Akka User List" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected] <javascript:>.
>> To post to this group, send email to [email protected]
>> <javascript:>.
>> Visit this group at http://groups.google.com/group/akka-user.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> --
>
> Patrik Nordwall
> Typesafe <http://typesafe.com/> - Reactive apps on the JVM
> Twitter: @patriknw
>
>
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
