OS used : Cent OS 6 on all nodes except *10*.125.138.59 ( which runs Cent
OS 7)
All of them are running Cassandra 2.0.17
output of the test :
host ip: 10.124.114.113
host DC : WDC
distance of host: LOCAL
host is up: true
cassandra version : 2.0.17
host ip: 10.124.114.108
host DC : WDC
distance of host: LOCAL
host is up: true
cassandra version : 2.0.17
host ip: 10.124.114.110
host DC : WDC
distance of host: LOCAL
host is up: true
cassandra version : 2.0.17
host ip: 10.124.114.118
host DC : WDC
distance of host: LOCAL
host is up: true
cassandra version : 2.0.17
host ip: 10.125.138.59
host DC : WDC
distance of host: LOCAL
host is up: true
cassandra version : 2.0.17
host ip: 10.124.114.97
host DC : WDC
distance of host: LOCAL
host is up: true
cassandra version : 2.0.17
host ip: 10.124.114.105
host DC : WDC
distance of host: LOCAL
host is up: true
cassandra version : 2.0.17
host ip: 10.124.114.98
host DC : WDC
distance of host: LOCAL
host is up: true
cassandra version : 2.0.17
On Fri, Apr 15, 2016 at 6:47 PM, Eric Stevens <migh...@gmail.com> wrote:
> Thanks for that, that helps a lot. The next thing to check might be
> whether or not your application actually has access to the other nodes.
> With that topology, and assuming all the nodes you included in your
> original graph are in the 'WDC' data center, I'd be inclined to look for a
> network issue of some kind.
>
> Also, it probably doesn't matter, but what OS / Distribution are you
> running the servers and clients on?
>
> Check with netcat or something that you can reach all the configured ports
> from your application server, but also the driver itself offers some
> introspection into its view of individual connection health. This is a
> little bit ugly, but this is how we include information about connection
> status in an API for health monitoring from a Scala application using the
> Java driver; hopefully you can use it to see how to access information
> about the driver's view of host health from the application's perspective.
> Most importantly I'd suggest looking for host.isUp status and
> LoadBalancingPolicy.distance(host) to see that it considers all the hosts
> in your target datacenter to be LOCAL.
>
> "hosts" -> {
> val hosts: Map[String, Map[String, mutable.Set[Host]]] =
> connection.getMetadata
> .getAllHosts.asScala
> .groupBy(_.getDatacenter)
> .mapValues(_.groupBy(_.getRack))
> val lbp: LoadBalancingPolicy =
> connection.getConfiguration.getPolicies.getLoadBalancingPolicy
> JsObject(hosts.map { case (dc: String, rackAndHosts) =>
> dc -> JsObject(rackAndHosts.map { case (rack: String, hosts:
> mutable.Set[Host]) =>
> rack -> JsArray(hosts.map { host =>
> Json.obj(
> "address" -> host.getAddress.toString,
> "socketAddress" -> host.getSocketAddress.toString,
> "cassandraVersion" -> host.getCassandraVersion.toString,
> "isUp" -> host.isUp,
> "hostDistance" -> lbp.distance(host).toString
> )
> }.toSeq)
> }.toSeq)
> }.toSeq)
> },
>
>
> On Wed, Apr 13, 2016 at 10:50 PM Anishek Agarwal <anis...@gmail.com>
> wrote:
>
>> here is the output: every node in a single DC is in the same rack.
>>
>> Datacenter: WDC5
>>
>> ================
>>
>> Status=Up/Down
>>
>> |/ State=Normal/Leaving/Joining/Moving
>>
>> -- Address Load Tokens Owns (effective) Host ID
>> Rack
>>
>> UN 10.125.138.33 299.22 GB 256 64.2%
>> 8aaa6015-d444-4551-a3c5-3257536df476 RAC1
>>
>> UN 10.125.138.125 329.38 GB 256 70.3%
>> 70be44a2-de17-41f1-9d3a-6a0be600eedf RAC1
>>
>> UN 10.125.138.129 305.11 GB 256 65.5%
>> 0fbc7f44-7062-4996-9eba-2a05ae1a7032 RAC1
>>
>> Datacenter: WDC
>>
>> ===============
>>
>> Status=Up/Down
>>
>> |/ State=Normal/Leaving/Joining/Moving
>>
>> -- Address Load Tokens Owns (effective) Host ID
>> Rack
>>
>> UN 10.124.114.105 151.09 GB 256 38.0%
>> c432357d-bf81-4eef-98e1-664c178a3c23 RAC1
>>
>> UN 10.124.114.110 150.15 GB 256 36.9%
>> 6f92d32e-1c64-4145-83d7-265c331ea408 RAC1
>>
>> UN 10.124.114.108 170.1 GB 256 41.3%
>> 040ae7e5-3f1e-4874-8738-45edbf576e12 RAC1
>>
>> UN 10.124.114.98 165.34 GB 256 37.6%
>> cdc69c7d-b9d6-4abd-9388-1cdcd35d946c RAC1
>>
>> UN 10.124.114.113 145.22 GB 256 35.7%
>> 1557af04-e658-4751-b984-8e0cdc41376e RAC1
>>
>> UN 10.125.138.59 162.65 GB 256 38.6%
>> 9ba1b7b6-5655-456e-b1a1-6f429750fc96 RAC1
>>
>> UN 10.124.114.97 164.03 GB 256 36.9%
>> c918e497-498e-44c3-ab01-ab5cb4d48b09 RAC1
>>
>> UN 10.124.114.118 139.62 GB 256 35.1%
>> 2bb0c265-a5d4-4cd4-8f50-13b5a9a891c9 RAC1
>>
>> On Thu, Apr 14, 2016 at 4:48 AM, Eric Stevens <migh...@gmail.com> wrote:
>>
>>> The output of nodetool status would really help answer some questions.
>>> I take it the 8 hosts in your graph are in the same DC. Are the four
>>> serving writes in the same logical or physical rack (as Cassandra sees it),
>>> while the others are not?
>>>
>>> On Tue, Apr 12, 2016 at 10:48 PM Anishek Agarwal <anis...@gmail.com>
>>> wrote:
>>>
>>>> We have two DC one with the above 8 nodes and other with 3 nodes.
>>>>
>>>>
>>>>
>>>> On Tue, Apr 12, 2016 at 8:06 PM, Eric Stevens <migh...@gmail.com>
>>>> wrote:
>>>>
>>>>> Maybe include nodetool status here? Are the four nodes serving reads
>>>>> in one DC (local to your driver's config) while the others are in another?
>>>>>
>>>>> On Tue, Apr 12, 2016, 1:01 AM Anishek Agarwal <anis...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> hello,
>>>>>>
>>>>>> we have 8 nodes in one cluster and attached is the traffic patterns
>>>>>> across the nodes.
>>>>>>
>>>>>> its very surprising that only 4 nodes show transmitting (purple)
>>>>>> packets.
>>>>>>
>>>>>> our driver configuration on clients has the following load balancing
>>>>>> configuration :
>>>>>>
>>>>>> new TokenAwarePolicy(
>>>>>> new
>>>>>> DCAwareRoundRobinPolicy(configuration.get(Constants.LOCAL_DATA_CENTRE_NAME,
>>>>>> "WDC")),
>>>>>> true)
>>>>>>
>>>>>>
>>>>>> any idea what is that we are missing which is leading to this skewed
>>>>>> data read patterns
>>>>>>
>>>>>> cassandra drivers as below:
>>>>>>
>>>>>> <dependency>
>>>>>> <groupId>com.datastax.cassandra</groupId>
>>>>>> <artifactId>cassandra-driver-core</artifactId>
>>>>>> <version>2.1.6</version>
>>>>>> </dependency>
>>>>>> <dependency>
>>>>>> <groupId>com.datastax.cassandra</groupId>
>>>>>> <artifactId>cassandra-driver-mapping</artifactId>
>>>>>> <version>2.1.6</version>
>>>>>> </dependency>
>>>>>>
>>>>>> cassandra version is 2.0.17
>>>>>>
>>>>>> Thanks in advance for the help.
>>>>>>
>>>>>> Anishek
>>>>>>
>>>>>>
>>>>
>>
