I forgot to mention,
When I run ` nc 10.3.1.76 32060 -v` I see nothing (not success, not failure.
Just no output and it hangs)
Also, my bono-depl.yaml file says "containerPort: 5060" not 30060. Is that
right?
When I run a tcp dump on the test machine I see dump.txt (attached)
(I'm not sure how this mailing list will cope with attachments)
The bono logs from that time are attached as bono_log.txt. (I inserted comments
with ### to make it clear when the test was running.)
For some reason there is no soft link called /var/log/ellis/ellis_current.txt
The contents of ellis-err.log is "No handlers could be found for logger
"metaswitch.utils"" I'm not sure whether that appeared during the test or not.
ellis_20180604T070000Z.txt is empty. (No new entries during the test)
I modified the log level on both ellis and bono, and restarted the bono and
ellis service respectively, prior to running the test.
Regards,
Matt
From: Davis, Matthew
Sent: Monday, 4 June 2018 5:32 PM
To: 'Adam Lindley' <[email protected]>;
[email protected]
Subject: RE: [Project Clearwater] Issues with clearwater-docker homestead and
homestead-prov under Kubernetes
Hi Adam,
Let's focus on the 3 differences:
Changing the PUBLIC IP to match my rig IP.
Where is this? In bono-deply.yaml? Anywhere else? (should I have it in .env?)
Should this include the port number?
Which IP should this be? Currently I have it equal to the Kubernetes API IP
address.
When I run ` kubectl config view | grep 10` I see `server:
https://10.3.1.76:6443`<https://10.3.1.76:6443%60> so I set PUBLIC_IP to
10.3.1.76.
What do you see? I think 6443 might be a non-standard port for Kubernetes.
Could that be a source of the problem?
My .env file currently says "PUBLIC_IP=" (without an address)
Changed the image pull source to our internal one: Hmm, maybe my images were
built wrong? Once my pull requests are merged, I'll do a fresh clone and
rebuild.
Changed the zone in the config map to my own one, 'ajl.svc.cw-k8s.test': Why
are you using the non-default zone ajl.svc.cw-k8s.test? Where did that come
from? I thought my original Azure issue was because I used a zone which wasn't
default.svc.cluster.local. What is that parameter? What does it mean? Why is
the default what it is?
My test command is:
rake test[default.svc.cw-k8s.test] PROXY=10.3.1.76 PROXY_PORT=30060
SIGNUP_CODE='secret' ELLIS=10.3.1.76:30080 TESTS="Basic call - mainline"
I'm wrapping secret in single apostrophes. Without the apostrophes the result
is the same.
Every pod (except etcd) has a shared_config file which says:
# Keys
signup_key=secret
turn_workaround=secret
ellis_api_key=secret
ellis_cookie_key=secret
Thanks,
Matt
From: Adam Lindley [mailto:[email protected]]
Sent: Friday, 1 June 2018 3:29 AM
To: Davis, Matthew
<[email protected]<mailto:[email protected]>>;
[email protected]<mailto:[email protected]>
Subject: RE: [Project Clearwater] Issues with clearwater-docker homestead and
homestead-prov under Kubernetes
Hey Matthew,
Sorry to hear it isn't working yet. Looking at the output of rake test, you're
hitting a forbidden in the RestClient, and so it's something on the Http calls.
I think your issue may just be a mismatch in the value of 'SIGNUP_CODE' that
you're passing in to the rake test command, and what's on your deployment. This
should default to 'secret', and I don't see anything in your configmap that
would change that, so can you double check the test command includes "
SIGNUP_CODE='secret' ". If you have made any changes to the signup key then
obviously it'll need to match those. If you want to double check the value,
take a look in the /etc/clearwater/shared_config file in one of the pods.
I have double checked a deployment on our rig, copying your provided yaml files
over directly to make sure there isn't anything odd in there, and the live
tests were able to run fine, following the deployment steps you're using. I did
have some trouble getting the script to work as copied over, but think that's
just outlook formatting quotes wrong etc. Manually running the commands it all
worked as expected. The only changes I made were:
* Changing the PUBLIC IP to match my rig IP.
* Changed the image pull source to our internal one
* Changed the zone in the config map to my own one, 'ajl.svc.cw-k8s.test'
With this all deployed, the following test command passed with no issue:
rake test[ajl.svc.cw-k8s.test] PROXY=10.230.16.1 PROXY_PORT=30060
SIGNUP_CODE='secret' ELLIS=10.230.16.1:30080 TESTS="Basic call - mainline"
If the issue isn't the signup key, can we try getting some more diags that we
can take a look into? In particular, I think we would benefit from:
* A packet capture on the node you are running the live tests on, when you
hit the errors below
* The bono logs, at debug level, from the same time. To set up debug
logging, you need to add 'log_level=5' to /etc/clearwater/user_settings
(creating if needed), and restart the bono service
* The ellis logs from the same time
Running the tcpdump on the test node should mean we get to see the full set of
flows, and you can likely read through that yourself to work out any following
issues you find hiding behind this next one.
Any other diagnostics you can gather would obviously also be useful, but with
the above, assuming traffic is reaching the pods, we should be able to work out
the issue.
On your connectivity tests, you won't be able to connect to the bono service
using 'nc localhost 30060', because that attempts to connect using the
localhost IP. We have set the bono service up to listen on the 'PUBLIC_IP',
i.e. the host IP. If you try running 'nc 10.3.1.76 30060 -v' you should see
successful connection. (Or on whichever host IP you have configured it to
listen).
The log output you are seeing on restarting Bono is also benign. These are
again simply an artefact of some behaviour we want to have in VMs, but that is
not needed in these containers. You can safely ignore this output.
Good luck, and let us know where you get with debugging.
Cheers,
Adam
From: Davis, Matthew [mailto:[email protected]]
Sent: 30 May 2018 08:27
To: Adam Lindley
<[email protected]<mailto:[email protected]>>;
[email protected]<mailto:[email protected]>
Subject: RE: [Project Clearwater] Issues with clearwater-docker homestead and
homestead-prov under Kubernetes
Hi Adam,
# Openstack Install
I only mentioned the helm charts just in case the almost empty charts on my
machine were the source of the error. I personally have no experience with
Helm, so I can't help you with any development.
I applied that latest change to the bono service port number. It still doesn't
work.
How can I check whether the rake tests are failing on the bono side, as opposed
to failing on the ellis side? Maybe the reason tcpdumps shows nothing in Bono
during the rake tests is because the tests failed to create a user in ellis,
and never got to the bono part?
Rake output:
```
Basic Call - Mainline (TCP) - Failed
RestClient::Forbidden thrown:
- 403 Forbidden
-
/home/ubuntu/.rvm/gems/ruby-1.9.3-p551/gems/rest-client-1.8.0/lib/restclient/abstract_response.rb:74:in
`return!'
-
/home/ubuntu/.rvm/gems/ruby-1.9.3-p551/gems/rest-client-1.8.0/lib/restclient/request.rb:495:in
`process_result'
...
```
If I go inside the ellis container and run `nc localhost 80 -v` I see that it
establishes a connection.
If I go inside the bono container and run `nc localhost 5060 -v` or `nc
localhost 30060 -v` it fails to connect. So from within the bono pod I cannot
connect to the localhost. To me that suggests that the problem is caused by
something inside bono, not the networking between pods. What happens when you
try `nc localhost 32060 -v` in your deployment?
The logs inside bono are an echo of an error message from sprout. Does that
matter?
```
30-05-2018 06:46:59.432 UTC [7f45527e4700] Status sip_connection_pool.cpp:428:
Recycle TCP connection slot 4
30-05-2018 06:47:06.222 UTC [7f4575ae0700] Status alarm.cpp:244: Reraising all
alarms with a known state
30-05-2018 06:47:06.222 UTC [7f4575ae0700] Status alarm.cpp:37: sprout issued
1012.3 alarm
30-05-2018 06:47:06.222 UTC [7f4575ae0700] Status alarm.cpp:37: sprout issued
1013.3 alarm
```
Those timestamps don't correspond to the rake tests. They just happen every 30
seconds.
When I restart the bono service it says: `63: ulimit: open files: cannot modify
limit: Invalid argument`
Does that matter? (I've seen that error message everywhere. I have no idea what
it means)
I've appended the yaml files to the end of this email.
# Azure Install
I had a chat to Microsoft. It seems that your hunch was correct. HTTP
Application Routing only works on port 80 and 443. Furthermore, I cannot simply
route SIP calls through port 443 because the routing does some HTTP specific
packet inspection things. So I'll have to give up on that approach and go for a
more vanilla, manually configured NodePort approach (either still on AKS but
without HTTP Application Routing, or on Openstack). So I'm even more keen to
solve the aforementioned issues.
# Yamls and stuff
I'm pasting them again just in case I've forgotten something. 10.3.1.76 is the
ip address of my cluster.
Here's a script I'm using to tear down and rebuild everything. (Just incase
`kubectl apply -f something.yaml` doesn't actually propagate the change fully)
The while loops in this script are just to wait until the previous step has
finished.
```
set -x
cd clearwater-docker-master/kubernetes
kubectl delete -f ./
kubectl delete configmap env-vars
set -e
echo 'waiting until old pods are all deleted'
while [ $(kubectl get pods | grep ^NAME -v | wc -l ) -neq 0]
do
sleep 5
done
echo "creating new pods"
kubectl create configmap env-vars --from-literal=ZONE=default.svc.cluster.local
kubectl apply -f ./
while [ $(kubectl get pods | grep "2/2" | grep bono | wc -l) -neq 1 ]
do
sleep 5
done
BONO=$(kubectl get pods | grep "2/2" | grep bono | awk '{ print $1 }')
echo "Bono is up as $BONO"
kubectl exec -it $BONO -- apt-get -y install vim
kubectl exec -it $BONO -- sed -i -e 's/--pcscf=5060,5058/--pcscf=30060,5058/g'
/etc/init.d/bono
kubectl exec -it $BONO service bono restart
while [ $(kubectl get pods | grep "0/" | wc -l) -neq 0 ]
do
sleep 5
done
echo "All pods are up now"
kubectl get pods
echo "Done"
```
`kubectl get services`
```
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
AGE
astaire ClusterIP None <none> 11311/TCP
1h
bono NodePort 10.0.168.197 <none>
3478:32214/TCP,30060:30060/TCP,5062:30144/TCP 1h
cassandra ClusterIP None <none>
7001/TCP,7000/TCP,9042/TCP,9160/TCP 1h
chronos ClusterIP None <none> 7253/TCP
1h
ellis NodePort 10.0.53.199 <none> 80:30080/TCP
1h
etcd ClusterIP None <none>
2379/TCP,2380/TCP,4001/TCP 1h
homer ClusterIP None <none> 7888/TCP
1h
homestead ClusterIP None <none> 8888/TCP
1h
homestead-prov ClusterIP None <none> 8889/TCP
1h
kubernetes ClusterIP 10.0.0.1 <none> 443/TCP
5d
ralf ClusterIP None <none> 10888/TCP
1h
sprout ClusterIP None <none> 5052/TCP,5054/TCP
1h
```
bono-depl.yaml
```
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: bono
spec:
replicas: 1
selector:
matchLabels:
service: bono
template:
metadata:
labels:
service: bono
snmp: enabled
spec:
containers:
- image: "mlda065/bono:latest"
imagePullPolicy: Always
name: bono
ports:
- containerPort: 22
- containerPort: 3478
- containerPort: 5060
- containerPort: 5062
- containerPort: 5060
protocol: "UDP"
- containerPort: 5062
protocol: "UDP"
envFrom:
- configMapRef:
name: env-vars
env:
- name: MY_POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: PUBLIC_IP
value: 10.3.1.76
livenessProbe:
exec:
command: ["/bin/bash", "/usr/share/kubernetes/liveness.sh", "3478
5062"]
initialDelaySeconds: 30
readinessProbe:
exec:
command: ["/bin/bash", "/usr/share/kubernetes/liveness.sh", "3478
5062"]
volumeMounts:
- name: bonologs
mountPath: /var/log/bono
- image: busybox
name: tailer
command: [ "tail", "-F", "/var/log/bono/bono_current.txt" ]
volumeMounts:
- name: bonologs
mountPath: /var/log/bono
volumes:
- name: bonologs
emptyDir: {}
imagePullSecrets:
- name: ~
restartPolicy: Always
```
Bono-svc.yaml
```
apiVersion: v1
kind: Service
metadata:
name: bono
spec:
type: NodePort
ports:
- name: "3478"
port: 3478
- name: "5060"
port: 30060
nodePort: 30060
- name: "5062"
port: 5062
selector:
service: bono
```
Ellis service
```
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: ellis
spec:
replicas: 1
template:
metadata:
labels:
service: ellis
spec:
containers:
- image: "mlda065/ellis:latest"
imagePullPolicy: Always
name: ellis
ports:
- containerPort: 22
- containerPort: 80
envFrom:
- configMapRef:
name: env-vars
env:
- name: MY_POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
livenessProbe:
tcpSocket:
port: 80
initialDelaySeconds: 30
readinessProbe:
tcpSocket:
port: 80
imagePullSecrets:
- name: ~
restartPolicy: Always
```
Ellis-svc.yaml
```
apiVersion: v1
kind: Service
metadata:
name: ellis
spec:
type: NodePort
ports:
- name: "http"
port: 80
nodePort: 30080
selector:
service: ellis
```
`kubectl describe configmap env-vars`
```
Name: env-vars
Namespace: default
Labels: <none>
Annotations:
kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","data":{"ZONE":"default.svc.cluster.local"},"kind":"ConfigMap","metadata":{"annotations":{},"name":"env-vars","namespace":"default"}...
Data
====
ZONE:
----
default.svc.cluster.local
Events: <none>
```
Thanks,
Matt
Telstra Graduate Engineer
CTO | Cloud SDN NFV
From: Adam Lindley [mailto:[email protected]]
Sent: Friday, 25 May 2018 6:42 PM
To: Davis, Matthew
<[email protected]<mailto:[email protected]>>;
[email protected]<mailto:[email protected]>
Subject: RE: [Project Clearwater] Issues with clearwater-docker homestead and
homestead-prov under Kubernetes
Hi Matthew,
Our Helm support is a recent addition, and came from another external
contributor. See the Pull Request at
https://github.com/Metaswitch/clearwater-docker/pull/85 for the details :)
As it stands at the moment, the chart is good enough for deploying and
re-creating a full standard deployment through Helm, but I don't believe it
handles more of the complexities of upgrading a clearwater deployment that it
potentially could.
We haven't yet done any significant work in setting up Helm charts, or
integrating with them in a more detailed manner, so if that's something you're
interested in as well, we'd love to work with you to get some more enhancements
in. Especially if you have other expert contacts who know more in this area.
(I'm removing some of the thread in the email below, to keep us below the list
limits. The online archives will keep all the info though)
Cheers,
Adam
### Start test B
04-06-2018 08:16:37.322 UTC [7f71f15e9700] Status alarm.cpp:244: Reraising all
alarms with a known state
04-06-2018 08:16:37.322 UTC [7f71f15e9700] Status alarm.cpp:37: sprout issued
1012.3 alarm
04-06-2018 08:16:37.322 UTC [7f71f15e9700] Status alarm.cpp:37: sprout issued
1013.3 alarm
### End test B
04-06-2018 08:17:07.322 UTC [7f71f15e9700] Status alarm.cpp:244: Reraising all
alarms with a known state
04-06-2018 08:17:07.322 UTC [7f71f15e9700] Status alarm.cpp:37: sprout issued
1012.3 alarm
04-06-2018 08:17:07.322 UTC [7f71f15e9700] Status alarm.cpp:37: sprout issued
1013.3 alarm
04-06-2018 08:17:37.322 UTC [7f71f15e9700] Status alarm.cpp:244: Reraising all
alarms with a known state
04-06-2018 08:17:37.322 UTC [7f71f15e9700] Status alarm.cpp:37: sprout issued
1012.3 alarm
04-06-2018 08:17:37.322 UTC [7f71f15e9700] Status alarm.cpp:37: sprout issued
1013.3 alarm
08:16:39.695011 IP (tos 0x0, ttl 64, id 56114, offset 0, flags [DF], proto TCP
(6), length 73)
192.168.61.12.48788 > 13.95.105.108.443: Flags [P.], cksum 0x74bb
(incorrect -> 0x10d5), seq 2253557675:2253557708, ack 2321556308, win 782,
length 33
08:16:39.695034 IP (tos 0x0, ttl 64, id 56115, offset 0, flags [DF], proto TCP
(6), length 73)
192.168.61.12.48788 > 13.95.105.108.443: Flags [P.], cksum 0x74bb
(incorrect -> 0x294a), seq 33:66, ack 1, win 782, length 33
08:16:39.695040 IP (tos 0x0, ttl 64, id 56116, offset 0, flags [DF], proto TCP
(6), length 70)
192.168.61.12.48788 > 13.95.105.108.443: Flags [P.], cksum 0x74b8
(incorrect -> 0x43bc), seq 66:96, ack 1, win 782, length 30
08:16:40.020305 IP (tos 0x0, ttl 25, id 2482, offset 0, flags [DF], proto TCP
(6), length 40)
13.95.105.108.443 > 192.168.61.12.48788: Flags [.], cksum 0xb5bc (correct),
ack 96, win 310, length 0
08:16:40.066501 IP (tos 0x0, ttl 25, id 2483, offset 0, flags [DF], proto TCP
(6), length 229)
13.95.105.108.443 > 192.168.61.12.48788: Flags [P.], cksum 0xadf0
(correct), seq 1:190, ack 96, win 310, length 189
08:16:40.066528 IP (tos 0x0, ttl 64, id 56117, offset 0, flags [DF], proto TCP
(6), length 40)
192.168.61.12.48788 > 13.95.105.108.443: Flags [.], cksum 0x749a (incorrect
-> 0xb310), ack 190, win 805, length 0
08:16:44.165719 IP (tos 0x0, ttl 64, id 5373, offset 0, flags [DF], proto TCP
(6), length 60)
192.168.61.12.39014 > 10.3.1.76.30080: Flags [S], cksum 0x0932 (incorrect
-> 0x31e4), seq 3376861218, win 26730, options [mss 8910,sackOK,TS val
459588831 ecr 0,nop,wscale 7], length 0
08:16:44.167773 IP (tos 0x0, ttl 59, id 0, offset 0, flags [DF], proto TCP (6),
length 60)
10.3.1.76.30080 > 192.168.61.12.39014: Flags [S.], cksum 0x5674 (correct),
seq 4218996597, ack 3376861219, win 26480, options [mss 1336,sackOK,TS val
1217130100 ecr 459588831,nop,wscale 7], length 0
08:16:44.167814 IP (tos 0x0, ttl 64, id 5374, offset 0, flags [DF], proto TCP
(6), length 52)
192.168.61.12.39014 > 10.3.1.76.30080: Flags [.], cksum 0x092a (incorrect
-> 0xeb62), ack 1, win 209, options [nop,nop,TS val 459588832 ecr 1217130100],
length 0
08:16:44.168036 IP (tos 0x0, ttl 64, id 5375, offset 0, flags [DF], proto TCP
(6), length 257)
192.168.61.12.39014 > 10.3.1.76.30080: Flags [P.], cksum 0x09f7 (incorrect
-> 0x09ec), seq 1:206, ack 1, win 209, options [nop,nop,TS val 459588832 ecr
1217130100], length 205
08:16:44.168053 IP (tos 0x0, ttl 64, id 5376, offset 0, flags [DF], proto TCP
(6), length 123)
192.168.61.12.39014 > 10.3.1.76.30080: Flags [P.], cksum 0x0971 (incorrect
-> 0x4601), seq 206:277, ack 1, win 209, options [nop,nop,TS val 459588832 ecr
1217130100], length 71
08:16:44.168752 IP (tos 0x0, ttl 59, id 10499, offset 0, flags [DF], proto TCP
(6), length 52)
10.3.1.76.30080 > 192.168.61.12.39014: Flags [.], cksum 0xea8d (correct),
ack 206, win 216, options [nop,nop,TS val 1217130101 ecr 459588832], length 0
08:16:44.168764 IP (tos 0x0, ttl 59, id 10500, offset 0, flags [DF], proto TCP
(6), length 52)
10.3.1.76.30080 > 192.168.61.12.39014: Flags [.], cksum 0xea46 (correct),
ack 277, win 216, options [nop,nop,TS val 1217130101 ecr 459588832], length 0
08:16:44.247723 IP (tos 0x0, ttl 59, id 10501, offset 0, flags [DF], proto TCP
(6), length 502)
10.3.1.76.30080 > 192.168.61.12.39014: Flags [P.], cksum 0xe141 (correct),
seq 1:451, ack 277, win 216, options [nop,nop,TS val 1217130180 ecr 459588832],
length 450
08:16:44.247739 IP (tos 0x0, ttl 64, id 5377, offset 0, flags [DF], proto TCP
(6), length 52)
192.168.61.12.39014 > 10.3.1.76.30080: Flags [.], cksum 0x092a (incorrect
-> 0xe81f), ack 451, win 218, options [nop,nop,TS val 459588852 ecr
1217130180], length 0
08:16:44.248119 IP (tos 0x0, ttl 64, id 5378, offset 0, flags [DF], proto TCP
(6), length 52)
192.168.61.12.39014 > 10.3.1.76.30080: Flags [F.], cksum 0x092a (incorrect
-> 0xe81e), seq 277, ack 451, win 218, options [nop,nop,TS val 459588852 ecr
1217130180], length 0
08:16:44.248658 IP (tos 0x0, ttl 59, id 10502, offset 0, flags [DF], proto TCP
(6), length 52)
10.3.1.76.30080 > 192.168.61.12.39014: Flags [F.], cksum 0xe81e (correct),
seq 451, ack 278, win 216, options [nop,nop,TS val 1217130181 ecr 459588852],
length 0
08:16:44.248673 IP (tos 0x0, ttl 64, id 5379, offset 0, flags [DF], proto TCP
(6), length 52)
192.168.61.12.39014 > 10.3.1.76.30080: Flags [.], cksum 0x092a (incorrect
-> 0xe81c), ack 452, win 218, options [nop,nop,TS val 459588852 ecr
1217130181], length 0
08:16:44.252785 IP (tos 0x0, ttl 64, id 12077, offset 0, flags [DF], proto TCP
(6), length 60)
192.168.61.12.39016 > 10.3.1.76.30080: Flags [S], cksum 0x0932 (incorrect
-> 0xe75d), seq 4160678872, win 26730, options [mss 8910,sackOK,TS val
459588853 ecr 0,nop,wscale 7], length 0
08:16:44.253784 IP (tos 0x0, ttl 59, id 0, offset 0, flags [DF], proto TCP (6),
length 60)
10.3.1.76.30080 > 192.168.61.12.39016: Flags [S.], cksum 0xd6ac (correct),
seq 1314433409, ack 4160678873, win 26480, options [mss 1336,sackOK,TS val
1217130186 ecr 459588853,nop,wscale 7], length 0
08:16:44.253802 IP (tos 0x0, ttl 64, id 12078, offset 0, flags [DF], proto TCP
(6), length 52)
192.168.61.12.39016 > 10.3.1.76.30080: Flags [.], cksum 0x092a (incorrect
-> 0x6b9c), ack 1, win 209, options [nop,nop,TS val 459588853 ecr 1217130186],
length 0
08:16:44.253961 IP (tos 0x0, ttl 64, id 12079, offset 0, flags [DF], proto TCP
(6), length 393)
192.168.61.12.39016 > 10.3.1.76.30080: Flags [P.], cksum 0x0a7f (incorrect
-> 0xaad9), seq 1:342, ack 1, win 209, options [nop,nop,TS val 459588853 ecr
1217130186], length 341
08:16:44.253975 IP (tos 0x0, ttl 64, id 12080, offset 0, flags [DF], proto TCP
(6), length 62)
192.168.61.12.39016 > 10.3.1.76.30080: Flags [P.], cksum 0x0934 (incorrect
-> 0x731b), seq 342:352, ack 1, win 209, options [nop,nop,TS val 459588853 ecr
1217130186], length 10
08:16:44.254670 IP (tos 0x0, ttl 59, id 18249, offset 0, flags [DF], proto TCP
(6), length 52)
10.3.1.76.30080 > 192.168.61.12.39016: Flags [.], cksum 0x6a3f (correct),
ack 342, win 216, options [nop,nop,TS val 1217130187 ecr 459588853], length 0
08:16:44.254679 IP (tos 0x0, ttl 59, id 18250, offset 0, flags [DF], proto TCP
(6), length 52)
10.3.1.76.30080 > 192.168.61.12.39016: Flags [.], cksum 0x6a35 (correct),
ack 352, win 216, options [nop,nop,TS val 1217130187 ecr 459588853], length 0
08:16:44.536331 IP (tos 0x0, ttl 59, id 18251, offset 0, flags [DF], proto TCP
(6), length 447)
10.3.1.76.30080 > 192.168.61.12.39016: Flags [P.], cksum 0x366c (correct),
seq 1:396, ack 352, win 216, options [nop,nop,TS val 1217130469 ecr 459588853],
length 395
08:16:44.536384 IP (tos 0x0, ttl 64, id 12081, offset 0, flags [DF], proto TCP
(6), length 52)
192.168.61.12.39016 > 10.3.1.76.30080: Flags [.], cksum 0x092a (incorrect
-> 0x6747), ack 396, win 218, options [nop,nop,TS val 459588924 ecr
1217130469], length 0
08:16:44.536793 IP (tos 0x0, ttl 64, id 12082, offset 0, flags [DF], proto TCP
(6), length 52)
192.168.61.12.39016 > 10.3.1.76.30080: Flags [F.], cksum 0x092a (incorrect
-> 0x6746), seq 352, ack 396, win 218, options [nop,nop,TS val 459588924 ecr
1217130469], length 0
08:16:44.537463 IP (tos 0x0, ttl 59, id 18252, offset 0, flags [DF], proto TCP
(6), length 52)
10.3.1.76.30080 > 192.168.61.12.39016: Flags [F.], cksum 0x6746 (correct),
seq 396, ack 353, win 216, options [nop,nop,TS val 1217130470 ecr 459588924],
length 0
08:16:44.537479 IP (tos 0x0, ttl 64, id 12083, offset 0, flags [DF], proto TCP
(6), length 52)
192.168.61.12.39016 > 10.3.1.76.30080: Flags [.], cksum 0x092a (incorrect
-> 0x6744), ack 397, win 218, options [nop,nop,TS val 459588924 ecr
1217130470], length 0
08:16:44.537854 IP (tos 0x0, ttl 64, id 37916, offset 0, flags [DF], proto TCP
(6), length 60)
192.168.61.12.36924 > 10.3.1.76.30060: Flags [S], cksum 0x0932 (incorrect
-> 0x9150), seq 4038938912, win 26730, options [mss 8910,sackOK,TS val
459588924 ecr 0,nop,wscale 7], length 0
08:16:44.539255 IP (tos 0x0, ttl 59, id 50179, offset 0, flags [DF], proto TCP
(6), length 40)
10.3.1.76.30060 > 192.168.61.12.36924: Flags [R.], cksum 0x5f45 (correct),
seq 0, ack 4038938913, win 0, length 0
08:16:44.543929 IP (tos 0x0, ttl 64, id 19283, offset 0, flags [DF], proto TCP
(6), length 60)
192.168.61.12.39020 > 10.3.1.76.30080: Flags [S], cksum 0x0932 (incorrect
-> 0x2f07), seq 225742445, win 26730, options [mss 8910,sackOK,TS val 459588926
ecr 0,nop,wscale 7], length 0
08:16:44.545060 IP (tos 0x0, ttl 59, id 0, offset 0, flags [DF], proto TCP (6),
length 60)
10.3.1.76.30080 > 192.168.61.12.39020: Flags [S.], cksum 0x4d0f (correct),
seq 1417769851, ack 225742446, win 26480, options [mss 1336,sackOK,TS val
1217130478 ecr 459588926,nop,wscale 7], length 0
08:16:44.545079 IP (tos 0x0, ttl 64, id 19284, offset 0, flags [DF], proto TCP
(6), length 52)
192.168.61.12.39020 > 10.3.1.76.30080: Flags [.], cksum 0x092a (incorrect
-> 0xe1fe), ack 1, win 209, options [nop,nop,TS val 459588926 ecr 1217130478],
length 0
08:16:44.545269 IP (tos 0x0, ttl 64, id 19285, offset 0, flags [DF], proto TCP
(6), length 257)
192.168.61.12.39020 > 10.3.1.76.30080: Flags [P.], cksum 0x09f7 (incorrect
-> 0x0088), seq 1:206, ack 1, win 209, options [nop,nop,TS val 459588926 ecr
1217130478], length 205
08:16:44.545285 IP (tos 0x0, ttl 64, id 19286, offset 0, flags [DF], proto TCP
(6), length 123)
192.168.61.12.39020 > 10.3.1.76.30080: Flags [P.], cksum 0x0971 (incorrect
-> 0x3c9d), seq 206:277, ack 1, win 209, options [nop,nop,TS val 459588926 ecr
1217130478], length 71
08:16:44.546145 IP (tos 0x0, ttl 59, id 40083, offset 0, flags [DF], proto TCP
(6), length 52)
10.3.1.76.30080 > 192.168.61.12.39020: Flags [.], cksum 0xe129 (correct),
ack 206, win 216, options [nop,nop,TS val 1217130479 ecr 459588926], length 0
08:16:44.546155 IP (tos 0x0, ttl 59, id 40084, offset 0, flags [DF], proto TCP
(6), length 52)
10.3.1.76.30080 > 192.168.61.12.39020: Flags [.], cksum 0xe0e2 (correct),
ack 277, win 216, options [nop,nop,TS val 1217130479 ecr 459588926], length 0
08:16:44.618837 IP (tos 0x0, ttl 59, id 40085, offset 0, flags [DF], proto TCP
(6), length 502)
10.3.1.76.30080 > 192.168.61.12.39020: Flags [P.], cksum 0xd7e4 (correct),
seq 1:451, ack 277, win 216, options [nop,nop,TS val 1217130551 ecr 459588926],
length 450
08:16:44.618851 IP (tos 0x0, ttl 64, id 19287, offset 0, flags [DF], proto TCP
(6), length 52)
192.168.61.12.39020 > 10.3.1.76.30080: Flags [.], cksum 0x092a (incorrect
-> 0xdec3), ack 451, win 218, options [nop,nop,TS val 459588945 ecr
1217130551], length 0
08:16:44.619276 IP (tos 0x0, ttl 64, id 19288, offset 0, flags [DF], proto TCP
(6), length 52)
192.168.61.12.39020 > 10.3.1.76.30080: Flags [F.], cksum 0x092a (incorrect
-> 0xdec2), seq 277, ack 451, win 218, options [nop,nop,TS val 459588945 ecr
1217130551], length 0
08:16:44.620518 IP (tos 0x0, ttl 59, id 40086, offset 0, flags [DF], proto TCP
(6), length 52)
10.3.1.76.30080 > 192.168.61.12.39020: Flags [F.], cksum 0xdec1 (correct),
seq 451, ack 278, win 216, options [nop,nop,TS val 1217130553 ecr 459588945],
length 0
08:16:44.620537 IP (tos 0x0, ttl 64, id 19289, offset 0, flags [DF], proto TCP
(6), length 52)
192.168.61.12.39020 > 10.3.1.76.30080: Flags [.], cksum 0x092a (incorrect
-> 0xdebf), ack 452, win 218, options [nop,nop,TS val 459588945 ecr
1217130553], length 0
08:16:44.620574 IP (tos 0x0, ttl 64, id 1630, offset 0, flags [DF], proto TCP
(6), length 60)
192.168.61.12.39022 > 10.3.1.76.30080: Flags [S], cksum 0x0932 (incorrect
-> 0x2458), seq 3815490319, win 26730, options [mss 8910,sackOK,TS val
459588945 ecr 0,nop,wscale 7], length 0
08:16:44.621687 IP (tos 0x0, ttl 59, id 0, offset 0, flags [DF], proto TCP (6),
length 60)
10.3.1.76.30080 > 192.168.61.12.39022: Flags [S.], cksum 0x8929 (correct),
seq 3464409704, ack 3815490320, win 26480, options [mss 1336,sackOK,TS val
1217130554 ecr 459588945,nop,wscale 7], length 0
08:16:44.621711 IP (tos 0x0, ttl 64, id 1631, offset 0, flags [DF], proto TCP
(6), length 52)
192.168.61.12.39022 > 10.3.1.76.30080: Flags [.], cksum 0x092a (incorrect
-> 0x1e19), ack 1, win 209, options [nop,nop,TS val 459588945 ecr 1217130554],
length 0
08:16:44.621880 IP (tos 0x0, ttl 64, id 1632, offset 0, flags [DF], proto TCP
(6), length 322)
192.168.61.12.39022 > 10.3.1.76.30080: Flags [P.], cksum 0x0a38 (incorrect
-> 0x9c63), seq 1:271, ack 1, win 209, options [nop,nop,TS val 459588945 ecr
1217130554], length 270
08:16:44.622537 IP (tos 0x0, ttl 59, id 16978, offset 0, flags [DF], proto TCP
(6), length 52)
10.3.1.76.30080 > 192.168.61.12.39022: Flags [.], cksum 0x1d03 (correct),
ack 271, win 216, options [nop,nop,TS val 1217130555 ecr 459588945], length 0
08:16:44.646611 IP (tos 0x0, ttl 59, id 16979, offset 0, flags [DF], proto TCP
(6), length 503)
10.3.1.76.30080 > 192.168.61.12.39022: Flags [P.], cksum 0x09e3 (correct),
seq 1:452, ack 271, win 216, options [nop,nop,TS val 1217130579 ecr 459588945],
length 451
08:16:44.646624 IP (tos 0x0, ttl 64, id 1633, offset 0, flags [DF], proto TCP
(6), length 52)
192.168.61.12.39022 > 10.3.1.76.30080: Flags [.], cksum 0x092a (incorrect
-> 0x1b1f), ack 452, win 218, options [nop,nop,TS val 459588952 ecr
1217130579], length 0
08:16:44.646873 IP (tos 0x0, ttl 64, id 1634, offset 0, flags [DF], proto TCP
(6), length 52)
192.168.61.12.39022 > 10.3.1.76.30080: Flags [F.], cksum 0x092a (incorrect
-> 0x1b1e), seq 271, ack 452, win 218, options [nop,nop,TS val 459588952 ecr
1217130579], length 0
08:16:44.647473 IP (tos 0x0, ttl 59, id 16980, offset 0, flags [DF], proto TCP
(6), length 52)
10.3.1.76.30080 > 192.168.61.12.39022: Flags [F.], cksum 0x1b1e (correct),
seq 452, ack 272, win 216, options [nop,nop,TS val 1217130580 ecr 459588952],
length 0
08:16:44.647487 IP (tos 0x0, ttl 64, id 1635, offset 0, flags [DF], proto TCP
(6), length 52)
192.168.61.12.39022 > 10.3.1.76.30080: Flags [.], cksum 0x092a (incorrect
-> 0x1b1c), ack 453, win 218, options [nop,nop,TS val 459588952 ecr
1217130580], length 0
08:16:44.647914 IP (tos 0x0, ttl 64, id 34776, offset 0, flags [DF], proto TCP
(6), length 60)
192.168.61.12.39024 > 10.3.1.76.30080: Flags [S], cksum 0x0932 (incorrect
-> 0xd692), seq 2981577344, win 26730, options [mss 8910,sackOK,TS val
459588952 ecr 0,nop,wscale 7], length 0
08:16:44.648826 IP (tos 0x0, ttl 59, id 0, offset 0, flags [DF], proto TCP (6),
length 60)
10.3.1.76.30080 > 192.168.61.12.39024: Flags [S.], cksum 0x43d0 (correct),
seq 752110475, ack 2981577345, win 26480, options [mss 1336,sackOK,TS val
1217130582 ecr 459588952,nop,wscale 7], length 0
08:16:44.648846 IP (tos 0x0, ttl 64, id 34777, offset 0, flags [DF], proto TCP
(6), length 52)
192.168.61.12.39024 > 10.3.1.76.30080: Flags [.], cksum 0x092a (incorrect
-> 0xd8bf), ack 1, win 209, options [nop,nop,TS val 459588952 ecr 1217130582],
length 0
08:16:44.649015 IP (tos 0x0, ttl 64, id 34778, offset 0, flags [DF], proto TCP
(6), length 371)
192.168.61.12.39024 > 10.3.1.76.30080: Flags [P.], cksum 0x0a69 (incorrect
-> 0xb3fe), seq 1:320, ack 1, win 209, options [nop,nop,TS val 459588952 ecr
1217130582], length 319
08:16:44.649735 IP (tos 0x0, ttl 59, id 48090, offset 0, flags [DF], proto TCP
(6), length 52)
10.3.1.76.30080 > 192.168.61.12.39024: Flags [.], cksum 0xd779 (correct),
ack 320, win 216, options [nop,nop,TS val 1217130582 ecr 459588952], length 0
08:16:48.858548 IP (tos 0x0, ttl 64, id 34386, offset 0, flags [DF], proto TCP
(6), length 40)
192.168.61.12.48786 > 13.95.105.108.443: Flags [.], cksum 0x749a (incorrect
-> 0x0092), ack 172320321, win 488, length 0
08:16:49.186104 IP (tos 0x0, ttl 25, id 25835, offset 0, flags [DF], proto TCP
(6), length 40)
13.95.105.108.443 > 192.168.61.12.48786: Flags [.], cksum 0x0158 (correct),
ack 1, win 289, length 0
08:16:50.047812 IP (tos 0x0, ttl 59, id 48091, offset 0, flags [DF], proto TCP
(6), length 230)
10.3.1.76.30080 > 192.168.61.12.39024: Flags [P.], cksum 0x517d (correct),
seq 1:179, ack 320, win 216, options [nop,nop,TS val 1217135980 ecr 459588952],
length 178
08:16:50.047866 IP (tos 0x0, ttl 64, id 34779, offset 0, flags [DF], proto TCP
(6), length 52)
192.168.61.12.39024 > 10.3.1.76.30080: Flags [.], cksum 0x092a (incorrect
-> 0xbc69), ack 179, win 218, options [nop,nop,TS val 459590302 ecr
1217135980], length 0
08:16:50.048407 IP (tos 0x0, ttl 64, id 34780, offset 0, flags [DF], proto TCP
(6), length 52)
192.168.61.12.39024 > 10.3.1.76.30080: Flags [F.], cksum 0x092a (incorrect
-> 0xbc68), seq 320, ack 179, win 218, options [nop,nop,TS val 459590302 ecr
1217135980], length 0
08:16:50.049159 IP (tos 0x0, ttl 59, id 48092, offset 0, flags [DF], proto TCP
(6), length 52)
10.3.1.76.30080 > 192.168.61.12.39024: Flags [F.], cksum 0xbc67 (correct),
seq 179, ack 321, win 216, options [nop,nop,TS val 1217135982 ecr 459590302],
length 0
08:16:50.049174 IP (tos 0x0, ttl 64, id 34781, offset 0, flags [DF], proto TCP
(6), length 52)
192.168.61.12.39024 > 10.3.1.76.30080: Flags [.], cksum 0x092a (incorrect
-> 0xbc65), ack 180, win 218, options [nop,nop,TS val 459590302 ecr
1217135982], length 0
08:16:57.626548 IP (tos 0x0, ttl 64, id 51258, offset 0, flags [DF], proto TCP
(6), length 40)
192.168.61.12.48782 > 13.95.105.108.443: Flags [.], cksum 0x749a (incorrect
-> 0x7293), ack 2942428865, win 488, length 0
08:16:57.952089 IP (tos 0x0, ttl 24, id 23746, offset 0, flags [DF], proto TCP
(6), length 40)
13.95.105.108.443 > 192.168.61.12.48782: Flags [.], cksum 0x7359 (correct),
ack 1, win 289, length 0
08:17:04.255232 IP (tos 0x0, ttl 64, id 15611, offset 0, flags [DF], proto TCP
(6), length 73)
192.168.61.12.48784 > 13.95.105.108.443: Flags [P.], cksum 0x74bb
(incorrect -> 0x6430), seq 933121322:933121355, ack 3853971194, win 849, length
33
08:17:04.255254 IP (tos 0x0, ttl 64, id 15612, offset 0, flags [DF], proto TCP
(6), length 73)
192.168.61.12.48784 > 13.95.105.108.443: Flags [P.], cksum 0x74bb
(incorrect -> 0xcff0), seq 33:66, ack 1, win 849, length 33
08:17:04.255260 IP (tos 0x0, ttl 64, id 15613, offset 0, flags [DF], proto TCP
(6), length 72)
192.168.61.12.48784 > 13.95.105.108.443: Flags [P.], cksum 0x74ba
(incorrect -> 0x3950), seq 66:98, ack 1, win 849, length 32
08:17:04.582158 IP (tos 0x0, ttl 25, id 3500, offset 0, flags [DF], proto TCP
(6), length 40)
13.95.105.108.443 > 192.168.61.12.48784: Flags [.], cksum 0x1fcb (correct),
ack 98, win 354, length 0
08:17:05.039239 IP (tos 0x0, ttl 64, id 15614, offset 0, flags [DF], proto TCP
(6), length 73)
192.168.61.12.48784 > 13.95.105.108.443: Flags [P.], cksum 0x74bb
(incorrect -> 0x4f25), seq 98:131, ack 1, win 849, length 33
08:17:05.039256 IP (tos 0x0, ttl 64, id 15615, offset 0, flags [DF], proto TCP
(6), length 73)
192.168.61.12.48784 > 13.95.105.108.443: Flags [P.], cksum 0x74bb
(incorrect -> 0x2b06), seq 131:164, ack 1, win 849, length 33
08:17:05.039262 IP (tos 0x0, ttl 64, id 15616, offset 0, flags [DF], proto TCP
(6), length 72)
192.168.61.12.48784 > 13.95.105.108.443: Flags [P.], cksum 0x74ba
(incorrect -> 0x7ff8), seq 164:196, ack 1, win 849, length 32
08:17:05.375198 IP (tos 0x0, ttl 25, id 3501, offset 0, flags [DF], proto TCP
(6), length 40)
13.95.105.108.443 > 192.168.61.12.48784: Flags [.], cksum 0x1f69 (correct),
ack 196, win 354, length 0
08:17:06.296183 IP (tos 0x0, ttl 25, id 3502, offset 0, flags [DF], proto TCP
(6), length 178)
13.95.105.108.443 > 192.168.61.12.48784: Flags [P.], cksum 0xc3d6
(correct), seq 1:139, ack 196, win 354, length 138
08:17:06.296229 IP (tos 0x0, ttl 64, id 15617, offset 0, flags [DF], proto TCP
(6), length 40)
192.168.61.12.48784 > 13.95.105.108.443: Flags [.], cksum 0x749a (incorrect
-> 0x1cf0), ack 139, win 849, length 0
08:17:06.303170 IP (tos 0x0, ttl 25, id 3503, offset 0, flags [DF], proto TCP
(6), length 242)
13.95.105.108.443 > 192.168.61.12.48784: Flags [P.], cksum 0xb7a6
(correct), seq 139:341, ack 196, win 354, length 202
08:17:06.303200 IP (tos 0x0, ttl 64, id 15618, offset 0, flags [DF], proto TCP
(6), length 40)
192.168.61.12.48784 > 13.95.105.108.443: Flags [.], cksum 0x749a (incorrect
-> 0x1c26), ack 341, win 849, length 0
08:17:07.087350 IP (tos 0x0, ttl 64, id 15619, offset 0, flags [DF], proto TCP
(6), length 73)
192.168.61.12.48784 > 13.95.105.108.443: Flags [P.], cksum 0x74bb
(incorrect -> 0x3c68), seq 196:229, ack 341, win 849, length 33
08:17:07.087368 IP (tos 0x0, ttl 64, id 15620, offset 0, flags [DF], proto TCP
(6), length 73)
192.168.61.12.48784 > 13.95.105.108.443: Flags [P.], cksum 0x74bb
(incorrect -> 0x587a), seq 229:262, ack 341, win 849, length 33
08:17:07.087375 IP (tos 0x0, ttl 64, id 15621, offset 0, flags [DF], proto TCP
(6), length 70)
192.168.61.12.48784 > 13.95.105.108.443: Flags [P.], cksum 0x74b8
(incorrect -> 0xc7bc), seq 262:292, ack 341, win 849, length 30
08:17:07.412733 IP (tos 0x0, ttl 25, id 3504, offset 0, flags [DF], proto TCP
(6), length 40)
13.95.105.108.443 > 192.168.61.12.48784: Flags [.], cksum 0x1db5 (correct),
ack 292, win 354, length 0
08:17:09.169183 IP (tos 0x0, ttl 25, id 3505, offset 0, flags [DF], proto TCP
(6), length 320)
13.95.105.108.443 > 192.168.61.12.48784: Flags [P.], cksum 0x0f75
(correct), seq 341:621, ack 292, win 354, length 280
08:17:09.169231 IP (tos 0x0, ttl 64, id 15622, offset 0, flags [DF], proto TCP
(6), length 40)
192.168.61.12.48784 > 13.95.105.108.443: Flags [.], cksum 0x749a (incorrect
-> 0x1aae), ack 621, win 849, length 0
_______________________________________________
Clearwater mailing list
[email protected]
http://lists.projectclearwater.org/mailman/listinfo/clearwater_lists.projectclearwater.org
