This is an automated email from the ASF dual-hosted git repository.
bcall pushed a commit to branch 8.0.x
in repository https://gitbox.apache.org/repos/asf/trafficserver.git
The following commit(s) were added to refs/heads/8.0.x by this push:
new 3155ca5 Autest test extension modification using opensourced testing
tools
3155ca5 is described below
commit 3155ca5d169d0ec19f8393c17018962d1f0c4ca1
Author: Jesse Zhang <macisasandw...@gmail.com>
AuthorDate: Wed Jan 23 13:00:40 2019 -0600
Autest test extension modification using opensourced testing tools
Removes MicroServer, MicroDNS, Traffic-Replay, and SessionValidation out of
tests/tools into their independent BitBucket repo together with Autest.
All the relevant test extensions and the affected tests have been modified
to work with the new setup.
tests/bootstrap.py has also been updated to automatically install all the
dependencies for the tests to run properly.
Updated tests/README.md and cleaned up
---
tests/README.md | 29 +-
tests/bootstrap.py | 1 +
tests/gold_tests/autest-site/microDNS.test.ext | 13 +-
tests/gold_tests/autest-site/microserver.test.ext | 176 ++---
.../gold_tests/autest-site/traffic_replay.test.ext | 91 +++
.../chunked_encoding/chunked_encoding.test.py | 2 +-
tests/gold_tests/h2/gold/post_chunked.gold | 2 +-
tests/gold_tests/h2/http2.test.py | 15 +-
tests/gold_tests/remap/remap_https.test.py | 2 +-
tests/tools/lib/IPConstants.py | 48 --
tests/tools/lib/result.py | 117 ----
tests/tools/microDNS/uDNS.py | 207 ------
tests/tools/microServer/README.md | 49 --
tests/tools/microServer/uWServer.py | 716 ---------------------
.../{microServer => microserver}/ssl/server.crt | 0
.../{microServer => microserver}/ssl/server.pem | 0
tests/tools/sessionvalidation/__init__.py | 17 -
tests/tools/sessionvalidation/badsession.py | 35 -
tests/tools/sessionvalidation/request.py | 48 --
tests/tools/sessionvalidation/response.py | 49 --
tests/tools/sessionvalidation/session.py | 45 --
tests/tools/sessionvalidation/sessionvalidation.py | 259 --------
tests/tools/sessionvalidation/transaction.py | 40 --
tests/tools/traffic-replay/Config.py | 34 -
tests/tools/traffic-replay/NonSSL.py | 192 ------
tests/tools/traffic-replay/RandomReplay.py | 91 ---
tests/tools/traffic-replay/SSLReplay.py | 233 -------
tests/tools/traffic-replay/Scheduler.py | 88 ---
tests/tools/traffic-replay/WorkerTask.py | 49 --
tests/tools/traffic-replay/__main__.py | 44 --
tests/tools/traffic-replay/extractHeader.py | 91 ---
tests/tools/traffic-replay/h2Replay.py | 331 ----------
tests/tools/traffic-replay/mainProcess.py | 76 ---
tests/unit_tests/Makefile.am | 1 +
34 files changed, 217 insertions(+), 2974 deletions(-)
diff --git a/tests/README.md b/tests/README.md
index 619c0c0..8d07029 100644
--- a/tests/README.md
+++ b/tests/README.md
@@ -6,35 +6,34 @@ This directory contains different tests for Apache
Trafficserver. It is recommen
## Layout
The current layout is:
-**gold_tests/** - contains all the TSQA v4 based tests that run on the
Reusable Gold Testing System (AuTest)
-**tools/** - contain programs used to help with testing.
-
-In the future a directory called **"unit/"** will be added for adding unit
tests based on some standardized testing system.
-
+**gold_tests/** - contains all the TSQA v4 based tests that run on the
Reusable Gold Testing System (AuTest)
+**tools/** - contains programs used to help with testing.
+**include/** - contains headers used for unit testing.
## Scripts
-To help with easy running of the tests, there is a autest.sh and bootstrap.py
file.
+To help with easy running of the tests, there is autest.sh and bootstrap.py.
### autest.sh
-This file is a simple wrapper that will call the AuTest program in a python
virtualenv. If the virtualenv is not setup it will try to install system. That
will set up the Reusable Gold Testing System on most systems in a Python
virtual environment. The wrapper add some basic options to the command to point
to the location of the tests. Add --help for more details on options for
running autest test system.
+This file is a simple wrapper that will call the Reusable Gold Testing System
(Autest) program in a python virtualenv. If the virtualenv is not setup, the
script will try to install it on the system. That will set up the Autest on
most systems in a Python virtual environment. The wrapper add some basic
options to the command to point to the location of the tests. Use --help for
more details on options for running Autest.
### bootstrap.py
-This script should try to install python35 or better on the system, and needed
python packages for running the tests.
+This script will try to install python35 or better on the system, and the
needed python packages for running the tests.
-# Advance setup
+# Advanced setup
-AuTest can be install manually instead of using the wrapper script. The
advange of this is that it is often easier to debug issues with the testing
system, or the tests. There are two ways this can be done.
+AuTest and the relevant tools can be install manually instead of using the
wrapper script. The advange of this is that it is often easier to debug issues
with the testing system, or the tests. There are two ways this can be done.
1. run the bootstrap script then source the path with a "source
./env-test/bin/activate" command. At this point autest command should run
without the wrapper script
2. The other way is to make sure you install python 3.5 or better on your
system. From there install these python packages ( ie pip install ):
- hyper
- - git+https://bitbucket.org/dragon512/reusable-gold-testing-system.git
+ - git+https://bitbucket.org/autestsuite/reusable-gold-testing-system.git
+ -
[traffic-replay](https://bitbucket.org/autestsuite/trafficreplay/src/master/)
(This will automatically install
[MicroDNS](https://bitbucket.org/autestsuite/microdns/src/master/),
[MicroServer](https://bitbucket.org/autestsuite/microserver/src/master/),
[TrafficReplayLibrary](https://bitbucket.org/autestsuite/trafficreplaylibrary/src/master/),
and dnslib as part of the dependencies.)
# Writting tests for AuTest
When writting for the AuTest system please refer to the current documenation
on the [online
wiki](https://bitbucket.org/dragon512/reusable-gold-testing-system/wiki/Home)
for general use of the system.
## Documenation of AuTest extension for ATS.
-Autest allows for the creation of extension to help specilaize and simplify
test writting for a given application domian. Minus API addition the extension
code will check that python 3.5 or better is used. There is also a new command
line argumented added:
+Autest allows for the creation of extensions to help specialize and simplify
test writing for a given application domain. Minus API addition the extension
code will check that python 3.5 or better is used. There is also a new command
line argumented added specifically for Trafficserver:
--ats-bin < path to bin directory >
@@ -45,7 +44,7 @@ This command line argument will point to your build of ATS
you want to test. At
* command - optional argument defining what process to use. Defaults to
traffic_server.
* select_ports - have the testing system auto select the ports to use for
this instance of ATS
-This function will define a sandbox for an instance of trafficserver to run
under. The function will return a AuTest process object that will have a number
of files and variables define for making it easier to define a test.
+This function will define a sandbox for an instance of trafficserver to run
under. The function will return a AuTest process object that will have a number
of files and variables defined to make it easier for test definition.
#### Environment
The environment of the process will have a number of added environment
variables to control trafficserver running the in the sandbox location
correctly. This can be used to easily setup other commands that should run
under same environment.
@@ -64,7 +63,7 @@ tr.Processes.Default.Env=ts.Env
```
#### Variables
-These are the current variable that are define dynamically
+These are the current variables that are defined dynamically for Trafficserver
port - the ipv4 port to listen on
portv6 - the ipv4 port to listen on
@@ -72,7 +71,7 @@ manager_port - the manager port used. This is set even is
select_port is False
admin_port - the admin port used. This is set even is select_port is False
#### File objects
-A number of file object are define to help with adding values to a given
configuration value to for a test, or testing a value exists in a log file.
File that are defined currently are:
+A number of file objects are defined to help with adding values to a given
configuration value to for a test, or testing a value exists in a log file.
File that are defined currently are:
##### log files
* squid.log
diff --git a/tests/bootstrap.py b/tests/bootstrap.py
index 4446f26..0c472cd 100755
--- a/tests/bootstrap.py
+++ b/tests/bootstrap.py
@@ -31,6 +31,7 @@ pip_packages = [
"requests",
"dnslib",
"httpbin",
+ "traffic-replay" # this should install TRLib, MicroServer, MicroDNS,
Traffic-Replay
]
diff --git a/tests/gold_tests/autest-site/microDNS.test.ext
b/tests/gold_tests/autest-site/microDNS.test.ext
index e45a496..fdeffe0 100644
--- a/tests/gold_tests/autest-site/microDNS.test.ext
+++ b/tests/gold_tests/autest-site/microDNS.test.ext
@@ -16,12 +16,16 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-from ports import get_port
import json
import os
import sys
+import trlib.ipconstants as IPConstants
+from ports import get_port
+
# AddRecord registers a list of ip address against hostname
+
+
def AddRecord(hostname, list_ip_addr):
record = dict()
@@ -69,7 +73,6 @@ def addRecords(self, records=None, jsonFile=None):
def MakeDNServer(obj, name, filename="dns_file.json", port=False,
ip='INADDR_LOOPBACK', rr=False, default=None, options={}):
- server_path = os.path.join(obj.Variables.AtsTestToolsDir,
'microDNS/uDNS.py')
data_dir = os.path.join(obj.RunDirectory, name)
filepath = os.path.join(data_dir, filename)
obj.Variables.zone_file = filepath
@@ -90,7 +93,7 @@ def MakeDNServer(obj, name, filename="dns_file.json",
port=False, ip='INADDR_LOO
p = obj.Processes.Process(name)
if (port == False):
port = get_port(p, "Port")
- command = "python3 {0} {1} {2} {3}".format(server_path, ip, port, filepath)
+ command = "microdns {0} {1} {2}".format(ip, port, filepath)
if rr:
command += " --rr"
@@ -101,10 +104,6 @@ def MakeDNServer(obj, name, filename="dns_file.json",
port=False, ip='INADDR_LOO
p.Variables.DataDir = data_dir
p.ReturnCode = 0
- # to get the IP keywords in tools/lib
- sys.path.append(obj.Variables.AtsTestToolsDir)
- import lib.IPConstants as IPConstants
-
if IPConstants.isIPv6(ip):
p.Ready = When.PortOpenv6(port)
else:
diff --git a/tests/gold_tests/autest-site/microserver.test.ext
b/tests/gold_tests/autest-site/microserver.test.ext
index 17f93d9..bef027f 100644
--- a/tests/gold_tests/autest-site/microserver.test.ext
+++ b/tests/gold_tests/autest-site/microserver.test.ext
@@ -16,14 +16,20 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-from autest.api import AddWhenFunction
-from ports import get_port
import json
import socket
import ssl
import time
import sys
+from autest.api import AddWhenFunction
+from ports import get_port
+
+import trlib.ipconstants as IPConstants
+from trlib import Transaction, Request, Response, Session
+
+DEFAULT_LOOKUP_KEY = '{PATH}'
+
def addMethod(self, testName, request_header, functionName):
return
@@ -52,51 +58,15 @@ def getHeaderFieldVal(request_header, field):
return val
# addResponse adds customized response with respect to request_header.
request_header and response_header are both dictionaries
+def addResponse(self, filename, request_header, response_header):
+ client_request = Request.fromRequestLine(request_header["headers"],
request_header["body"], None if "options" not in request_header else
request_header["options"])
+ server_response = Response.fromRequestLine(response_header["headers"],
response_header["body"], None if "options" not in response_header else
response_header["options"])
+ # timestamp field is left None because that needs to be revised for better
implementation
+ txn = Transaction(client_request, None, server_response, None, None, None)
-def addResponse(self, filename, request_header, response_header):
- requestline = request_header["headers"].split("\r\n")[0]
- host_ = ""
- path_ = ""
- if requestline:
- url_part = requestline.split(" ")
- if len(url_part) > 1:
- if url_part[1].startswith("http"):
- path_ = url_part[1].split("/", 2)[2]
- host_, path_ = path_.split("/", 1)
- else:
- path_ = url_part[1].split("/", 1)[1]
-
- kpath = ""
-
- argsList = []
- keyslist = self.Variables.lookup_key.split("}")
- for keystr in keyslist:
- if keystr == '{PATH':
- kpath = kpath + path_
- continue
- if keystr == '{HOST':
- kpath = kpath + host_
- continue
- if keystr == '': # empty
- continue
- stringk = keystr.replace("{%", "")
- argsList.append(stringk)
- KeyList = []
- for argsL in argsList:
- field_val = getHeaderFieldVal(request_header, argsL)
- if field_val != None:
- KeyList.append(field_val)
- rl = "".join(KeyList) + kpath
- txn = dict()
- txn["timestamp"] = ""
- txn["uuid"] = rl
- txn["request"] = request_header
- txn["response"] = response_header
absFilepath = os.path.join(self.Variables.DataDir, filename)
addTransactionToSession(txn, absFilepath)
- # absFilepath=os.path.abspath(filename)
- # self.Setup.CopyAs(absFilepath,self.Variables.DataDir)
return
# adds transaction in json format to the specified file
@@ -110,18 +80,25 @@ def addTransactionToSession(txn, JFile):
jf = open(JFile, 'r')
jsondata = json.load(jf)
+ # hard coding only 1 session per file
+ # since for the purpose of testing, we don't need multiple sessions in a
file
if jsondata == None:
- jsondata = dict()
- jsondata["version"] = '0.2'
- jsondata["timestamp"] = "1234567890.098"
- jsondata["encoding"] = "url_encoded"
- jsondata["txns"] = list()
- jsondata["txns"].append(txn)
+ jsondata = {}
+ jsondata["sessions"] = []
+
+ jsondata["sessions"].append(Session(JFile.split("/")[-1], None, None,
[txn]).toJSON())
+ jsondata["meta"] = {}
+ jsondata["meta"]["version"] = "1.0"
else:
- jsondata["txns"].append(txn)
+ # hardcoding 0 because for testing we only have 1 session
+ jsondata["sessions"][0]["transactions"].append(txn.toJSON())
+
with open(JFile, 'w+') as jf:
jf.write(json.dumps(jsondata))
+def addSessionFromFiles(self, session_dir):
+ self.Setup.Copy(session_dir, self.Variables.DataDir)
+
# make headers with the key and values provided
def makeHeader(self, requestString, **kwargs):
@@ -132,63 +109,83 @@ def makeHeader(self, requestString, **kwargs):
return headerStr
-def uServerUpAndRunning(host, port, isSsl, isIPv6):
+def uServerUpAndRunning(serverHost, port, isSsl, isIPv6, request,
clientcert='', clientkey=''):
if isIPv6:
plain_sock = socket.socket(socket.AF_INET6)
else:
plain_sock = socket.socket(socket.AF_INET)
- sock = ssl.wrap_socket(plain_sock) if isSsl else plain_sock
+ if isSsl:
+ if clientcert != '' or clientkey != '':
+ sock = ssl.wrap_socket(plain_sock, keyfile=clientkey,
certfile=clientcert)
+ else:
+ sock = ssl.wrap_socket(plain_sock)
+ else:
+ sock = plain_sock
+
try:
- sock.connect((host, port))
+ sock.connect((serverHost, port))
except ConnectionRefusedError:
return False
- sock.sendall("GET /ruok HTTP/1.1\r\nHost:
{}\r\n\r\n".format(host).encode())
- decoded_output=''
+ sock.sendall(request.encode())
+ decoded_output = ''
while True:
+ host.WriteDebug("??")
output = sock.recv(4096) # suggested bufsize from docs.python.org
+ host.WriteDebug("!!")
if len(output) <= 0:
break
else:
- decoded_output+=output.decode()
+ decoded_output += output.decode()
sock.close()
sock = None
- expected_response="HTTP/1.1 200 OK\r\nConnection: close\r\nContent-Length:
4\r\n\r\nimok"
+ expected_response = "HTTP/1.1 200 OK\r\nConnection:
close\r\nContent-Length: 4\r\n\r\nimok"
if decoded_output == expected_response:
return True
- raise RuntimeError('\n'.join([
- 'Got invalid response from microserver:',
- '----',
- decoded_output,
- '----']))
-AddWhenFunction(uServerUpAndRunning)
+ host.WriteError('\n'.join([
+ 'Got invalid response from microserver:',
+ '----',
+ decoded_output,
+ '----']))
+
+
+AddWhenFunction(uServerUpAndRunning)
-def MakeOriginServer(obj, name, port=False, ip='INADDR_LOOPBACK', delay=False,
ssl=False, lookup_key='{PATH}', mode='test', options={}):
- # to get the IP keywords in tools/lib
- sys.path.append(obj.Variables.AtsTestToolsDir)
- import lib.IPConstants as IPConstants
- server_path = os.path.join(obj.Variables.AtsTestToolsDir,
'microServer/uWServer.py')
+def MakeOriginServer(obj, name, port=None, s_port=None, ip='INADDR_LOOPBACK',
delay=None, ssl=False, lookup_key=DEFAULT_LOOKUP_KEY, clientcert='',
clientkey='', both=False, options={}):
data_dir = os.path.join(obj.RunDirectory, name)
- # create Process
p = obj.Processes.Process(name)
- if (port == False):
- port = get_port(p, "Port")
-
ipaddr = IPConstants.getIP(ip)
- if (delay == False):
- delay = 0
+ command = "microserver --data-dir {0} --ip_address {1} --lookupkey
'{2}'".format(data_dir, ipaddr, lookup_key)
+
+ if delay:
+ command += " --delay {0}".format(delay)
+
+ if both or ssl:
+ if not s_port:
+ s_port = get_port(p, "SSL_Port")
+
+ command += " --both" if both else " --ssl"
+ key = clientkey if clientkey else
os.path.join(obj.Variables["AtsTestToolsDir"], "microserver", "ssl",
"server.pem")
+ cert = clientcert if clientcert else
os.path.join(obj.Variables["AtsTestToolsDir"], "microserver", "ssl",
"server.crt")
+ command += " --key {0}".format(key)
+ command += " --cert {0}".format(cert)
+ command += " --s_port {0}".format(s_port)
- command = "python3 {0} --data-dir {1} --port {2} --ip_address {3} --delay
{4} -m test --ssl {5} --lookupkey '{6}' -m {7}".format(
- server_path, data_dir, port, ipaddr, delay, ssl, lookup_key, mode)
+ # this might break if user specifies both both and ssl
+ if not ssl: # in both or HTTP only mode
+ if not port:
+ port = get_port(p, "Port")
+
+ command += " --port {0}".format(port)
for flag, value in options.items():
- command += " {} {}".format(flag, value)
+ command += " {} {}".format(flag, value if value else '')
p.Command = command
p.Setup.MakeDir(data_dir)
@@ -196,20 +193,33 @@ def MakeOriginServer(obj, name, port=False,
ip='INADDR_LOOPBACK', delay=False, s
p.Variables.lookup_key = lookup_key
AddMethodToInstance(p, addResponse)
AddMethodToInstance(p, addTransactionToSession)
+ AddMethodToInstance(p, addSessionFromFiles)
- # Set up health check.
- addResponse(p, "healthcheck.json", {
- "headers": "GET /ruok HTTP/1.1\r\nHost: {}\r\n\r\n".format(ipaddr),
+ custom_lookup_header = ''
+ keys = lookup_key.split("}")
+
+ for key in keys:
+ if key not in ['{PATH', '{URL', '{HOST', '{%Host']:
+ k = key.replace("{%", "")
+
+ if len(k) > 0:
+ custom_lookup_header += '{0}: healthcheck\r\n'.format(k)
+
+ healthcheck_request = {
+ "headers": "GET /ruok HTTP/1.1\r\nHost: {0}\r\n{1}\r\n".format(ipaddr,
custom_lookup_header),
"timestamp": "1469733493.993",
"body": ""
- }, {
+ }
+
+ # Set up health check.
+ addResponse(p, "healthcheck.json", healthcheck_request, {
"headers": "HTTP/1.1 200 OK\r\nConnection: close\r\n\r\n",
"timestamp": "1469733493.993",
"body": "imok",
- "options": "skipHooks"
+ "options": {"skipHooks": None}
})
- p.Ready = When.uServerUpAndRunning(ipaddr, port, ssl,
IPConstants.isIPv6(ip))
+ p.Ready = When.uServerUpAndRunning(ipaddr, s_port if ssl else port, ssl,
IPConstants.isIPv6(ip), healthcheck_request["headers"], clientcert=clientcert,
clientkey=clientkey)
p.ReturnCode = Any(None, 0)
return p
diff --git a/tests/gold_tests/autest-site/traffic_replay.test.ext
b/tests/gold_tests/autest-site/traffic_replay.test.ext
new file mode 100644
index 0000000..340fae9
--- /dev/null
+++ b/tests/gold_tests/autest-site/traffic_replay.test.ext
@@ -0,0 +1,91 @@
+'''
+'''
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# default 'mixed' for connection type since it doesn't hurt
+def Replay(obj, name, replay_dir, key=None, cert=None, conn_type='mixed',
options={}):
+ # ATS setup - one line because we leave records and remap config to user
+ ts = obj.MakeATSProcess("ts", select_ports=False) # select ports can be
disabled once we add ssl port selection in extension
+
+ # TEMP
+ ts.Variables.ssl_port = 4443
+
+ ts.addSSLfile(os.path.join(obj.Variables["AtsTestToolsDir"],
"microserver", "ssl", "server.pem"))
+ ts.addSSLfile(os.path.join(obj.Variables["AtsTestToolsDir"],
"microserver", "ssl", "server.crt"))
+
+ ts.Disk.ssl_multicert_config.AddLine(
+ 'dest_ip=* ssl_cert_name=server.pem ssl_key_name=server.pem'
+ )
+
+ # MicroServer setup - NOTE: expand to multiple microserver in future?
+ server = obj.MakeOriginServer("server", both=True, lookup_key='{%uuid}')
+ server.addSessionFromFiles(replay_dir)
+
+ # MicroDNS setup
+ dns = obj.MakeDNServer("dns", default=['127.0.0.1'])
+
+ # Traffic Replay setup
+ data_dir = os.path.join(obj.RunDirectory, name)
+
+ # NOTE: we are forcing mixed connection types for now for the sake of
simplicity
+
+ # if conn_type != 'nossl':
+ # if not key:
+ # host.WriteError("Must provide SSL key to traffic-replay.")
+
+ # if not cert:
+ # host.WriteError("Must provide SSL key to traffic-replay.")
+
+ # if not ts.Variables.ssl_port:
+ # host.WriteError("Must set traffic server with an ssl port")
+
+ # NOTE: does this need change?
+ hostIP = '127.0.0.1'
+
+ if not key:
+ key = os.path.join(obj.Variables["AtsTestToolsDir"], "microserver",
"ssl", "server.pem")
+
+ if not cert:
+ cert = os.path.join(obj.Variables["AtsTestToolsDir"], "microserver",
"ssl", "server.crt")
+
+ command = 'traffic-replay --log_dir {0} --type {1} --verify --host {2}
--port {3} --s_port {4} '.format(data_dir, conn_type, hostIP,
ts.Variables.port, ts.Variables.ssl_port)
+
+ if key:
+ command += "-k {0} ".format(key)
+
+ if cert:
+ command += "--ca_cert {0} ".format(cert)
+
+ if options:
+ for flag, value in options.items():
+ command += "{} {} ".format(flag, value if value else '')
+
+ tr = obj.AddTestRun(name)
+ tr.Command = command
+ # tr.Command = "echo Hi"
+ tr.Setup.MakeDir(data_dir)
+ tr.Setup.Copy(replay_dir, data_dir)
+ tr.Processes.Default.StartBefore(server)
+ tr.Processes.Default.StartBefore(ts,
ready=When.PortOpen(ts.Variables.ssl_port))
+ tr.Processes.Default.StartBefore(dns)
+ tr.ReturnCode = Any(None, 0)
+ tr.Processes.Default.Streams.All = Testers.ExcludesExpression("FAIL", "No
fails allowed.")
+
+ # return all the stuff in case user wants to do extra optimization
+ return (ts, server, dns, tr)
+
+AddTestRunSet(Replay)
diff --git a/tests/gold_tests/chunked_encoding/chunked_encoding.test.py
b/tests/gold_tests/chunked_encoding/chunked_encoding.test.py
index f511a21..5803d62 100644
--- a/tests/gold_tests/chunked_encoding/chunked_encoding.test.py
+++ b/tests/gold_tests/chunked_encoding/chunked_encoding.test.py
@@ -83,7 +83,7 @@ ts.Disk.remap_config.AddLine(
'map http://www.yetanotherexample.com
http://127.0.0.1:{0}'.format(server3.Variables.Port)
)
ts.Disk.remap_config.AddLine(
- 'map https://www.anotherexample.com
https://127.0.0.1:{0}'.format(server2.Variables.Port, ts.Variables.ssl_port)
+ 'map https://www.anotherexample.com
https://127.0.0.1:{0}'.format(server2.Variables.SSL_Port, ts.Variables.ssl_port)
)
diff --git a/tests/gold_tests/h2/gold/post_chunked.gold
b/tests/gold_tests/h2/gold/post_chunked.gold
index ad47100..0ff06d1 100644
--- a/tests/gold_tests/h2/gold/post_chunked.gold
+++ b/tests/gold_tests/h2/gold/post_chunked.gold
@@ -1 +1 @@
-0123456789
\ No newline at end of file
+abbbbbbbbb
\ No newline at end of file
diff --git a/tests/gold_tests/h2/http2.test.py
b/tests/gold_tests/h2/http2.test.py
index cd9d9de..7065c05 100644
--- a/tests/gold_tests/h2/http2.test.py
+++ b/tests/gold_tests/h2/http2.test.py
@@ -112,14 +112,15 @@ tr.Processes.Default.ReturnCode = 0
tr.Processes.Default.Streams.stdout = "gold/chunked.gold"
tr.StillRunningAfter = server
+# NOTE: Skipping this test run because traffic-replay doesn't currently
support H2
# Test Case 4: Multiple request
-client_path = os.path.join(Test.Variables.AtsTestToolsDir, 'traffic-replay/')
-tr = Test.AddTestRun()
-tr.Processes.Default.Command = "python3 {0} -type {1} -log_dir {2} -port {3}
-host '127.0.0.1' -s_port {4} -v -colorize False".format(
- client_path, 'h2', server.Variables.DataDir, ts.Variables.port,
ts.Variables.ssl_port)
-tr.Processes.Default.ReturnCode = 0
-tr.Processes.Default.Streams.stdout = "gold/replay.gold"
-tr.StillRunningAfter = server
+# client_path = os.path.join(Test.Variables.AtsTestToolsDir, 'traffic-replay/')
+# tr = Test.AddTestRun()
+# tr.Processes.Default.Command = "python3 {0} -type {1} -log_dir {2} -port {3}
-host '127.0.0.1' -s_port {4} -v -colorize False".format(
+# client_path, 'h2', server.Variables.DataDir, ts.Variables.port,
ts.Variables.ssl_port)
+# tr.Processes.Default.ReturnCode = 0
+# tr.Processes.Default.Streams.stdout = "gold/replay.gold"
+# tr.StillRunningAfter = server
# Test Case 5:h2_active_timeout
tr = Test.AddTestRun()
diff --git a/tests/gold_tests/remap/remap_https.test.py
b/tests/gold_tests/remap/remap_https.test.py
index 661863d..efaa444 100644
--- a/tests/gold_tests/remap/remap_https.test.py
+++ b/tests/gold_tests/remap/remap_https.test.py
@@ -61,7 +61,7 @@ ts.Disk.remap_config.AddLine(
'map https://www.example.com:{1}
http://127.0.0.1:{0}'.format(server.Variables.Port, ts.Variables.ssl_port)
)
ts.Disk.remap_config.AddLine(
- 'map https://www.anotherexample.com
https://127.0.0.1:{0}'.format(server2.Variables.Port, ts.Variables.ssl_port)
+ 'map https://www.anotherexample.com
https://127.0.0.1:{0}'.format(server2.Variables.SSL_Port,ts.Variables.ssl_port)
)
diff --git a/tests/tools/lib/IPConstants.py b/tests/tools/lib/IPConstants.py
deleted file mode 100644
index 0531137..0000000
--- a/tests/tools/lib/IPConstants.py
+++ /dev/null
@@ -1,48 +0,0 @@
-'''
-'''
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import ipaddress
-
-# convenience functions
-
-IPkw = {'INADDR_LOOPBACK':'127.0.0.1',
- 'IN6ADDR_LOOPBACK':'::1',
- 'INADDR_ANY':'0.0.0.0',
- 'IN6ADDR_ANY':'::'}
-
-def isIPv6(addr):
- if addr in IPkw:
- addr = IPkw[addr]
-
-
- return ipaddress.ip_address(addr).version == 6
-
-
-def isIPv4(addr):
- if addr in IPkw:
- addr = IPkw[addr]
-
- return ipaddress.ip_address(addr).version == 4
-
-
-def getIP(addr):
- if addr in IPkw:
- addr = IPkw[addr]
-
- return str(ipaddress.ip_address(addr))
-
diff --git a/tests/tools/lib/result.py b/tests/tools/lib/result.py
deleted file mode 100644
index 7322e65..0000000
--- a/tests/tools/lib/result.py
+++ /dev/null
@@ -1,117 +0,0 @@
-#!/bin/env python3
-'''
-'''
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import sys
-
-
-class TermColors:
- ''' Collection of colors for printing out to terminal '''
- HEADER = '\033[95m'
- OKBLUE = '\033[94m'
- OKGREEN = '\033[92m'
- WARNING = '\033[93m'
- FAIL = '\033[91m'
- BOLD = '\033[1m'
- UNDERLINE = '\033[4m'
- ENDC = '\033[0m'
-
-
-ignoredFields = {'age', 'set-cookie', 'server', 'date', 'last-modified',
- 'via', 'expires', 'cache-control', 'vary', 'connection'} #
all lower case
-
-
-class Result(object):
- ''' Result encapsulates the result of a single session replay '''
-
- def __init__(self, test_name, expected_response, received_response,
recv_resp_body=None):
- ''' expected_response and received_response can be any datatype the
caller wants as long as they are the same datatype '''
- self._test_name = test_name
- self._expected_response = expected_response
- self._received_response = received_response
- self._received_response_body = recv_resp_body
-
- def getTestName(self):
- return self._test_name
-
- def getResultBool(self):
- return self._expected_response == self._received_response
-
- def getRespBody(self):
- if self._received_response_body:
- return self._received_response_body
- else:
- return ""
-
- def Compare(self, received_dict, expected_dict, src=None):
- global ignoredFields
- # print("RECEIVED")
- # print(received_dict)
- # print("RECIEVED CACHE CONTROL")
- # print(received_dict['Cache-Control'.lower()])
- # print("EXPECTED")
- # print(expected_dict)
- try:
- for key in received_dict:
- # print(key)
- if key.lower() in expected_dict and key.lower() not in
ignoredFields:
- # print("{0} ==?
{1}".format(expected_dict[key.lower()],received_dict[key]))
- if received_dict[key.lower()] !=
expected_dict[key.lower()]:
- print("{0}Difference in the field \"{1}\": \n
received:\n{2}\n expected:\n{3}{4}".format(
- TermColors.FAIL, key, received_dict[key],
expected_dict[key], TermColors.ENDC))
- return False
- if key.lower() == 'content-length' and
self._received_response_body:
- if int(received_dict[key.lower()]) !=
len(self._received_response_body):
- print("{0}Difference in received content length
and actual body length \ncontent-length: {1} \nbody: {2}\nbody length:
{3}{4}".format(
- TermColors.FAIL, received_dict[key.lower()],
self._received_response_body, len(
- self._received_response_body,
TermColors.ENDC)
- ))
- return False
-
- except:
- e = sys.exc_info()
- if src:
- print("In {0}: ".format(src), end='')
- print("Error in comparing key ", e, key, "expected",
expected_dict[key.lower()], "received", received_dict[key])
- return False
- return True
-
- def getResult(self, received_dict, expected_dict, colorize=False):
- global ignoredFields
- retval = False
- ''' Return a nicely formatted result string with color if requested '''
- if self.getResultBool() and self.Compare(received_dict, expected_dict,
self._test_name):
- if colorize:
- outstr = "{0}PASS{1}".format(
- TermColors.OKGREEN, TermColors.ENDC)
-
- else:
- outstr = "PASS"
-
- retval = True
-
- else:
- if colorize:
- outstr = "{0}FAIL{1}: expected {2}, received {3}, session
file: {4}".format(
- TermColors.FAIL, TermColors.ENDC, self._expected_response,
self._received_response, self._test_name)
-
- else:
- outstr = "FAIL: expected {0}, received {1}".format(
- self._expected_response, self._received_response)
-
- return (retval, outstr)
diff --git a/tests/tools/microDNS/uDNS.py b/tests/tools/microDNS/uDNS.py
deleted file mode 100644
index 297cdcb..0000000
--- a/tests/tools/microDNS/uDNS.py
+++ /dev/null
@@ -1,207 +0,0 @@
-# coding=utf-8
-
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import datetime
-import sys
-import time
-import threading
-import traceback
-import socketserver
-import argparse
-import codecs
-import json
-from dnslib import *
-
-sys.path.append(
- os.path.normpath(
- os.path.join(
- os.path.dirname(os.path.abspath(__file__)),
- '..'
- )
- )
-)
-
-import lib.IPConstants as IPConstants
-
-TTL = 60 * 5 # completely arbitrary TTL value
-round_robin = False
-default_records = list()
-records = dict()
-
-class DomainName(str):
- def __getattr__(self, item):
- return DomainName(item + '.' + self)
-
-
-class BaseRequestHandler(socketserver.BaseRequestHandler):
-
- def get_data(self):
- raise NotImplementedError
-
- def send_data(self, data):
- raise NotImplementedError
-
- def handle(self):
- now = datetime.datetime.utcnow().strftime('%Y-%m-%d %H:%M:%S.%f')
- print("\n\n%s request %s (%s %s):" % (self.__class__.__name__[:3],
now, self.client_address[0],
- self.client_address[1]))
- try:
- data = self.get_data()
- self.send_data(dns_response(data))
- except Exception:
- traceback.print_exc(file=sys.stderr)
-
-
-class TCPRequestHandler(BaseRequestHandler):
-
- def get_data(self):
- data = self.request.recv(8192).strip()
- sz = int(codecs.encode(data[:2], 'hex'), 16)
- if sz < len(data) - 2:
- raise Exception("Wrong size of TCP packet")
- elif sz > len(data) - 2:
- raise Exception("Too big TCP packet")
- return data[2:]
-
- def send_data(self, data):
- sz = codecs.decode(hex(len(data))[2:].zfill(4), 'hex')
- return self.request.sendall(sz + data)
-
-
-class UDPRequestHandler(BaseRequestHandler):
-
- def get_data(self):
- return self.request[0].strip()
-
- def send_data(self, data):
- return self.request[1].sendto(data, self.client_address)
-
-
-def build_domain_mappings(path):
- with open(path) as f:
- zone_file = json.load(f)
-
- for domain in zone_file['mappings']:
- for d in iter(domain.keys()):
- # this loop only runs once, kind of a hack to access the only key
in the dict
- domain_name = DomainName(d)
- print("Domain name:", domain_name)
- # we can test using python's built-in ipaddress module, but this
should suffice
- records[domain_name] = [A(x) if ":" not in x else AAAA(x) for x in
domain[domain_name]]
- print(records[domain_name])
-
- if 'otherwise' in zone_file:
- default_records.extend([A(d) if ":" not in d else AAAA(d) for d in
zone_file['otherwise']])
-
-
-def add_authoritative_records(reply, domain):
- # ns1 and ns1 are hardcoded in, change if necessary
- reply.add_auth(RR(rname=domain, rtype=QTYPE.NS, rclass=1, ttl=TTL,
rdata=NS(domain.ns1)))
- reply.add_auth(RR(rname=domain, rtype=QTYPE.NS, rclass=1, ttl=TTL,
rdata=NS(domain.ns2)))
-
-
-def dns_response(data):
- ''' dns_response takes in the raw bytes from the socket and does all the
logic behind what
- RRs get returned as the response '''
- global default_records, records, TTL, round_robin
-
- request = DNSRecord.parse(data)
- print(request)
-
- reply = DNSRecord(DNSHeader(id=request.header.id, qr=1, aa=1, ra=1),
q=request.q)
- qname = request.q.qname
- qn = str(qname)
- qtype = request.q.qtype
- qt = QTYPE[qtype]
- found_specific = False
-
- # first look for a specific mapping
- for domain, rrs in records.items():
- if domain == qn or qn.endswith('.' + domain):
- # we are the authoritative name server for this domain and all
subdomains
- for rdata in rrs:
- # only include requested record types (ie. A, MX, etc)
- rqt = rdata.__class__.__name__
- if qt in ['*', rqt]:
- found_specific = True
- reply.add_answer(RR(rname=qname, rtype=getattr(QTYPE,
str(rqt)), rclass=1, ttl=TTL, rdata=rdata))
-
- # rotate the A entries if round robin is on
- if round_robin:
- a_records = [x for x in rrs if type(x) == A]
- records[domain] = a_records[1:] + a_records[:1] # rotate list
- break
-
- # else if a specific mapping is not found, return default A-records
- if not found_specific:
- for a in default_records:
- found_specific = True
- reply.add_answer(RR(rname=qname, rtype=QTYPE.A, rclass=1, ttl=TTL,
rdata=a))
-
- if round_robin:
- default_records = default_records[1:] + default_records[:1]
-
- if not found_specific:
- reply.header.set_rcode(3)
-
- print("---- Reply: ----\n", reply)
- return reply.pack()
-
-
-if __name__ == '__main__':
- # handle cmd line args
- parser = argparse.ArgumentParser()
- parser.add_argument("ip", type=str, help="Interface")
- parser.add_argument("port", type=int, help="port uDNS should listen on")
- parser.add_argument("zone_file", help="path to zone file")
- parser.add_argument("--rr", action='store_true',
- help='round robin load balances if multiple IP
addresses are present for 1 domain')
- args = parser.parse_args()
-
- if IPConstants.isIPv6(args.ip):
- # *UDPServer derives from TCPServer, so setting one will affect the
other
- socketserver.TCPServer.address_family = socket.AF_INET6
-
- # exit(1)
-
- if args.rr:
- round_robin = True
- build_domain_mappings(args.zone_file)
-
- ipaddr = IPConstants.getIP(args.ip)
-
- servers = [
- socketserver.ThreadingUDPServer((ipaddr, args.port),
UDPRequestHandler),
- socketserver.ThreadingTCPServer((ipaddr, args.port),
TCPRequestHandler),
- ]
-
- print("Starting DNS on address {0} port {1}...".format(ipaddr, args.port))
- for s in servers:
- thread = threading.Thread(target=s.serve_forever) # that thread will
start one more thread for each request
- thread.daemon = True # exit the server thread when the main thread
terminates
- thread.start()
-
- try:
- while 1:
- time.sleep(1)
- sys.stderr.flush()
- sys.stdout.flush()
-
- except KeyboardInterrupt:
- print("Got SigINT")
- # pass
- finally:
- for s in servers:
- s.shutdown()
diff --git a/tests/tools/microServer/README.md
b/tests/tools/microServer/README.md
deleted file mode 100644
index a7681a3..0000000
--- a/tests/tools/microServer/README.md
+++ /dev/null
@@ -1,49 +0,0 @@
-uWServer
-========
-
-uWServer is a mock HTTP server that takes predefined set of sessions for
serving response to HTTP requests. Each session includes one or more
transactions. A transaction is composed of an HTTP request and an HTTP response.
-uWServer accepts session data in JSON fromat only.
-
-
-Command:
-----------------
-
-`python3.5 uWServer.py --data-dir <PATH_TO_SESSION_DIR>`
-
-Options:
------------
-
-To see the options please run `python3.5 uWServer.py --help`
-
-Session Definitions:
---------------------
-
-Example session:
-
-```
-{
- "encoding": "url_encoded",
- "version": "0.2",
- "txns": [
- {
- "response": {
- "headers": "HTTP/1.1 200 OK\r\nConnection: close\r\n\r\n",
- "body": "",
- "timestamp": "1469733493.993"
- },
- "request": {
- "headers": "GET / HTTP/1.1\r\nHost: www.example.test\r\n\r\n",
- "body": "",
- "timestamp": "1469733493.993"
- },
- "uuid": "",
- "timestamp": ""
- }
- ],
- "timestamp": "1234567890.098"
-}
-```
-
-Each session should be in its own file, and any number of files may be created
to define sessions.
-
-The `response` map may include an `options` string, which is a comma-delimited
list of options to be enabled. Currently the only option supported is
`skipHooks`, which will ignore any hooks created for the matching
request/response pair. See **Options**.
diff --git a/tests/tools/microServer/uWServer.py
b/tests/tools/microServer/uWServer.py
deleted file mode 100644
index 8f6354a..0000000
--- a/tests/tools/microServer/uWServer.py
+++ /dev/null
@@ -1,716 +0,0 @@
-#!/bin/env python3
-'''
-'''
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import string
-import http.client
-import cgi
-import time
-import sys
-import json
-import os
-import threading
-from ipaddress import ip_address
-from http.server import BaseHTTPRequestHandler, HTTPServer
-from socketserver import ThreadingMixIn, ForkingMixIn, BaseServer
-from http import HTTPStatus
-import argparse
-import ssl
-import socket
-import importlib.util
-import time
-test_mode_enabled = True
-lookup_key_ = "{PATH}"
-__version__ = "1.1"
-
-
-sys.path.append(
- os.path.normpath(
- os.path.join(
- os.path.dirname(os.path.abspath(__file__)),
- '..'
- )
- )
-)
-
-import sessionvalidation.sessionvalidation as sv
-import lib.IPConstants as IPConstants
-
-
-SERVER_PORT = 5005 # default port
-SERVER_DELAY = 0 # default delay
-HTTP_VERSION = 'HTTP/1.1'
-G_replay_dict = {}
-
-count = 0
-
-# Simple class to hold lists of callbacks associated with a key.
-
-
-class HookSet:
- # Helper class to provide controlled access to the HookSet to the loading
module.
- class Registrar:
- def __init__(self, hook_set):
- self.hooks = hook_set
-
- def register(self, hook, cb):
- self.hooks.register(hook, cb)
-
- def __init__(self):
- self.hooks = {}
- self.modules = []
- self.registrar = HookSet.Registrar(self)
- # Define all the valid hooks here.
- for item in ['ReadRequestHook']:
- if isinstance(item, list):
- hook = item[0]
- label = item[1]
- else:
- hook = label = item
- exec("HookSet.{} = '{}'".format(label, hook))
- exec("HookSet.Registrar.{} = '{}'".format(label, hook))
- self.hooks[hook] = []
-
- def load(self, source):
- try:
- spec = importlib.util.spec_from_file_location('Observer', source)
- mod = importlib.util.module_from_spec(spec)
- mod.Hooks = self.registrar
- spec.loader.exec_module(mod)
- except ImportError:
- print("Failed to import {}".format(source))
- else:
- self.modules.append(mod)
-
- # Add a callback cb to the hook.
- # Error if the hook isn't defined.
- def register(self, hook, cb):
- if hook in self.hooks:
- self.hooks[hook].append(cb)
- else:
- raise ValueError("{} is not a valid hook name".format(hook))
-
- # Invoke a hook. Pass on any additional arguments to the callback.
- def invoke(self, hook, *args, **kwargs):
- cb_list = self.hooks[hook]
- if cb_list == None:
- raise ValueError("{} is not a valid hook name to
invoke".format(hook))
- else:
- for cb in cb_list:
- cb(*args, **kwargs)
-
-
-class ThreadingServer(ThreadingMixIn, HTTPServer):
- '''This class forces the creation of a new thread on each connection'''
-
- def __init__(self, local_addr, handler_class, options):
- HTTPServer.__init__(self, local_addr, handler_class)
- self.hook_set = HookSet()
- if (options.load):
- self.hook_set.load(options.load)
-
-
-class ForkingServer(ForkingMixIn, HTTPServer):
- '''This class forces the creation of a new process on each connection'''
- pass
-
-
-class SSLServer(ThreadingMixIn, HTTPServer):
- def __init__(self, server_address, HandlerClass, options):
- BaseServer.__init__(self, server_address, HandlerClass)
- pwd = os.path.dirname(os.path.realpath(__file__))
- keys = os.path.join(pwd, options.key)
- certs = os.path.join(pwd, options.cert)
- self.options = options
- self.hook_set = HookSet()
-
- self.daemon_threads = True
- self.protocol_version = 'HTTP/1.1'
-
- if options.load:
- self.hook_set.load(options.load)
-
- if options.clientverify:
- self.socket = ssl.wrap_socket(socket.socket(self.address_family,
self.socket_type),
- keyfile=keys, certfile=certs,
server_side=True, cert_reqs=ssl.CERT_REQUIRED,
ca_certs='/etc/ssl/certs/ca-certificates.crt')
- else:
- self.socket = ssl.wrap_socket(socket.socket(self.address_family,
self.socket_type),
- keyfile=keys, certfile=certs,
server_side=True)
-
- self.server_bind()
- self.server_activate()
- print("Port Configured for SSL communication")
-
-
-class MyHandler(BaseHTTPRequestHandler):
- def handleExpect100Continue(self, contentLength, chunked=False):
- print("....expect", contentLength)
- self.wfile.write(bytes('HTTP/1.1 100 Continue\r\n\r\n', 'UTF-8'))
- if(not chunked):
- message = self.rfile.read(contentLength)
- else:
- readChunks()
-
- def getLookupKey(self, requestline):
- global lookup_key_
- kpath = ""
- path = ""
- url_part = requestline.split(" ")
- if url_part:
- if url_part[1].startswith("http"):
- path = url_part[1].split("/", 2)[2]
- host_, path = path.split("/", 1)
- else:
- path = url_part[1].split("/", 1)[1]
- argsList = []
- keyslist = lookup_key_.split("}")
- for keystr in keyslist:
- if keystr == '{PATH':
- kpath = kpath + path
- continue # do not include path in the list of header fields
- if keystr == '{HOST':
- kpath = kpath + host_
- continue
- stringk = keystr.replace("{%", "")
- argsList.append(stringk)
- KeyList = []
- for argsL in argsList:
- print("args", argsL, len(argsL))
- if len(argsL) > 0:
- val = self.headers.get(argsL)
- if val:
- field_val, __ = cgi.parse_header(val)
- else:
- field_val = None
- if field_val != None:
- KeyList.append(field_val)
- key = "".join(KeyList) + kpath
- print("lookup key", key, len(key))
-
- return key
-
- def parseRequestline(self, requestline):
- testName = None
- return testName
-
- def testMode(self, requestline):
- print(requestline)
- key = self.parseRequestline(requestline)
-
- self.send_response(200)
- self.send_header('Connection', 'close')
- self.end_headers()
-
- def get_response_code(self, header):
- # this could totally go wrong
- return int(header.split(' ')[1])
-
- def generator(self):
- yield 'micro'
- yield 'server'
- yield 'apache'
- yield 'traffic'
- yield 'server'
-
- def send_response(self, code, message=None):
- ''' Override `send_response()`'s tacking on of server and date header
lines. '''
- self.send_response_only(code, message)
-
- def createDummyBodywithLength(self, numberOfbytes):
- if numberOfbytes == 0:
- return None
- body = 'a'
- while numberOfbytes != 1:
- body += 'b'
- numberOfbytes -= 1
- return body
-
- def writeChunkedData(self):
- for chunk in self.generator():
- response_string = bytes('%X\r\n%s\r\n' % (len(chunk), chunk),
'UTF-8')
- self.wfile.write(response_string)
- response_string = bytes('0\r\n\r\n', 'UTF-8')
- self.wfile.write(response_string)
-
- def readChunks(self):
- raw_data = b''
- raw_size = self.rfile.readline(65537)
- size = str(raw_size, 'UTF-8').rstrip('\r\n')
- # print("==========================================>",size)
- size = int(size, 16)
- while size > 0:
- chunk = self.rfile.read(size + 2) # 2 for reading /r/n
- raw_data += chunk
- raw_size = self.rfile.readline(65537)
- size = str(raw_size, 'UTF-8').rstrip('\r\n')
- size = int(size, 16)
- chunk = self.rfile.readline(65537) # read the extra blank newline
\r\n after the last chunk
-
- def send_header(self, keyword, value):
- """Send a MIME header to the headers buffer."""
- if self.request_version != 'HTTP/0.9':
- if not hasattr(self, '_headers_buffer'):
- self._headers_buffer = []
- self._headers_buffer.append(
- ("%s: %s\r\n" % (keyword, value)).encode('UTF-8', 'strict'))
# original code used latin-1.. seriously?
-
- if keyword.lower() == 'connection':
- if value.lower() == 'close':
- self.close_connection = True
- elif value.lower() == 'keep-alive':
- self.close_connection = False
-
- def parse_request(self):
- """Parse a request (internal).
-
- The request should be stored in self.raw_requestline; the results
- are in self.command, self.path, self.request_version and
- self.headers. Any matching response is in self.response.
-
- Return True for success, False for failure; on failure, an
- error is sent back.
-
- """
-
- global count, test_mode_enabled, G_replay_dict
-
- self.command = None # set in case of error on the first line
- self.request_version = version = self.default_request_version
- self.close_connection = True
- requestline = str(self.raw_requestline, 'UTF-8')
- requestline = requestline.rstrip('\r\n')
- self.requestline = requestline
-
- # Examine the headers and look for a Connection directive.
- try:
- self.headers = http.client.parse_headers(self.rfile,
- _class=self.MessageClass)
- key = self.getLookupKey(self.requestline)
- self.resp = G_replay_dict[key] if key in G_replay_dict else None
-
- if self.resp is None or 'skipHooks' not in self.resp.getOptions():
- self.server.hook_set.invoke(HookSet.ReadRequestHook,
self.headers)
- # read message body
- if self.headers.get('Content-Length') != None:
- bodysize = int(self.headers.get('Content-Length'))
- #print("length of the body is",bodysize)
- message = self.rfile.read(bodysize)
- #print("message body",message)
- elif self.headers.get('Transfer-Encoding', "") == 'chunked':
- # print(self.headers)
- self.readChunks()
- except http.client.LineTooLong:
- self.send_error(
- HTTPStatus.BAD_REQUEST,
- "Line too long")
- return False
- except http.client.HTTPException as err:
- self.send_error(
- HTTPStatus.REQUEST_HEADER_FIELDS_TOO_LARGE,
- "Too many headers",
- str(err)
- )
- return False
-
- words = requestline.split()
- if len(words) == 3:
- command, path, version = words
- if version[:5] != 'HTTP/':
- self.send_error(
- HTTPStatus.BAD_REQUEST,
- "Bad request version (%r)" % version)
- return False
- try:
- base_version_number = version.split('/', 1)[1]
- version_number = base_version_number.split(".")
- # RFC 2145 section 3.1 says there can be only one "." and
- # - major and minor numbers MUST be treated as
- # separate integers;
- # - HTTP/2.4 is a lower version than HTTP/2.13, which in
- # turn is lower than HTTP/12.3;
- # - Leading zeros MUST be ignored by recipients.
- if len(version_number) != 2:
- raise ValueError
- version_number = int(version_number[0]), int(version_number[1])
- except (ValueError, IndexError):
- self.send_error(
- HTTPStatus.BAD_REQUEST,
- "Bad request version (%r)" % version)
- return False
- if version_number >= (1, 1) and self.protocol_version >=
"HTTP/1.1":
- self.close_connection = False
- if version_number >= (2, 0):
- self.send_error(
- HTTPStatus.HTTP_VERSION_NOT_SUPPORTED,
- "Invalid HTTP Version (%s)" % base_version_number)
- return False
- elif len(words) == 2:
- command, path = words
- self.close_connection = True
- if command != 'GET':
- self.send_error(
- HTTPStatus.BAD_REQUEST,
- "Bad HTTP/0.9 request type (%r)" % command)
- return False
- elif not words:
- count += 1
- print("bla bla on 157 {0} => {1}".format(count,
self.close_connection))
- return False
- else:
- self.send_error(
- HTTPStatus.BAD_REQUEST,
- "Bad request syntax (%r)" % requestline)
- return False
- self.command, self.path, self.request_version = command, path, version
-
- conntype = self.headers.get('Connection', "")
- if conntype.lower() == 'close':
- self.close_connection = True
- elif (conntype.lower() == 'keep-alive' and
- self.protocol_version >= "HTTP/1.1"):
- self.close_connection = False
-
- return True
-
- def do_GET(self):
- global G_replay_dict, test_mode_enabled
- if test_mode_enabled:
- time.sleep(time_delay)
-
- try:
- response_string = None
- chunkedResponse = False
- if self.resp is None:
- self.send_response(404)
- self.send_header('Server', 'MicroServer')
- self.send_header('Connection', 'close')
- self.end_headers()
- return
-
- else:
- headers = self.resp.getHeaders().split('\r\n')
-
- # set status codes
- status_code = self.get_response_code(headers[0])
- self.send_response(status_code)
-
- # set headers
- for header in headers[1:]: # skip first one b/c it's response
code
- if header == '':
- continue
- elif 'Content-Length' in header:
- if 'Access-Control' in header: # skipping
Access-Control-Allow-Credentials, Access-Control-Allow-Origin, Content-Length
- header_parts = header.split(':', 1)
- header_field = str(header_parts[0].strip())
- header_field_val = str(header_parts[1].strip())
- self.send_header(header_field, header_field_val)
- continue
- lengthSTR = header.split(':')[1]
- length = lengthSTR.strip(' ')
- if test_mode_enabled: # the length of the body is
given priority in test mode rather than the value in Content-Length. But in
replay mode Content-Length gets the priority
- if not (self.resp.getBody()): # Don't attach
content-length yet if body is present in the response specified by tester
- self.send_header('Content-Length', str(length))
- else:
- self.send_header('Content-Length', str(length))
- response_string =
self.createDummyBodywithLength(int(length))
- continue
- if 'Transfer-Encoding' in header:
- self.send_header('Transfer-Encoding', 'Chunked')
- response_string = '%X\r\n%s\r\n' % (len('ats'), 'ats')
- chunkedResponse = True
- continue
-
- header_parts = header.split(':', 1)
- header_field = str(header_parts[0].strip())
- header_field_val = str(header_parts[1].strip())
- self.send_header(header_field, header_field_val)
- # End for
- if test_mode_enabled:
- if self.resp.getBody():
- length = len(bytes(self.resp.getBody(), 'UTF-8'))
- response_string = self.resp.getBody()
- self.send_header('Content-Length', str(length))
- self.end_headers()
-
- if (chunkedResponse):
- self.writeChunkedData()
- elif response_string != None and response_string != '':
- self.wfile.write(bytes(response_string, 'UTF-8'))
- except:
- e = sys.exc_info()
- print("Error", e, self.headers)
- self.send_response(400)
- self.send_header('Connection', 'close')
- self.end_headers()
-
- def do_HEAD(self):
- if self.resp is None:
- self.send_response(404)
- self.send_header('Connection', 'close')
- self.end_headers()
- return
-
- headers = self.resp.getHeaders().split('\r\n')
-
- # set status codes
- status_code = self.get_response_code(headers[0])
- self.send_response(status_code)
-
- # set headers
- for header in headers[1:]: # skip first one b/c it's response code
- if header == '':
- continue
- elif 'Content-Length' in header:
- self.send_header('Content-Length', '0')
- continue
-
- header_parts = header.split(':', 1)
- header_field = str(header_parts[0].strip())
- header_field_val = str(header_parts[1].strip())
- self.send_header(header_field, header_field_val)
-
- self.end_headers()
-
- def do_POST(self):
- response_string = None
- chunkedResponse = False
- global test_mode_enabled
- try:
-
- if self.resp is None:
- self.send_response(404)
- self.send_header('Connection', 'close')
- self.end_headers()
- return
- else:
- resp_headers = self.resp.getHeaders().split('\r\n')
- # set status codes
- status_code = self.get_response_code(resp_headers[0])
- #print("response code",status_code)
- self.send_response(status_code)
- #print("reposen is ",resp_headers)
- # set headers
- for header in resp_headers[1:]: # skip first one b/c it's
response code
-
- if header == '':
- continue
- elif 'Content-Length' in header:
- if 'Access-Control' in header: # skipping
Access-Control-Allow-Credentials, Access-Control-Allow-Origin, Content-Length
- header_parts = header.split(':', 1)
- header_field = str(header_parts[0].strip())
- header_field_val = str(header_parts[1].strip())
- self.send_header(header_field, header_field_val)
- continue
-
- lengthSTR = header.split(':')[1]
- length = lengthSTR.strip(' ')
- if test_mode_enabled: # the length of the body is
given priority in test mode rather than the value in Content-Length. Otherwise,
Content-Length gets the priority
- if not (self.resp.getBody()): # Don't attach
content-length yet if body is present in the response specified by tester
- self.send_header('Content-Length', str(length))
- else:
- self.send_header('Content-Length', str(length))
- response_string =
self.createDummyBodywithLength(int(length))
- continue
- if 'Transfer-Encoding' in header:
- self.send_header('Transfer-Encoding', 'Chunked')
- response_string = '%X\r\n%s\r\n' %
(len('microserver'), 'microserver')
- chunkedResponse = True
- continue
-
- header_parts = header.split(':', 1)
- header_field = str(header_parts[0].strip())
- header_field_val = str(header_parts[1].strip())
- #print("{0} === >{1}".format(header_field,
header_field_val))
- self.send_header(header_field, header_field_val)
- # End for loop
- if test_mode_enabled:
- if self.resp.getBody():
- length = len(bytes(self.resp.getBody(), 'UTF-8'))
- response_string = self.resp.getBody()
- self.send_header('Content-Length', str(length))
- self.end_headers()
-
- if (chunkedResponse):
- self.writeChunkedData()
- elif response_string != None and response_string != '':
- self.wfile.write(bytes(response_string, 'UTF-8'))
- except:
- e = sys.exc_info()
- print("Error", e, self.headers)
- self.send_response(400)
- self.send_header('Connection', 'close')
- self.end_headers()
-
-
-def populate_global_replay_dictionary(sessions):
- ''' Populates the global dictionary of {uuid (string): reponse (Response
object)} '''
- global G_replay_dict
- for session in sessions:
- for txn in session.getTransactionIter():
- G_replay_dict[txn._uuid] = txn.getResponse()
-
- print("size", len(G_replay_dict))
-
-# tests will add responses to the dictionary where key is the testname
-
-
-def addResponseHeader(key, response_header):
- G_replay_dict[key] = response_header
-
-
-def _path(exists, arg):
- path = os.path.abspath(arg)
- if not os.path.exists(path) and exists:
- msg = '"{0}" is not a valid path'.format(path)
- raise argparse.ArgumentTypeError(msg)
- return path
-
-
-def _bool(arg):
-
- opt_true_values = set(['y', 'yes', 'true', 't', '1', 'on', 'all'])
- opt_false_values = set(['n', 'no', 'false', 'f', '0', 'off', 'none', None])
-
- tmp = arg.lower() if arg is not None else None
- if tmp in opt_true_values:
- return True
- elif tmp in opt_false_values:
- return False
- else:
- msg = 'Invalid value Boolean value : "{0}"\n Valid options are
{1}'.format(arg,
-
opt_true_values | opt_false_values)
- raise ValueError(msg)
-
-
-def _argparse_bool(arg):
- try:
- _bool(arg)
- except ValueError as ve:
- raise argparse.ArgumentTypeError(ve)
-
-
-def main():
- global test_mode_enabled
- parser = argparse.ArgumentParser()
-
- parser.add_argument("--data-dir", "-d",
- type=lambda x: _path(True, x),
- required=True,
- help="Directory with data file"
- )
-
- parser.add_argument("--ip_address", "-ip",
- type=str,
- default='INADDR_LOOPBACK',
- help="IP address of the interface to serve on"
- )
-
- parser.add_argument("--port", "-p",
- type=int,
- default=SERVER_PORT,
- help="Port to use")
-
- parser.add_argument("--delay", "-dy",
- type=float,
- default=SERVER_DELAY,
- help="Response delay")
-
- parser.add_argument("--timeout", "-t",
- type=float,
- default=None,
- help="socket time out in seconds")
-
- parser.add_argument('-V', '--version', action='version', version='%(prog)s
{0}'.format(__version__))
-
- parser.add_argument("--mode", "-m",
- type=str,
- default="test",
- help="Mode of operation")
- parser.add_argument("--ssl", "-ssl",
- type=str,
- default="False",
- help="SSL port")
- parser.add_argument("--key", "-k",
- type=str,
- default="ssl/server.pem",
- help="key for ssl connnection")
- parser.add_argument("--cert", "-cert",
- type=str,
- default="ssl/server.crt",
- help="certificate")
- parser.add_argument("--clientverify", "-cverify",
- type=_argparse_bool,
- default=False,
- help="verify client cert")
- parser.add_argument("--load",
- dest='load',
- type=str,
- default='',
- help="A file which will install observers on hooks")
- parser.add_argument("--lookupkey",
- type=str,
- default="{PATH}",
- help="format string used as a key for response lookup:
\
- example: \"{%%Host}{%%Server}{PATH}\",
\"{HOST}{PATH}\", \"{PATH}\"\
- All the args preceded by %% are header fields in the
request\
- The only two acceptable arguments which are not header
fields are : fqdn (represented by HOST) and the url path (represented by PATH)
in a request line.\
- Example: given a client request as << GET
/some/resource/location HTTP/1.1\nHost: hahaha.com\n\n >>, if the user wishes
the host field and the path to be used for the response lookup\
- then the required format will be {%%Host}{PATH}")
-
- args = parser.parse_args()
- options = args
- global time_delay
- time_delay = options.delay
-
- # set up global dictionary of {uuid (string): response (Response object)}
- s = sv.SessionValidator(args.data_dir)
- populate_global_replay_dictionary(s.getSessionIter())
- print("Dropped {0} sessions for being
malformed".format(len(s.getBadSessionList())))
-
- # start server
- try:
- socket_timeout = args.timeout
- test_mode_enabled = args.mode == "test"
- global lookup_key_
- lookup_key_ = args.lookupkey
- MyHandler.protocol_version = HTTP_VERSION
-
- if IPConstants.isIPv6(options.ip_address):
- print("Server running on IPv6")
- HTTPServer.address_family = socket.AF_INET6
-
- if options.ssl == "True" or options.ssl == "true":
- server = SSLServer((IPConstants.getIP(options.ip_address),
options.port), MyHandler, options)
- else:
- server = ThreadingServer((IPConstants.getIP(options.ip_address),
options.port), MyHandler, options)
-
- server.timeout = 5
- print("Started server on port {0}".format(options.port))
- server_thread = threading.Thread(target=server.serve_forever())
- server_thread.daemon = True
- server_thread.start()
-
- except KeyboardInterrupt:
- print("\n=== ^C received, shutting down httpserver ===")
- server.socket.close()
- # s_server.socket.close()
- sys.exit(0)
-
-
-if __name__ == '__main__':
- main()
diff --git a/tests/tools/microServer/ssl/server.crt
b/tests/tools/microserver/ssl/server.crt
similarity index 100%
rename from tests/tools/microServer/ssl/server.crt
rename to tests/tools/microserver/ssl/server.crt
diff --git a/tests/tools/microServer/ssl/server.pem
b/tests/tools/microserver/ssl/server.pem
similarity index 100%
rename from tests/tools/microServer/ssl/server.pem
rename to tests/tools/microserver/ssl/server.pem
diff --git a/tests/tools/sessionvalidation/__init__.py
b/tests/tools/sessionvalidation/__init__.py
deleted file mode 100644
index bcbf685..0000000
--- a/tests/tools/sessionvalidation/__init__.py
+++ /dev/null
@@ -1,17 +0,0 @@
-'''
-'''
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
diff --git a/tests/tools/sessionvalidation/badsession.py
b/tests/tools/sessionvalidation/badsession.py
deleted file mode 100644
index 7f55de2..0000000
--- a/tests/tools/sessionvalidation/badsession.py
+++ /dev/null
@@ -1,35 +0,0 @@
-'''
-'''
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-
-class BadSession(object):
- '''
- Session encapsulates a single BAD user session. Bad meaning that for some
reason the session is invalid.
-
- _filename is the filename of the bad JSON session
- _reason is a string with some kind of explanation on why the session was
bad
- '''
-
- def __repr__(self):
- return "<Session {{'filename': {0}, 'reason': {1}>".format(
- self._filename, self._reason
- )
-
- def __init__(self, filename, reason):
- self._filename = filename
- self._reason = reason
diff --git a/tests/tools/sessionvalidation/request.py
b/tests/tools/sessionvalidation/request.py
deleted file mode 100644
index 39598d7..0000000
--- a/tests/tools/sessionvalidation/request.py
+++ /dev/null
@@ -1,48 +0,0 @@
-'''
-'''
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-import hashlib
-
-
-class Request(object):
- ''' Request encapsulates a single request from the UA '''
-
- def getTimestamp(self):
- return self._timestamp
-
- def getHeaders(self):
- return self._headers
-
- def getBody(self):
- return self._body
-
- def getHeaderMD5(self):
- ''' Returns the MD5 hash of the headers
-
- This is used to do a unique mapping to a request/response transaction
'''
- return hashlib.md5(self._headers.encode()).hexdigest()
-
- def __repr__(self):
- # return str(self._timestamp)
- return "<Request: {{'timestamp': {0}, 'headers': {1}, 'body':
{2}}}>".format(
- str(self._timestamp), str(self._headers), str(self._body)
- )
-
- def __init__(self, timestamp, headers, body):
- self._timestamp = timestamp
- self._headers = headers
- self._body = body
diff --git a/tests/tools/sessionvalidation/response.py
b/tests/tools/sessionvalidation/response.py
deleted file mode 100644
index faa5f97..0000000
--- a/tests/tools/sessionvalidation/response.py
+++ /dev/null
@@ -1,49 +0,0 @@
-'''
-'''
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import re
-
-
-class Response(object):
- ''' Response encapsulates a single request from the UA '''
-
- def getTimestamp(self):
- return self._timestamp
-
- def getHeaders(self):
- return self._headers
-
- def getBody(self):
- return self._body
-
- def getOptions(self):
- return self._options
-
- def __repr__(self):
- return "<Response: {{'timestamp': {0}, 'headers': {1}, 'body': {2},
'options': {3}}}>".format(
- self._timestamp, self._headers, self._body, self._options
- )
-
- def __init__(self, timestamp, headers, body, options_string):
- self._timestamp = timestamp
- self._headers = headers
- self._body = body
- if options_string:
- self._options = re.compile(r'\s*,\s*').split(options_string)
- else:
- self._options = list()
diff --git a/tests/tools/sessionvalidation/session.py
b/tests/tools/sessionvalidation/session.py
deleted file mode 100644
index e8bb0e2..0000000
--- a/tests/tools/sessionvalidation/session.py
+++ /dev/null
@@ -1,45 +0,0 @@
-'''
-'''
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-import sessionvalidation.transaction as transaction
-
-
-class Session(object):
- ''' Session encapsulates a single user session '''
-
- def getTransactionList(self):
- ''' Returns a list of transaction objects '''
- return self._transaction_list
-
- def getTransactionIter(self):
- ''' Returns an iterator of transaction objects '''
- return iter(self._transaction_list)
-
- def returnFirstTransaction(self):
- return self._transaction_list[0]
-
- def __repr__(self):
- return "<Session {{'filename': {0}, 'version': {1}, 'timestamp: {2},
'encoding': {3}, 'transaction_list': {4}}}>".format(
- self._filename, self._version, self._timestamp, self._encoding,
repr(self._transaction_list)
- )
-
- def __init__(self, filename, version, timestamp, transaction_list,
encoding=None):
- self._filename = filename
- self._version = version
- self._timestamp = timestamp
- self._encoding = encoding
- self._transaction_list = transaction_list
diff --git a/tests/tools/sessionvalidation/sessionvalidation.py
b/tests/tools/sessionvalidation/sessionvalidation.py
deleted file mode 100644
index 7ff97e6..0000000
--- a/tests/tools/sessionvalidation/sessionvalidation.py
+++ /dev/null
@@ -1,259 +0,0 @@
-'''
-'''
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-import json
-import os
-
-import sessionvalidation.session as session
-import sessionvalidation.transaction as transaction
-import sessionvalidation.request as request
-import sessionvalidation.response as response
-
-# valid_HTTP_request_methods = ['GET', 'POST', 'HEAD']
-# custom_HTTP_request_methods = ['PULL'] # transaction monitor plugin for ATS
may have custom methods
-allowed_HTTP_request_methods = ['GET', 'POST', 'HEAD', 'PULL']
-G_CUSTOM_METHODS = False
-G_VERBOSE_LOG = True
-
-
-def _verbose_print(msg, verbose_on=False):
- ''' Print msg if verbose_on is set to True or G_VERBOSE_LOG is set to
True'''
- if verbose_on or G_VERBOSE_LOG:
- print(msg)
-
-
-class SessionValidator(object):
- '''
- SessionValidator parses, validates, and exports an API for a given set of
JSON sessions generated from Apache Traffic Server
-
- SessionValidator is initialized with a path to a directory of JSON
sessions. It then automatically parses and validates all the
- session in the directory. After initialization, the user may use the
provided API
-
- TODO :
- Provide a list of guaranteed fields for each type of object (ie a
Transaction has a request and a response, a request has ...)
- '''
-
- def parse(self):
- '''
- Constructs Session objects from JSON files on disk and stores objects
into _sessions
-
- All sessions missing required fields (ie. a session timestamp, a
response for every request, etc) are
- dropped and the filename is stored inside _bad_sessions
- '''
-
- log_filenames = [os.path.join(self._json_log_dir, f) for f in
os.listdir(
- self._json_log_dir) if
os.path.isfile(os.path.join(self._json_log_dir, f))]
-
- for fname in log_filenames:
- with open(fname) as f:
- # first attempt to load the JSON
- try:
- sesh = json.load(f)
- except:
- self._bad_sessions.append(fname)
- _verbose_print("Warning: JSON parse error on
file={0}".format(fname))
- print("Warning: JSON parse error on
file={0}".format(fname))
- continue
-
- # then attempt to extract all the required fields from the JSON
- try:
- session_timestamp = sesh['timestamp']
- session_version = sesh['version']
- session_txns = list()
- for txn in sesh['txns']:
- # create transaction Request object
- txn_request = txn['request']
-
- txn_request_body = ''
- if 'body' in txn_request:
- txn_request_body = txn_request['body']
- txn_request_obj =
request.Request(txn_request['timestamp'], txn_request['headers'],
txn_request_body)
- # Create transaction Response object
- txn_response = txn['response']
- txn_response_body = ''
- if 'body' in txn_response:
- txn_response_body = txn_response['body']
- txn_response_obj =
response.Response(txn_response['timestamp'], txn_response['headers'],
txn_response_body,
- txn_response.get('options'))
-
- # create Transaction object
- txn_obj = transaction.Transaction(txn_request_obj,
txn_response_obj, txn['uuid'])
- session_txns.append(txn_obj)
- session_obj = session.Session(fname, session_version,
session_timestamp, session_txns)
-
- except KeyError as e:
- self._bad_sessions.append(fname)
- print("Warning: parse error on key={0} for
file={1}".format(e, fname))
- _verbose_print("Warning: parse error on key={0} for
file={1}".format(e, fname))
- continue
-
- self._sessions.append(session_obj)
-
- def validate(self):
- ''' Prunes out all the invalid Sessions in _sessions '''
-
- good_sessions = list()
-
- for sesh in self._sessions:
- if SessionValidator.validateSingleSession(sesh):
- good_sessions.append(sesh)
- else:
- self._bad_sessions.append(sesh._filename)
-
- self._sessions = good_sessions
-
- @staticmethod
- def validateSingleSession(sesh):
- ''' Takes in a single Session object as input, returns whether or not
the Session is valid '''
-
- retval = True
-
- try:
- # first validate fields
- if not sesh._filename:
- _verbose_print("bad session filename")
- retval = False
- elif not sesh._version:
- _verbose_print("bad session version")
- retval = False
- elif float(sesh._timestamp) <= 0:
- _verbose_print("bad session timestamp")
- retval = False
- elif not bool(sesh.getTransactionList()):
- _verbose_print("session has no transaction list")
- retval = False
-
- # validate Transactions now
- for txn in sesh.getTransactionIter():
- if not SessionValidator.validateSingleTransaction(txn):
- retval = False
-
- except ValueError as e:
- _verbose_print("most likely an invalid session timestamp")
- retval = False
-
- return retval
-
- @staticmethod
- def validateSingleTransaction(txn):
- ''' Takes in a single Transaction object as input, and returns whether
or not the Transaction is valid '''
-
- txn_req = txn.getRequest()
- txn_resp = txn.getResponse()
- retval = True
-
- #valid_HTTP_request_methods = ['GET', 'HEAD', 'POST', 'PUT', 'DELETE',
'TRACE', 'OPTIONS', 'CONNECT', 'PATCH']
- # we can later uncomment the previous line to support more HTTP methods
- valid_HTTP_versions = ['HTTP/1.0', 'HTTP/1.1', 'HTTP/2.0']
-
- try:
- # validate request first
- if not txn_req:
- _verbose_print("no transaction request")
- retval = False
- elif txn_req.getBody() == None:
- _verbose_print("transaction body is set to None")
- retval = False
- elif float(txn_req.getTimestamp()) <= 0:
- _verbose_print("invalid transaction request timestamp")
- retval = False
- elif txn_req.getHeaders().split()[0] not in
allowed_HTTP_request_methods:
- _verbose_print("invalid HTTP method for transaction
{0}".format(txn_req.getHeaders().split()[0]))
- retval = False
- elif not txn_req.getHeaders().endswith("\r\n\r\n"):
- _verbose_print("transaction request headers didn't end with
\\r\\n\\r\\n")
- retval = False
- elif txn_req.getHeaders().split()[2] not in valid_HTTP_versions:
- _verbose_print("invalid HTTP version in request")
- retval = False
-
- # if the Host header is not present and vaild we reject this
transaction
- found_host = False
- for header in txn_req.getHeaders().split('\r\n'):
- split_header = header.split(' ')
- if split_header[0] == 'Host:':
- found_host = True
- host_header_no_space = len(split_header) == 1
- host_header_with_space = len(split_header) == 2 and
split_header[1] == ''
- if host_header_no_space or host_header_with_space:
- found_host = False
- if not found_host:
- print("missing host", txn_req)
- _verbose_print("transaction request Host header doesn't have
specified host")
- retval = False
-
- # now validate response
- if not txn_resp:
- _verbose_print("no transaction response")
- retval = False
- elif txn_resp.getBody() == None:
- _verbose_print("transaction response body set to None")
- retval = False
- elif float(txn_resp.getTimestamp()) <= 0:
- _verbose_print("invalid transaction response timestamp")
- retval = False
- elif txn_resp.getHeaders().split()[0] not in valid_HTTP_versions:
- _verbose_print("invalid HTTP response header")
- retval = False
- elif not txn_resp.getHeaders().endswith("\r\n\r\n"):
- _verbose_print("transaction response headers didn't end with
\\r\\n\\r\\n")
- retval = False
-
- # if any of the 3xx responses have bodies, then the must reject
this transaction, since 3xx
- # errors by definition can't have bodies
- response_line = txn_resp.getHeaders().split('\r\n')[0]
- response_code = response_line.split(' ')[1]
- if response_code.startswith('3') and txn_resp.getBody():
- _verbose_print("transaction response was 3xx and had a body")
- retval = False
-
- except ValueError as e:
- _verbose_print("most likely an invalid transaction timestamp")
- retval = False
-
- except IndexError as e:
- _verbose_print("most likely a bad transaction header")
- retval = False
-
- return retval
-
- def getSessionList(self):
- ''' Returns the list of Session objects '''
- return self._sessions
-
- def getSessionIter(self):
- ''' Returns an iterator of the Session objects '''
- return iter(self._sessions)
-
- def getBadSessionList(self):
- ''' Returns a list of bad session filenames (list of strings) '''
- return self._bad_sessions
-
- def getBadSessionListIter(self):
- ''' Returns an iterator of bad session filenames (iterator of strings)
'''
- return iter(self._bad_sessions)
-
- def __init__(self, json_log_dir, allow_custom=False):
- global valid_HTTP_request_methods
- global G_CUSTOM_METHODS
- G_CUSTOM_METHODS = allow_custom
- self._json_log_dir = json_log_dir
- self._bad_sessions = list() # list of filenames
- self._sessions = list() # list of _good_ session objects
-
- self.parse()
- self.validate()
diff --git a/tests/tools/sessionvalidation/transaction.py
b/tests/tools/sessionvalidation/transaction.py
deleted file mode 100644
index 19950ab..0000000
--- a/tests/tools/sessionvalidation/transaction.py
+++ /dev/null
@@ -1,40 +0,0 @@
-'''
-'''
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import sessionvalidation.request as request
-import sessionvalidation.response as response
-
-
-class Transaction(object):
- ''' Tranaction encapsulates a single UA transaction '''
-
- def getRequest(self):
- return self._request
-
- def getResponse(self):
- return self._response
-
- def __repr__(self):
- return "<Transaction {{'uuid': {0}, 'request': {1}, 'response':
{2}}}>".format(
- self._uuid, self._request, self._response
- )
-
- def __init__(self, request, response, uuid):
- self._request = request
- self._response = response
- self._uuid = uuid
diff --git a/tests/tools/traffic-replay/Config.py
b/tests/tools/traffic-replay/Config.py
deleted file mode 100644
index 48d3fc3..0000000
--- a/tests/tools/traffic-replay/Config.py
+++ /dev/null
@@ -1,34 +0,0 @@
-#!/bin/env python3
-'''
-'''
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# SSL config
-ca_certs = None
-keyfile = None
-
-# Proxy config
-proxy_host = "127.0.0.1"
-proxy_ssl_port = 443
-proxy_nonssl_port = 8080
-
-# process and thread config
-nProcess = 4
-nThread = 4
-
-# colorize output
-colorize = True
diff --git a/tests/tools/traffic-replay/NonSSL.py
b/tests/tools/traffic-replay/NonSSL.py
deleted file mode 100644
index 3d6b85b..0000000
--- a/tests/tools/traffic-replay/NonSSL.py
+++ /dev/null
@@ -1,192 +0,0 @@
-#!/bin/env python3
-'''
-'''
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import socket
-import requests
-import os
-from threading import Thread
-import sys
-from multiprocessing import current_process
-import sessionvalidation.sessionvalidation as sv
-import lib.result as result
-import extractHeader
-import mainProcess
-import json
-import gzip
-bSTOP = False
-
-
-def createDummyBodywithLength(numberOfbytes):
- if numberOfbytes <= 0:
- return None
- body = 'a'
- while numberOfbytes != 1:
- body += 'b'
- numberOfbytes -= 1
- return body
-
-
-def handleResponse(response, *args, **kwargs):
- print(response.status_code)
- # resp=args[0]
- #expected_output_split = resp.getHeaders().split('\r\n')[ 0].split(' ', 2)
- #expected_output = (int(expected_output_split[1]), str(
expected_output_split[2]))
- #r = result.Result(session_filename, expected_output[0],
response.status_code)
- # print(r.getResultString(colorize=True))
-# make sure len of the message body is greater than length
-
-
-def gen():
- yield 'pforpersia,champaignurbana'.encode('utf-8')
- yield 'there'.encode('utf-8')
-
-
-def txn_replay(session_filename, txn, proxy, result_queue, request_session):
- """ Replays a single transaction
- :param request_session: has to be a valid requests session"""
- req = txn.getRequest()
- resp = txn.getResponse()
-
- # Construct HTTP request & fire it off
- txn_req_headers = req.getHeaders()
- txn_req_headers_dict = extractHeader.header_to_dict(txn_req_headers)
- txn_req_headers_dict['Content-MD5'] = txn._uuid # used as unique
identifier
- if 'body' in txn_req_headers_dict:
- del txn_req_headers_dict['body']
-
- #print("Replaying session")
- try:
- # response =
request_session.request(extractHeader.extract_txn_req_method(txn_req_headers),
- # 'http://' +
extractHeader.extract_host(txn_req_headers) +
extractHeader.extract_GET_path(txn_req_headers),
- #
headers=txn_req_headers_dict,stream=False) # making stream=False raises
contentdecoding exception? kill me
- method = extractHeader.extract_txn_req_method(txn_req_headers)
- response = None
- body = None
- content = None
- if 'Transfer-Encoding' in txn_req_headers_dict:
- # deleting the host key, since the STUPID post/get functions are
going to add host field anyway, so there will be multiple host fields in the
header
- # This confuses the ATS and it returns 400 "Invalid HTTP request".
I don't believe this
- # BUT, this is not a problem if the data is not chunked encoded..
Strange, huh?
- del txn_req_headers_dict['Host']
- if 'Content-Length' in txn_req_headers_dict:
- #print("ewww !")
- del txn_req_headers_dict['Content-Length']
- body = gen()
- if 'Content-Length' in txn_req_headers_dict:
- nBytes = int(txn_req_headers_dict['Content-Length'])
- body = createDummyBodywithLength(nBytes)
- #print("request session is",id(request_session))
- if method == 'GET':
- r1 = request_session.request('GET',
'http://'+extractHeader.extract_host(txn_req_headers)+extractHeader.extract_GET_path(
- txn_req_headers), headers=txn_req_headers_dict, data=body)
- responseHeaders = r1.headers
- responseContent = r1.content # byte array
-
- #print("len: {0} received
{1}".format(responseHeaders['Content-Length'], responseContent))
-
- elif method == 'POST':
- r1 = request_session.request('POST',
'http://'+extractHeader.extract_host(txn_req_headers)+extractHeader.extract_GET_path(
- txn_req_headers), headers=txn_req_headers_dict, data=body)
- responseHeaders = r1.headers
- responseContent = r1.content
-
- #print("len: {0} received
{1}".format(responseHeaders['Content-Length'], responseContent))
- elif method == 'HEAD':
- r1 = request_session.request('HEAD',
'http://'+extractHeader.extract_host(txn_req_headers)+extractHeader.extract_GET_path(
- txn_req_headers), headers=txn_req_headers_dict, data=body)
- responseHeaders = r1.headers
- responseContent = r1.content
- else: # EXPERIMENTAL
- r1 = request_session.request(method,
'http://'+extractHeader.extract_host(txn_req_headers)+extractHeader.extract_GET_path(
- txn_req_headers), headers=txn_req_headers_dict, data=body)
- responseHeaders = r1.headers
- responseContent = r1.content
-
- #gzip_file = gzip.GzipFile(fileobj=responseContent)
- #shutil.copyfileobj(gzip_file, f)
-
- expected = extractHeader.responseHeader_to_dict(resp.getHeaders())
- # print("------------EXPECTED-----------")
- # print(expected)
- # print("------------RESP--------------")
- # print(responseHeaders)
- # print()
-
- if mainProcess.verbose:
- expected_output_split = resp.getHeaders().split('\r\n')[0].split('
', 2)
- expected_output = (int(expected_output_split[1]),
str(expected_output_split[2]))
- r = result.Result(session_filename, expected_output[0],
r1.status_code, responseContent)
- b_res, res = r.getResult(responseHeaders, expected, colorize=True)
- print(res)
-
- if not b_res:
- print("Received response")
- print(responseHeaders)
- print("Expected response")
- print(expected)
- # result_queue.put(r)
- except UnicodeEncodeError as e:
- # these unicode errors are due to the interaction between Requests and
our wiretrace data.
- # TODO fix
- print("UnicodeEncodeError exception")
-
- except requests.exceptions.ContentDecodingError as e:
- print("ContentDecodingError", e)
- except:
- e = sys.exc_info()
- print("ERROR in NonSSLReplay: ", e, response, session_filename)
-
-
-def session_replay(input, proxy, result_queue):
- global bSTOP
- ''' Replay all transactions in session
-
- This entire session will be replayed in one requests.Session (so one
socket / TCP connection)'''
- # if timing_control:
- # time.sleep(float(session._timestamp)) # allow other threads to run
- while bSTOP == False:
- for session in iter(input.get, 'STOP'):
- # print(bSTOP)
- if session == 'STOP':
- print("Queue is empty")
- bSTOP = True
- break
- with requests.Session() as request_session:
- request_session.proxies = proxy
- for txn in session.getTransactionIter():
- try:
- txn_replay(session._filename, txn, proxy,
result_queue, request_session)
- except:
- e = sys.exc_info()
- print("ERROR in replaying: ", e,
txn.getRequest().getHeaders())
- bSTOP = True
- #print("Queue is empty")
- input.put('STOP')
- break
-
-
-def client_replay(input, proxy, result_queue, nThread):
- Threads = []
- for i in range(nThread):
- t = Thread(target=session_replay, args=[input, proxy, result_queue])
- t.start()
- Threads.append(t)
-
- for t1 in Threads:
- t1.join()
diff --git a/tests/tools/traffic-replay/RandomReplay.py
b/tests/tools/traffic-replay/RandomReplay.py
deleted file mode 100644
index f6bf869..0000000
--- a/tests/tools/traffic-replay/RandomReplay.py
+++ /dev/null
@@ -1,91 +0,0 @@
-#!/bin/env python3
-'''
-'''
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import socket
-import requests
-import os
-from threading import Thread
-import sys
-from multiprocessing import current_process
-import sessionvalidation.sessionvalidation as sv
-from collections import deque
-import collections
-import lib.result as result
-import extractHeader
-import mainProcess
-import json
-import gzip
-import NonSSL
-import SSLReplay
-import h2Replay
-import itertools
-import random
-bSTOP = False
-
-
-def session_replay(input, proxy, result_queue):
- global bSTOP
- ''' Replay all transactions in session
-
- This entire session will be replayed in one requests.Session (so one
socket / TCP connection)'''
- # if timing_control:
- # time.sleep(float(session._timestamp)) # allow other threads to run
- while bSTOP == False:
- for session in iter(input.get, 'STOP'):
- # print(bSTOP)
- if session == 'STOP':
- print("Queue is empty")
- bSTOP = True
- break
- with requests.Session() as request_session:
- request_session.proxies = proxy
- for txn in session.getTransactionIter():
- type = random.randint(1, 1000)
- try:
- if type % 3 == 0:
- NonSSL.txn_replay(session._filename, txn, proxy,
result_queue, request_session)
- elif type % 3 == 1:
- SSLReplay.txn_replay(session._filename, txn,
proxy, result_queue, request_session)
- elif type % 3 == 2:
- h2Replay.txn_replay(session._filename, txn, proxy,
result_queue, request_session)
- except:
- e = sys.exc_info()
- print("ERROR in replaying: ", e,
txn.getRequest().getHeaders())
- bSTOP = True
- #print("Queue is empty")
- input.put('STOP')
- break
-
-
-def client_replay(input, proxy, result_queue, nThread):
- Threads = []
- for i in range(nThread):
-
- t2 = Thread(target=SSLReplay.session_replay, args=[input, proxy,
result_queue])
- t = Thread(target=NonSSL.session_replay, args=[input, proxy,
result_queue])
- t1 = Thread(target=h2Replay.session_replay, args=[input, proxy,
result_queue])
- t2.start()
- t.start()
- t1.start()
- Threads.append(t)
- Threads.append(t2)
- Threads.append(t1)
-
- for t1 in Threads:
- t1.join()
diff --git a/tests/tools/traffic-replay/SSLReplay.py
b/tests/tools/traffic-replay/SSLReplay.py
deleted file mode 100644
index c75b6a5..0000000
--- a/tests/tools/traffic-replay/SSLReplay.py
+++ /dev/null
@@ -1,233 +0,0 @@
-#!/bin/env python3
-'''
-'''
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import http.client
-import socket
-import ssl
-import pprint
-# import gevent
-import requests
-import os
-#import threading
-import sys
-from multiprocessing import current_process
-import sessionvalidation.sessionvalidation as sv
-import lib.result as result
-import extractHeader
-# from gevent import monkey, sleep
-from threading import Thread
-import mainProcess
-import json
-import extractHeader
-import time
-import Config
-bSTOP = False
-
-
-class ProxyHTTPSConnection(http.client.HTTPSConnection):
- "This class allows communication via SSL."
-
- default_port = http.client.HTTPS_PORT
-
- # XXX Should key_file and cert_file be deprecated in favour of context?
-
- def __init__(self, host, port=None, key_file=None, cert_file=None,
- timeout=socket._GLOBAL_DEFAULT_TIMEOUT,
- source_address=None, *, context=None,
- check_hostname=None, server_name=None):
- # http.client.HTTPSConnection.__init__(self)
- super().__init__(host, port, key_file, cert_file, timeout,
source_address, context=context, check_hostname=check_hostname)
- '''
- self.key_file = key_file
- self.cert_file = cert_file
- if context is None:
- context = ssl._create_default_https_context()
- will_verify = context.verify_mode != ssl.CERT_NONE
- if check_hostname is None:
- check_hostname = context.check_hostname
- if check_hostname and not will_verify:
- raise ValueError("check_hostname needs a SSL context with "
- "either CERT_OPTIONAL or CERT_REQUIRED")
- if key_file or cert_file:
- context.load_cert_chain(cert_file, key_file)
- self._context = context
- self._check_hostname = check_hostname
- '''
- self.server_name = server_name
-
- def connect(self):
- "Connect to a host on a given (SSL) port."
- http.client.HTTPConnection.connect(self)
-
- if self._tunnel_host:
- server_hostname = self._tunnel_host
- else:
- server_hostname = self.server_name
- self.sock = self._context.wrap_socket(self.sock,
- do_handshake_on_connect=True,
- server_side=False,
- server_hostname=server_hostname)
- if not self._context.check_hostname and self._check_hostname:
- try:
- ssl.match_hostname(self.sock.getpeercert(), server_hostname)
- except Exception:
- self.sock.shutdown(socket.SHUT_RDWR)
- self.sock.close()
- raise
-
-
-def txn_replay(session_filename, txn, proxy, result_queue, request_session):
- """ Replays a single transaction
- :param request_session: has to be a valid requests session"""
- req = txn.getRequest()
- resp = txn.getResponse()
- responseDict = {}
- # Construct HTTP request & fire it off
- txn_req_headers = req.getHeaders()
- txn_req_headers_dict = extractHeader.header_to_dict(txn_req_headers)
- txn_req_headers_dict['Content-MD5'] = txn._uuid # used as unique
identifier
- if 'body' in txn_req_headers_dict:
- del txn_req_headers_dict['body']
-
- #print("Replaying session")
- try:
- # response =
request_session.request(extractHeader.extract_txn_req_method(txn_req_headers),
- # 'http://' +
extractHeader.extract_host(txn_req_headers) +
extractHeader.extract_GET_path(txn_req_headers),
- #
headers=txn_req_headers_dict,stream=False) # making stream=False raises
contentdecoding exception? kill me
- method = extractHeader.extract_txn_req_method(txn_req_headers)
- response = None
- body = None
- content = None
- if 'Transfer-Encoding' in txn_req_headers_dict:
- # deleting the host key, since the STUPID post/get functions are
going to add host field anyway, so there will be multiple host fields in the
header
- # This confuses the ATS and it returns 400 "Invalid HTTP request".
I don't believe this
- # BUT, this is not a problem if the data is not chunked encoded..
Strange, huh?
- del txn_req_headers_dict['Host']
- if 'Content-Length' in txn_req_headers_dict:
- #print("ewww !")
- del txn_req_headers_dict['Content-Length']
- body = gen()
- if 'Content-Length' in txn_req_headers_dict:
- nBytes = int(txn_req_headers_dict['Content-Length'])
- body = createDummyBodywithLength(nBytes)
- #print("request session is",id(request_session))
-
- # NOTE: request_session here is actually python's HTTPSConnection,
which is different from that in NonSSL, which uses the requests library -_-
- if method == 'GET':
- request_session.request('GET', extractHeader.extract_GET_path(
- txn_req_headers), headers=txn_req_headers_dict, body=body)
- r1 = request_session.getresponse()
- responseContent = r1.read() # byte array
-
- elif method == 'POST':
- request_session.request('POST', extractHeader.extract_GET_path(
- txn_req_headers), headers=txn_req_headers_dict, body=body)
- r1 = request_session.getresponse()
- responseContent = r1.read()
-
- elif method == 'HEAD':
- request_session.request('HEAD', extractHeader.extract_GET_path(
- txn_req_headers), headers=txn_req_headers_dict, body=body)
- r1 = request_session.getresponse()
- responseContent = r1.read()
- else: # EXPERIMENTAL
- request_session.request(method, extractHeader.extract_GET_path(
- txn_req_headers), headers=txn_req_headers_dict, body=body)
- r1 = request_session.getresponse()
- responseContent = r1.read()
-
- responseHeaders =
extractHeader.responseHeaderTuple_to_dict(r1.getheaders())
- expected = extractHeader.responseHeader_to_dict(resp.getHeaders())
- # print("------------EXPECTED-----------")
- # print(expected)
- # print("------------RESP--------------")
- # print(responseHeaders)
- # print()
- if mainProcess.verbose:
- expected_output_split = resp.getHeaders().split('\r\n')[0].split('
', 2)
- expected_output = (int(expected_output_split[1]),
str(expected_output_split[2]))
- r = result.Result(session_filename, expected_output[0], r1.status,
responseContent)
- b_res, res = r.getResult(responseHeaders, expected, colorize=True)
- print(res)
-
- if not res:
- print("Received response")
- print(responseHeaders)
- print("Expected response")
- print(expected)
- # result_queue.put(r)
- except UnicodeEncodeError as e:
- # these unicode errors are due to the interaction between Requests and
our wiretrace data.
- # TODO fix
- print("UnicodeEncodeError exception")
-
- except requests.exceptions.ContentDecodingError as e:
- print("ContentDecodingError", e)
- except:
- e = sys.exc_info()
- print("ERROR in SSLReplay: ", e, response, session_filename)
-
-
-def client_replay(input, proxy, result_queue, nThread):
- Threads = []
- for i in range(nThread):
- t = Thread(target=session_replay, args=[input, proxy, result_queue])
- t.start()
- Threads.append(t)
-
- for t1 in Threads:
- t1.join()
-
-
-def session_replay(input, proxy, result_queue):
- ''' Replay all transactions in session
-
- This entire session will be replayed in one requests.Session (so one
socket / TCP connection)'''
- # if timing_control:
- # time.sleep(float(session._timestamp)) # allow other threads to run
- global bSTOP
- sslSocks = []
- while bSTOP == False:
- for session in iter(input.get, 'STOP'):
- txn = session.returnFirstTransaction()
- req = txn.getRequest()
- # Construct HTTP request & fire it off
- txn_req_headers = req.getHeaders()
- txn_req_headers_dict =
extractHeader.header_to_dict(txn_req_headers)
- sc = ssl.SSLContext(protocol=ssl.PROTOCOL_SSLv23)
- sc.load_cert_chain(Config.ca_certs, keyfile=Config.keyfile)
- conn = ProxyHTTPSConnection(Config.proxy_host,
Config.proxy_ssl_port, cert_file=Config.ca_certs,
- key_file=Config.keyfile, context=sc,
server_name=txn_req_headers_dict['Host'])
- for txn in session.getTransactionIter():
- try:
- # print(txn._uuid)
- txn_replay(session._filename, txn, proxy, result_queue,
conn)
- except:
- e = sys.exc_info()
- print("ERROR in replaying: ", e,
txn.getRequest().getHeaders())
- #sslSocket.bStop = False
-
- bSTOP = True
- print("stopping now")
- input.put('STOP')
- break
-
- # time.sleep(0.5)
- for sslSock in sslSocks:
- sslSock.ssl_sock.close()
diff --git a/tests/tools/traffic-replay/Scheduler.py
b/tests/tools/traffic-replay/Scheduler.py
deleted file mode 100644
index a1a4353..0000000
--- a/tests/tools/traffic-replay/Scheduler.py
+++ /dev/null
@@ -1,88 +0,0 @@
-#!/bin/env python3
-'''
-'''
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import time
-import random
-import json
-from multiprocessing import Process, Queue, current_process
-import sessionvalidation.sessionvalidation as sv
-import WorkerTask
-import time
-
-
-def LaunchWorkers(path, nProcess, proxy, replay_type, nThread):
- ms1 = time.time()
- s = sv.SessionValidator(path, allow_custom=True)
- sessions = s.getSessionList()
- sessions.sort(key=lambda session: session._timestamp)
- Processes = []
- Qsize = 25000 # int (1.5 * len(sessions)/(nProcess))
- QList = [Queue(Qsize) for i in range(nProcess)]
- print("Dropped {0} sessions for being malformed. Number of correct
sessions {1}".format(
- len(s.getBadSessionList()), len(sessions)))
- print(range(nProcess))
- OutputQ = Queue()
- #======================================== Pre-load queues
- for session in sessions:
- if replay_type == 'mixed':
- if nProcess < 2:
- raise ValueError("For mixed replay type, there should be at
least 2 processes.")
- # odd Qs for SSL sessions, even Qs for nonSSL sessions
- num = random.randint(0, nProcess - 1)
-
- # get the first transaction in each session, which is indictive if
session is over SSL or not
- if "https" in
session.returnFirstTransaction().getRequest().getHeaders():
- # spin until we get an odd number
- while num & 1 == 0:
- num = random.randint(0, nProcess - 1)
- else:
- # nonSSL sessions get put here into even Qs
- while num & 1 == 1:
- num = random.randint(0, nProcess - 1)
-
- QList[num].put(session)
- else:
- # if nProcess == 1:
- # QList[0].put(session)
- # else:
- QList[random.randint(0, nProcess - 1)].put(session)
- # if QList[0].qsize() > 10 :
- # break
- #=============================================== Launch Processes
- # for i in range(nProcess):
- # QList[i].put('STOP')
- for i in range(nProcess):
- QList[i].put('STOP')
-
- if replay_type == 'mixed':
- if i & 1: # odd/SSL
- p = Process(target=WorkerTask.worker, args=[QList[i], OutputQ,
proxy, 'ssl', nThread])
- else: # even/nonSSL
- p = Process(target=WorkerTask.worker, args=[QList[i], OutputQ,
proxy, 'nossl', nThread])
- else:
- p = Process(target=WorkerTask.worker, args=[QList[i], OutputQ,
proxy, replay_type, nThread])
-
- p.daemon = False
- Processes.append(p)
- p.start()
-
- for p in Processes:
- p.join()
- ms2 = time.time()
- print("OK enough, it is time to exit, running time in seconds", (ms2 -
ms1))
diff --git a/tests/tools/traffic-replay/WorkerTask.py
b/tests/tools/traffic-replay/WorkerTask.py
deleted file mode 100644
index 839e696..0000000
--- a/tests/tools/traffic-replay/WorkerTask.py
+++ /dev/null
@@ -1,49 +0,0 @@
-#!/bin/env python3
-'''
-'''
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import socket
-import requests
-import os
-#import threading
-import sys
-from multiprocessing import current_process
-import sessionvalidation.sessionvalidation as sv
-import lib.result as result
-import extractHeader
-import NonSSL
-import SSLReplay
-import h2Replay
-import RandomReplay
-
-
-def worker(input, output, proxy, replay_type, nThread):
- #progress_bar = Bar(" Replaying sessions
{0}".format(current_process().name), max=input.qsize())
- #print("playing
{0}=>{1}:{2}".format(current_process().name,session._timestamp,proxy))
- if replay_type == 'nossl':
- NonSSL.client_replay(input, proxy, output, nThread)
- elif replay_type == 'ssl':
- SSLReplay.client_replay(input, proxy, output, nThread)
- elif replay_type == 'h2':
- h2Replay.client_replay(input, proxy, output, nThread)
- elif replay_type == 'random':
- RandomReplay.client_replay(input, proxy, output, nThread)
-
- # progress_bar.next()
- # progress_bar.finish()
- print("process{0} has exited".format(current_process().name))
diff --git a/tests/tools/traffic-replay/__main__.py
b/tests/tools/traffic-replay/__main__.py
deleted file mode 100644
index 4b64e51..0000000
--- a/tests/tools/traffic-replay/__main__.py
+++ /dev/null
@@ -1,44 +0,0 @@
-#!/bin/env python3
-'''
-'''
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from __future__ import absolute_import, division, print_function
-import mainProcess
-import argparse
-import Config
-
-if __name__ == '__main__':
-
- parser = argparse.ArgumentParser()
- parser.add_argument("-type", action='store', dest='replay_type',
- help="Replay type: ssl/random/h2/nossl/mixed (at least
2 processes needed for mixed)")
- parser.add_argument("-log_dir", type=str, help="directory of JSON replay
files")
- parser.add_argument("-v", dest="verbose", help="verify response status
code", action="store_true")
- parser.add_argument("-host", help="proxy/host to send the requests to",
default=Config.proxy_host)
- parser.add_argument("-port", type=int, help=" The non secure port of ATS
to send the request to",
- default=Config.proxy_nonssl_port)
- parser.add_argument("-s_port", type=int, help="secure port",
default=Config.proxy_ssl_port)
- parser.add_argument("-ca_cert", help="Certificate to present",
default=Config.ca_certs)
- parser.add_argument("-colorize", type=str, help="specify whether to use
colorize the output", default='True')
-
- args = parser.parse_args()
-
- # Let 'er loose
- #main(args.log_dir, args.hostname, int(args.port), args.threads,
args.timing, args.verbose)
- Config.colorize = True if args.colorize == 'True' else False
- mainProcess.main(args.log_dir, args.replay_type, args.verbose,
pHost=args.host, pNSSLport=args.port, pSSL=args.s_port)
diff --git a/tests/tools/traffic-replay/extractHeader.py
b/tests/tools/traffic-replay/extractHeader.py
deleted file mode 100644
index a815183..0000000
--- a/tests/tools/traffic-replay/extractHeader.py
+++ /dev/null
@@ -1,91 +0,0 @@
-#!/bin/env python3
-'''
-'''
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import sessionvalidation.sessionvalidation as sv
-
-
-def extract_txn_req_method(headers):
- ''' Extracts the HTTP request method from the header in a string format '''
- line = (headers.split('\r\n'))[0]
- return (line.split(' '))[0]
-
-
-def extract_host(headers):
- ''' Returns the host header from the given headers '''
- lines = headers.split('\r\n')
- for line in lines:
- if 'Host:' in line:
- return line.split(' ')[1]
- return "notfound"
-
-def responseHeaderTuple_to_dict(header):
- header_dict = {}
-
- for key, val in header:
- if key.lower() in header_dict:
- header_dict[key.lower()] += ", {0}".format(val)
- else:
- header_dict[key.lower()] = val
-
- return header_dict
-
-def responseHeader_to_dict(header):
- headerFields = header.split('\r\n', 1)[1]
- fields = headerFields.split('\r\n')
- header = [x for x in fields if (x != u'')]
- headers = {}
- for line in header:
- split_here = line.find(":")
- # append multiple headers into a single string
- if line[:split_here].lower() in headers:
- headers[line[:split_here].lower()] += ",
{0}".format(line[(split_here + 1):].strip())
- else:
- headers[line[:split_here].lower()] = line[(split_here +
1):].strip()
-
- return headers
-
-
-def header_to_dict(header):
- ''' Convert a HTTP header in string format to a python dictionary
- Returns a dictionary of header values
- '''
- header = header.split('\r\n')
- header = [x for x in header if (x != u'')]
- headers = {}
- for line in header:
- should_skip = False
-
- # we have to ignore the intital request line with the HTTP method in it
- for method in sv.allowed_HTTP_request_methods:
- if method in line:
- should_skip = True
-
- if should_skip: # ignore initial request line
- continue
-
- split_here = line.find(":")
- headers[line[:split_here]] = line[(split_here + 1):].strip()
-
- return headers
-
-
-def extract_GET_path(headers):
- ''' Extracts the HTTP request URL from the header in a string format '''
- line = (headers.split('\r\n'))[0]
- return (line.split(' '))[1]
diff --git a/tests/tools/traffic-replay/h2Replay.py
b/tests/tools/traffic-replay/h2Replay.py
deleted file mode 100644
index 2026c74..0000000
--- a/tests/tools/traffic-replay/h2Replay.py
+++ /dev/null
@@ -1,331 +0,0 @@
-#!/bin/env python3
-'''
-'''
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import os
-from threading import Thread
-import sys
-from multiprocessing import current_process
-import sessionvalidation.sessionvalidation as sv
-import lib.result as result
-import extractHeader
-import mainProcess
-import json
-from hyper import HTTP20Connection
-from hyper.tls import wrap_socket, H2_NPN_PROTOCOLS, H2C_PROTOCOL
-from hyper.common.bufsocket import BufferedSocket
-import hyper
-import socket
-import logging
-import h2
-from h2.connection import H2Configuration
-import threading
-import Config
-
-log = logging.getLogger(__name__)
-bSTOP = False
-hyper.tls._context = hyper.tls.init_context()
-hyper.tls._context.check_hostname = False
-hyper.tls._context.verify_mode = hyper.compat.ssl.CERT_NONE
-
-
-class _LockedObject(object):
- """
- A wrapper class that hides a specific object behind a lock.
-
- The goal here is to provide a simple way to protect access to an object
- that cannot safely be simultaneously accessed from multiple threads. The
- intended use of this class is simple: take hold of it with a context
- manager, which returns the protected object.
- """
-
- def __init__(self, obj):
- self.lock = threading.RLock()
- self._obj = obj
-
- def __enter__(self):
- self.lock.acquire()
- return self._obj
-
- def __exit__(self, _exc_type, _exc_val, _exc_tb):
- self.lock.release()
-
-
-class h2ATS(HTTP20Connection):
-
- def __init_state(self):
- """
- Initializes the 'mutable state' portions of the HTTP/2 connection
- object.
-
- This method exists to enable HTTP20Connection objects to be reused if
- they're closed, by resetting the connection object to its basic state
- whenever it ends up closed. Any situation that needs to recreate the
- connection can call this method and it will be done.
-
- This is one of the only methods in hyper that is truly private, as
- users should be strongly discouraged from messing about with connection
- objects themselves.
- """
-
- config1 = H2Configuration(
- client_side=True,
- header_encoding='utf-8',
- validate_outbound_headers=False,
- validate_inbound_headers=False,
-
- )
- self._conn = _LockedObject(h2.connection.H2Connection(config=config1))
-
- # Streams are stored in a dictionary keyed off their stream IDs. We
- # also save the most recent one for easy access without having to walk
- # the dictionary.
- #
- # We add a set of all streams that we or the remote party forcefully
- # closed with RST_STREAM, to avoid encountering issues where frames
- # were already in flight before the RST was processed.
- #
- # Finally, we add a set of streams that recently received data. When
- # using multiple threads, this avoids reading on threads that have just
- # acquired the I/O lock whose streams have already had their data read
- # for them by prior threads.
- self.streams = {}
- self.recent_stream = None
- self.next_stream_id = 1
- self.reset_streams = set()
- self.recent_recv_streams = set()
-
- # The socket used to send data.
- self._sock = None
-
- # Instantiate a window manager.
- #self.window_manager = self.__wm_class(65535)
-
- return
-
- def __init__(self, host, **kwargs):
- HTTP20Connection.__init__(self, host, **kwargs)
- self.__init_state()
-
- def connect(self):
- """
- Connect to the server specified when the object was created. This is a
- no-op if we're already connected.
-
- Concurrency
- -----------
-
- This method is thread-safe. It may be called from multiple threads, and
- is a noop for all threads apart from the first.
-
- :returns: Nothing.
-
- """
- #print("connecting to ATS")
- with self._lock:
- if self._sock is not None:
- return
- sni = self.host
- if not self.proxy_host:
- host = self.host
- port = self.port
- else:
- host = self.proxy_host
- port = self.proxy_port
-
- sock = socket.create_connection((host, port))
-
- if self.secure:
- #assert not self.proxy_host, "Proxy with HTTPS not supported."
- sock, proto = wrap_socket(sock, sni, self.ssl_context,
- force_proto=self.force_proto)
- else:
- proto = H2C_PROTOCOL
-
- log.debug("Selected NPN protocol: %s", proto)
- assert proto in H2_NPN_PROTOCOLS or proto == H2C_PROTOCOL
-
- self._sock = BufferedSocket(sock, self.network_buffer_size)
-
- self._send_preamble()
-
-
-def createDummyBodywithLength(numberOfbytes):
- if numberOfbytes == 0:
- return None
- body = 'a'
- while numberOfbytes != 1:
- body += 'b'
- numberOfbytes -= 1
- return body
-
-
-def handleResponse(response, *args, **kwargs):
- print(response.status_code)
- # resp=args[0]
- #expected_output_split = resp.getHeaders().split('\r\n')[ 0].split(' ', 2)
- #expected_output = (int(expected_output_split[1]), str(
expected_output_split[2]))
- #r = result.Result(session_filename, expected_output[0],
response.status_code)
- # print(r.getResultString(colorize=True))
-# make sure len of the message body is greater than length
-
-
-def gen():
- yield 'pforpersia,champaignurbana'.encode('utf-8')
- yield 'there'.encode('utf-8')
-
-
-def txn_replay(session_filename, txn, proxy, result_queue, h2conn,
request_IDs):
- """ Replays a single transaction
- :param request_session: has to be a valid requests session"""
- req = txn.getRequest()
- resp = txn.getResponse()
- # Construct HTTP request & fire it off
- txn_req_headers = req.getHeaders()
- txn_req_headers_dict = extractHeader.header_to_dict(txn_req_headers)
- txn_req_headers_dict['Content-MD5'] = txn._uuid # used as unique
identifier
- if 'body' in txn_req_headers_dict:
- del txn_req_headers_dict['body']
- responseID = -1
- #print("Replaying session")
- try:
- # response =
request_session.request(extractHeader.extract_txn_req_method(txn_req_headers),
- # 'http://' +
extractHeader.extract_host(txn_req_headers) +
extractHeader.extract_GET_path(txn_req_headers),
- #
headers=txn_req_headers_dict,stream=False) # making stream=False raises
contentdecoding exception? kill me
- method = extractHeader.extract_txn_req_method(txn_req_headers)
- response = None
- mbody = None
- #txn_req_headers_dict['Host'] = "localhost"
- if 'Transfer-Encoding' in txn_req_headers_dict:
- # deleting the host key, since the STUPID post/get functions are
going to add host field anyway, so there will be multiple host fields in the
header
- # This confuses the ATS and it returns 400 "Invalid HTTP request".
I don't believe this
- # BUT, this is not a problem if the data is not chunked encoded..
Strange, huh?
- #del txn_req_headers_dict['Host']
- if 'Content-Length' in txn_req_headers_dict:
- #print("ewww !")
- del txn_req_headers_dict['Content-Length']
- mbody = gen()
- if 'Content-Length' in txn_req_headers_dict:
- nBytes = int(txn_req_headers_dict['Content-Length'])
- mbody = createDummyBodywithLength(nBytes)
- if 'Connection' in txn_req_headers_dict:
- del txn_req_headers_dict['Connection']
- #str2 = extractHeader.extract_host(txn_req_headers)+
extractHeader.extract_GET_path(txn_req_headers)
- # print(str2)
- if method == 'GET':
- responseID = h2conn.request('GET',
url=extractHeader.extract_GET_path(
- txn_req_headers), headers=txn_req_headers_dict, body=mbody)
- # print("get response", responseID)
- return responseID
- # request_IDs.append(responseID)
- #response = h2conn.get_response(id)
- # print(response.headers)
- # if 'Content-Length' in response.headers:
- # content = response.read()
- #print("len: {0} received
{1}".format(response.headers['Content-Length'],content))
-
- elif method == 'POST':
- responseID = h2conn.request('POST',
url=extractHeader.extract_GET_path(
- txn_req_headers), headers=txn_req_headers_dict, body=mbody)
- print("get response", responseID)
- return responseID
-
- elif method == 'HEAD':
- responseID = h2conn.request('HEAD',
url=extractHeader.extract_GET_path(txn_req_headers),
headers=txn_req_headers_dict)
- print("get response", responseID)
- return responseID
-
- except UnicodeEncodeError as e:
- # these unicode errors are due to the interaction between Requests and
our wiretrace data.
- # TODO fix
- print("UnicodeEncodeError exception")
-
- except:
- e = sys.exc_info()
- print("ERROR in requests: ", e, response, session_filename)
-
-
-def session_replay(input, proxy, result_queue):
- global bSTOP
- ''' Replay all transactions in session
-
- This entire session will be replayed in one requests.Session (so one
socket / TCP connection)'''
- # if timing_control:
- # time.sleep(float(session._timestamp)) # allow other threads to run
- while bSTOP == False:
- for session in iter(input.get, 'STOP'):
- print(bSTOP)
- if session == 'STOP':
- print("Queue is empty")
- bSTOP = True
- break
- txn = session.returnFirstTransaction()
- req = txn.getRequest()
- # Construct HTTP request & fire it off
- txn_req_headers = req.getHeaders()
- txn_req_headers_dict =
extractHeader.header_to_dict(txn_req_headers)
- with h2ATS(txn_req_headers_dict['Host'], secure=True,
proxy_host=Config.proxy_host, proxy_port=Config.proxy_ssl_port) as h2conn:
- request_IDs = []
- respList = []
- for txn in session.getTransactionIter():
- try:
- ret = txn_replay(session._filename, txn, proxy,
result_queue, h2conn, request_IDs)
- respList.append(txn.getResponse())
- request_IDs.append(ret)
- #print("txn return value is ",ret)
- except:
- e = sys.exc_info()
- print("ERROR in replaying: ", e,
txn.getRequest().getHeaders())
- for id in request_IDs:
- expectedH = respList.pop(0)
- # print("extracting",id)
- response = h2conn.get_response(id)
- #print("code
{0}:{1}".format(response.status,response.headers))
- response_dict = {}
- if mainProcess.verbose:
- for field, value in response.headers.items():
- response_dict[field.decode('utf-8')] =
value.decode('utf-8')
-
- expected_output_split =
expectedH.getHeaders().split('\r\n')[0].split(' ', 2)
- expected_output = (int(expected_output_split[1]),
str(expected_output_split[2]))
- r = result.Result("", expected_output[0],
response.status, response.read())
- expected_Dict =
extractHeader.responseHeader_to_dict(expectedH.getHeaders())
- b_res, res = r.getResult(response_dict, expected_Dict,
colorize=Config.colorize)
- print(res)
-
- if not b_res:
- print("Received response")
- print(response_dict)
- print("Expected response")
- print(expected_Dict)
-
- bSTOP = True
- #print("Queue is empty")
- input.put('STOP')
- break
-
-
-def client_replay(input, proxy, result_queue, nThread):
- Threads = []
- for i in range(nThread):
- t = Thread(target=session_replay, args=[input, proxy, result_queue])
- t.start()
- Threads.append(t)
-
- for t1 in Threads:
- t1.join()
diff --git a/tests/tools/traffic-replay/mainProcess.py
b/tests/tools/traffic-replay/mainProcess.py
deleted file mode 100644
index bde8de4..0000000
--- a/tests/tools/traffic-replay/mainProcess.py
+++ /dev/null
@@ -1,76 +0,0 @@
-#!/bin/env python3
-'''
-'''
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-
-import sys
-import json
-import socket
-import os
-import threading
-import time
-import argparse
-import subprocess
-import shlex
-from multiprocessing import Pool, Process
-from collections import deque
-#from progress.bar import Bar
-
-sys.path.append(
- os.path.normpath(
- os.path.join(
- os.path.dirname(os.path.abspath(__file__)),
- '..'
- )
- )
-)
-
-import sessionvalidation.sessionvalidation as sv
-import lib.result as result
-import WorkerTask
-import Scheduler
-import Config
-verbose = False
-
-
-def check_for_ats(hostname, port):
- ''' Checks to see if ATS is running on `hostname` and `port`
- If not running, this function will terminate the script
- '''
- sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
- result = sock.connect_ex((hostname, port))
- if result != 0:
- # hostname:port is not being listened to
- print('==========')
- print('Error: Apache Traffic Server is not running on
{0}:{1}'.format(hostname, port))
- print('Aborting')
- print('==========')
- sys.exit()
-# Note: this function can't handle multi-line (ie wrapped line) headers
-# Hopefully this isn't an issue because multi-line headers are deprecated now
-
-
-def main(path, replay_type, Bverbose, pHost=Config.proxy_host,
pNSSLport=Config.proxy_nonssl_port, pSSL=Config.proxy_ssl_port):
- global verbose
- verbose = Bverbose
- check_for_ats(pHost, pNSSLport)
- Config.proxy_host = pHost
- Config.proxy_nonssl_port = pNSSLport
- Config.proxy_ssl_port = pSSL
- proxy = {"http": "http://{0}:{1}".format(Config.proxy_host,
Config.proxy_nonssl_port)}
- Scheduler.LaunchWorkers(path, Config.nProcess, proxy, replay_type,
Config.nThread)
diff --git a/tests/unit_tests/Makefile.am b/tests/unit_tests/Makefile.am
index 84db12b..791c600 100644
--- a/tests/unit_tests/Makefile.am
+++ b/tests/unit_tests/Makefile.am
@@ -27,3 +27,4 @@ unit_tests_SOURCES = main.cpp
clang-tidy-local: $(DIST_SOURCES)
$(CXX_Clang_Tidy)
+
\ No newline at end of file
