Hello. > Is this common? At this rate, the tests would take about 2 days to complete > and there'd probably be lots of failures.
Yes. System tests are heavy and resource expensive. You can run a single test with the TC_PATH environment variable. ``` TC_PATHS="tests/kafkatest/tests/client/pluggable_test.py" bash tests/docker/run_tests.sh ``` As you may know, there is no infrastructure to run system tests on public servers. I’m usually using some external machine from cloud providers to check system tests for my PR’s. > 29 июня 2020 г., в 17:57, Gokul Ramanan Subramanian <gokul24...@gmail.com> > написал(а): > > I have been running the tests for over 5 hours now, and only 60 or so out > of the 600 tests are through. Many tests are timing out. For example: > > [INFO:2020-06-29 07:52:54,606]: RunnerClient: > kafkatest.tests.streams.streams_cooperative_rebalance_upgrade_test.StreamsCooperativeRebalanceUpgradeTest.test_upgrade_to_cooperative_rebalance.upgrade_from_version=2.3.1: > FAIL: Never saw 'Processed [0-9]* records so far' message ducker@ducker07 > Traceback (most recent call last): > File > "/usr/local/lib/python2.7/dist-packages/ducktape/tests/runner_client.py", > line 132, in run > data = self.run_test() > File > "/usr/local/lib/python2.7/dist-packages/ducktape/tests/runner_client.py", > line 189, in run_test > return self.test_context.function(self.test) > File "/usr/local/lib/python2.7/dist-packages/ducktape/mark/_mark.py", > line 428, in wrapper > return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs) > File > "/opt/kafka-dev/tests/kafkatest/tests/streams/streams_cooperative_rebalance_upgrade_test.py", > line 90, in test_upgrade_to_cooperative_rebalance > verify_running(processor, self.processing_message) > File "/opt/kafka-dev/tests/kafkatest/tests/streams/utils/util.py", line > 21, in verify_running > err_msg="Never saw '%s' message " % message + > str(processor.node.account)) > File > "/usr/local/lib/python2.7/dist-packages/ducktape/cluster/remoteaccount.py", > line 705, in wait_until > allow_fail=True) == 0, **kwargs) > File "/usr/local/lib/python2.7/dist-packages/ducktape/utils/util.py", > line 41, in wait_until > raise TimeoutError(err_msg() if callable(err_msg) else err_msg) > TimeoutError: Never saw 'Processed [0-9]* records so far' message > ducker@ducker07 > > My Macbook pro has over 4GB of RAM free and between 15-45 % idle CPU. So, I > don't think I am running low on resources. > > Is this common? At this rate, the tests would take about 2 days to complete > and there'd probably be lots of failures. > > On Mon, Jun 29, 2020 at 11:21 AM Gokul Ramanan Subramanian < > gokul24...@gmail.com> wrote: > >> Thanks. This worked. I had to use the Docker desktop app, not >> docker-machine, with which it gave errors around permissions setting up >> /etc/hosts. >> >> On Mon, Jun 29, 2020 at 10:24 AM Nikolay Izhikov <nizhi...@apache.org> >> wrote: >> >>> Hello, >>> >>> I successfully run system tests on Mac with Docker. >>> I followed the instruction on [1] and it works like a charm. >>> >>> [1] >>> https://github.com/apache/kafka/tree/trunk/tests#running-tests-using-docker >>> >>> >>>> 29 июня 2020 г., в 12:14, Gokul Ramanan Subramanian < >>> gokul24...@gmail.com> написал(а): >>>> >>>> Hi. >>>> >>>> Has anyone had luck running Kafka system tests on a Mac. I have a MacOS >>>> Mojave 10.14.6. I got Python 3.6.9 using pyenv. However, the command >>>> *ducktape tests/kafkatest/tests* yields the following error, making it >>> look >>>> like some Python incompatibility issue. >>>> >>>> $ ducktape tests/kafkatest/tests >>>> Traceback (most recent call last): >>>> File "/Users/gokusubr/.pyenv/versions/3.6.9/bin/ducktape", line 11, in >>>> <module> >>>> load_entry_point('ducktape', 'console_scripts', 'ducktape')() >>>> File >>>> >>> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py", >>>> line 487, in load_entry_point >>>> return get_distribution(dist).load_entry_point(group, name) >>>> File >>>> >>> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py", >>>> line 2728, in load_entry_point >>>> return ep.load() >>>> File >>>> >>> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py", >>>> line 2346, in load >>>> return self.resolve() >>>> File >>>> >>> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py", >>>> line 2352, in resolve >>>> module = __import__(self.module_name, fromlist=['__name__'], level=0) >>>> File >>>> >>> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/ducktape-0.7.6-py3.6.egg/ducktape/command_line/main.py", >>>> line 127 >>>> print "parameters are not valid json: " + str(e.message) >>>> ^ >>>> SyntaxError: invalid syntax >>>> >>>> I followed the instructions in tests/README.md to setup a cluster of 9 >>>> worker machines. That worked well. When I ran *python setup.py develop* >>> to >>>> install the necessary dependencies (including ducktape), I got similar >>>> errors to above, but the overall command completed successfully. >>>> >>>> Any help appreciated. >>>> >>>> Thanks. >>> >>>
