Re: [W3af-develop] W3af Ubuntu 13.10
Don't want to re-open this, but just a FYI: https://github.com/axiak/pybloomfiltermmap/issues/29 In other words, 0.3.11 of pybloomfiltermmap installation works flawlessly, while 0.3.12 fails with gcc compilation errors :( If we would have had = this would have been an issue for us too. On Tue, Feb 18, 2014 at 2:15 PM, Taras ox...@oxdef.info wrote: Andres, Ok, I've got your opinion. Let's close this discussion. 17.02.2014 00:04, Andres Riancho пишет: Taras, On Sun, Feb 16, 2014 at 4:28 PM, Taras ox...@oxdef.info wrote: Andres, I think it is my last attempt to change your opinion :) From the list of software you have provided I have found only flask, scrapy and tastypie in Ubuntu repo. Results of apt-cache show output are below inline. The problem is w3af built-in dependency checker duplicates OS (e.g. Debian/Ubuntu) packaging system. They can conflict in some cases. For example, I want to make package of w3af for Ubuntu 13.10. There is package python-xml version 3.2.0 in repository. At the same time w3af requires lxml version exactly 2.3.2. How can I make package of w3af? Should I add sudo pip install into preinstall script? Most likely not, that doesn't sound well. I don't know the right answer because I'm not packaging expert. The package maintainer can always apply a patch on top of the original software to remove the dependency check completely (I think Luciano did something like this [0]) is he believes it is the best thing to do. Then he's taking the responsibility of that change. My responsibility is to tell you that with these specific package versions it works; then people do whatever they want with it. [0] http://packages.ubuntu.com/precise/w3af-console - search for diff Have you got any feedback from w3af package maintainers for Debian/Ubuntu and other distributions after you had add strict dependencies? There are no active package maintainers for w3af. They even don't care, or don't want to maintain this software; so no, no package maintainer told me anything about the ==. As I said above, they can apply a diff to the software before packaging it, as done by Luciano a while ago (not only for the dependency). Is it important for you that w3af can be installed via simple command apt-get install w3af or through Ubuntu Software Center with single mouse click? Yes, and not. Some users would find it awesome to be able to install it from the repo; but this has proven to be (at least for w3af) a failed path. I'm not going to maintain a package for each distribution, because I don't care enough as a user myself. Packagers who have come to the project have either failed to release their initial package or released it and then moved their free time to something else. In this process, they left very old versions of w3af in the repositories of all linux distributions; which don't even make sense for users. If users can install w3af with: git clone ... cd w3af ./w3af_console # Yields error with all dependencies to install /tmp/install_w3af_dependencies.sh Then I'm happy. If it is important for you then I recommend to add maintainers into this discussion and ask if it is easy for them to make package of w3af with such requirements. My opinion is that they don't care about the w3af package. If it is not so important and git clone + pip install is preferable way of installation then thread can be closed. In the past I've thought that having w3af in the linux distribution repos was THE BEST THING, now... not so much, because: * Software packages are difficult to maintain * Each time a new dependency is added the maintainer needs to create a new package for that (python-foo) and then maintain that one also * The whole process takes time, so from the minute I put something in the repo to the time the new package is there it can be months; and hackers love to use the latest and they will come to the repo anyways Not 100% a workaround, this is also a best practice! https://devcenter.heroku.com/articles/python-pip#the-basics Could you please show at least one example of well-know software with such requirements? I went through this list of the Top10 Python projects by github (not sure how they choose that) and found many that either had no dependencies or were not in a format in which we could compare them with what we were talking about. Then found the following: * Strict dependencies used for this part of the project: https://github.com/torchbox/wagtail/blob/master/requirements-dev.txt * Gt used for the user installable part: https://github.com/torchbox/wagtail/blob/master/setup.py * These guys install whatever is available on pypi: https://github.com/jmcarp/robobrowser/blob/master/requirements.txt * Flask installs Gt: https://github.com/mitsuhiko/flask/blob/master/setup.py Depends: python-itsdangerous, python (= 2.7), python-jinja2 (= 2.4), python ( 2.8),
Re: [W3af-develop] W3af Ubuntu 13.10
Andres, I think it is my last attempt to change your opinion :) From the list of software you have provided I have found only flask, scrapy and tastypie in Ubuntu repo. Results of apt-cache show output are below inline. The problem is w3af built-in dependency checker duplicates OS (e.g. Debian/Ubuntu) packaging system. They can conflict in some cases. For example, I want to make package of w3af for Ubuntu 13.10. There is package python-xml version 3.2.0 in repository. At the same time w3af requires lxml version exactly 2.3.2. How can I make package of w3af? Should I add sudo pip install into preinstall script? Have you got any feedback from w3af package maintainers for Debian/Ubuntu and other distributions after you had add strict dependencies? Is it important for you that w3af can be installed via simple command apt-get install w3af or through Ubuntu Software Center with single mouse click? If it is important for you then I recommend to add maintainers into this discussion and ask if it is easy for them to make package of w3af with such requirements. If it is not so important and git clone + pip install is preferable way of installation then thread can be closed. Not 100% a workaround, this is also a best practice! https://devcenter.heroku.com/articles/python-pip#the-basics Could you please show at least one example of well-know software with such requirements? I went through this list of the Top10 Python projects by github (not sure how they choose that) and found many that either had no dependencies or were not in a format in which we could compare them with what we were talking about. Then found the following: * Strict dependencies used for this part of the project: https://github.com/torchbox/wagtail/blob/master/requirements-dev.txt * Gt used for the user installable part: https://github.com/torchbox/wagtail/blob/master/setup.py * These guys install whatever is available on pypi: https://github.com/jmcarp/robobrowser/blob/master/requirements.txt * Flask installs Gt: https://github.com/mitsuhiko/flask/blob/master/setup.py Depends: python-itsdangerous, python (= 2.7), python-jinja2 (= 2.4), python ( 2.8), python-werkzeug (= 0.8) Recommends: python-pkg-resources, python-blinker * A mix between Gt and whatever is used here: https://github.com/Eugeny/ajenti/blob/dev/requirements.txt * Scrapy uses a mix of GT and whatever: https://github.com/scrapy/scrapy/blob/master/requirements.txt Depends: python2.7, python (= 2.7.1-0ubuntu2), python ( 2.8), python-twisted-core, python-twisted-web, python-twisted-conch, python-twisted-mail, python-libxml2, python-boto, python-w3lib Recommends: python-lxml, python-guppy, python-django, ipython, python-pygments, python-imaging, python-mysqldb * Django-tastypie uses the most complex of them all, which is rather interesting and makes me wonder why they didn't use == instead: https://github.com/toastdriven/django-tastypie/blob/master/setup.py . This is what I mean: 'dateutil(=1.5, !=2.0)' Replaces: python-django-tastypie (= 0.9.9-2) Depends: python (= 2.7.1-0ubuntu2), python ( 2.8), python-mimeparse (= 0.1.3), python-dateutil (= 1.5), python-django (= 1.2) Suggests: python-yaml, python-lxml The first one is an example of ==, the rest were just to show that now everyone agrees with me on what should be put on the requirements.txt file (or the setup.py, which acts like the same many times). Here are some other links where it says that == is a best practice: * https://lincolnloop.com/django-best-practices/deployment/bootstrap.html (Ctrl+f Pin your dependencies) * http://docs.dotcloud.com/tutorials/python/django/#specifying-requirements (Ctrl+f When you specify your requirements) And most importantly, the pip-installer user's guide: * http://www.pip-installer.org/en/latest/user_guide.html#ensuring-repeatability The requirements file was generated by pip freeze or you're sure it only contains requirements that specify a specific version. When we're talking about including a specific version in requirements.txt file or not, we're talking about repeatability. I want to be strict about repeatability, forcing all libraries to be exactly the ones I know will work because I've tested them in the CI; and your point is that it would be easier for users to install with less strict version requirements (which could lead to issues in some cases). Sadly, you believe in one thing and I can't seem to convince you of the benefits of ==, and the same applies the other way (I can't be convinced of the benefits of =). Unless I hear a definitive reason on why == is bad, I won't change it. By the way in w3af dev list I see fresh discussion about similar problems in Mageia Linux distro http://sourceforge.net/mailarchive/message.php?msg_id=31315478 I think that email thread was correctly answered? 1. Bring back dependency check with = condition Disagree with this, it will bring
Re: [W3af-develop] W3af Ubuntu 13.10
Andres, may be we will add to CC Luciano (luci...@debian.org) who is maintainer of w3af package in Debian? * The pdfminer issue occurred because we had this requirement: pdfminer (no version requirement) * If we specify something like: pdfminer=3, then we're fine until they release version 4 which breaks their API and w3af breaks Breaking of API is unusual and infrequent case in normal software. Agreed, but we already found one issue with this and don't want to find more in the future. How old is w3af project? How many times has this (breaking of 3rd party API) happened? If it's the first one then may be it's too excessive workaround? Of course it is possible that we will have some similar issues in the future. But as for me it is not reason to specify exact versions of dependencies. It is reason to keep really small number of core dependencies. And these dependencies should be well-maintained packages. I thought that specifying the exact version was the best solution, but at least for what you're saying, it is not. Can you propose a solution that will be bullet-proof? My view on w3af dependency management is: 1. Bring back dependency check with = condition 2. We should separate core and plugins requirements 3. We should make possible to run w3af without installation of all plugins dependencies. It can be with special argument to w3af_console called -l or --lazy. This parameter will force w3af not to check plugins dependencies (or even switch off dependency checker all!). If user specifies plugin with not installed external dependency w3af will show message how to install it using e.g. pip. Without such parameter w3af will run as currently. **So default behavior will not be changed.** 4. Such improvement will make possible to make easier e.g. Debian/Ubuntu package of w3af. Core dependencies will be in Depends: section and plugins dependencies will be Recommends: section. If there is no some plugin dependency in repository - no problem because user can install it via pip. If you agree with this I will code it. In another case it will break current package system ideology in Linux distros. Not sure why you say that? Could you please explain? Just try to find e.g. in Ubuntu repository package with such strict dependencies. It will be difficult task! Which command do I run to get such a list? I simply have tried to look on some well-known Python based packages like Sonata, Inkscape, Calibre, Exaile. Same is true for usual software: $ apt-cache show firefox Also, there should be a way in ubuntu packaging to solve this issue... I believe its not a big deal and we're not unique. I bet there are many packages which are in this dilemma: * Package A depends on library X version 1 * Package B depends on library X version 2 * A won't work with X.2 * B won't work with X.1 We certainly need a packaging expert for solving this part of the discussion! I don't know enough about it, or care enough to learn. If in the future someone wants to package w3af, I'll try to remember this discussion and let him know. * If we specify the version: pdfminer==3, then we're fine for ever. Yes, we're fine, but **who** and **how** will be able to install and use w3af? Virtualenv is not solution for the end user. Only for development. Who? Every user How? git clone ... cd w3af ./w3af_console follow steps in output The only problem I see here is that when following the steps in the output this might happen: * User installed in the past package A version 2 using apt-get install * User installs w3af using the instructions above * w3af requires A version 3 * By following the instructions, A.2 is overwritten by A.3 Is that what is worrying you? I really worry about how to run and package w3af without painful resolving dependencies in Debian/Ubuntu system. It should be as easy as installing any other well know software. -- Taras https://www.oxdef.info -- Android apps run on BlackBerry 10 Introducing the new BlackBerry 10.2.1 Runtime for Android apps. Now with support for Jelly Bean, Bluetooth, Mapview and more. Get your Android app in front of a whole new audience. Start now. http://pubads.g.doubleclick.net/gampad/clk?id=124407151iu=/4140/ostg.clktrk ___ W3af-develop mailing list W3af-develop@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/w3af-develop
Re: [W3af-develop] W3af Ubuntu 13.10
On Wed, Feb 12, 2014 at 1:15 PM, Taras ox...@oxdef.info wrote: Andres, Sorry for delayed reply. Not sure if I'm understanding your point. * The pdfminer issue occurred because we had this requirement: pdfminer (no version requirement) * If we specify something like: pdfminer=3, then we're fine until they release version 4 which breaks their API and w3af breaks Breaking of API is unusual and infrequent case in normal software. Agreed, but we already found one issue with this and don't want to find more in the future. I thought that specifying the exact version was the best solution, but at least for what you're saying, it is not. Can you propose a solution that will be bullet-proof? In another case it will break current package system ideology in Linux distros. Not sure why you say that? Could you please explain? Just try to find e.g. in Ubuntu repository package with such strict dependencies. It will be difficult task! Which command do I run to get such a list? Also, there should be a way in ubuntu packaging to solve this issue... I believe its not a big deal and we're not unique. I bet there are many packages which are in this dilemma: * Package A depends on library X version 1 * Package B depends on library X version 2 * A won't work with X.2 * B won't work with X.1 We certainly need a packaging expert for solving this part of the discussion! I don't know enough about it, or care enough to learn. If in the future someone wants to package w3af, I'll try to remember this discussion and let him know. * If we specify the version: pdfminer==3, then we're fine for ever. Yes, we're fine, but **who** and **how** will be able to install and use w3af? Virtualenv is not solution for the end user. Only for development. Who? Every user How? git clone ... cd w3af ./w3af_console follow steps in output The only problem I see here is that when following the steps in the output this might happen: * User installed in the past package A version 2 using apt-get install * User installs w3af using the instructions above * w3af requires A version 3 * By following the instructions, A.2 is overwritten by A.3 Is that what is worrying you? В письме от 1 февраля 2014 14:36:05 пользователь Taras написал: Andres, When I talked about packaging problem I meant problems with supported versions of e.g. python libs for current popular distros. Consider we have e.g. some Debian/Ubuntu distro and want to package/install w3af from official repo. w3af from feature/package branch requires lxml version exactly 2.3.2, but supported and packaged version of lxml for Ubuntu 13.10 is 3.2.0! $ apt-cache show python-lxml Package: python-lxml Priority: optional Section: python Installed-Size: 2390 Maintainer: Ubuntu Developers ubuntu-devel-disc...@lists.ubuntu.com Original-Maintainer: Matthias Klose d...@debian.org Architecture: amd64 Source: lxml Version: 3.2.0-1 Because of that you can't simply make and provide w3af thought official repo. No one package maintainer will support several packaged minor versions of single lib.And for the end user there is only one way to install and use w3af. It is virtualenv + git clone :( 1. It makes impossible to packageinstall w3af, e.g. into deb package, doesn't it? That's a good question, I'm not packaging expert but I suppose there is a solution? Also I suppose that this was an issue in the past, without the specific version requirement? Lets follow this timeline: * (assume) w3af is packaged in debian. Requires extra package python-pdfminer-v1. No check for specific version of any pip package. * foo is another debian package. Requires extra package python-pdfminer-v2 * User installs w3af: apt-get install w3af * Run w3af, it works * User installs foo: apt-get install foo - Command will warn that it will break the w3af install? (not sure, not a packaging expert) - Command will succeed and replace python-pdfminer-v1 with python-pdfminer-v2 * Run foo, it works * Run w3af, it fails because now python-pdfminer-v2, which changes the API is installed 2. If w3af requires 3rd party A version 1 and another application on the system also requires 3rd party A but version 1.1, how it will be solved by the user? First, lets understand that this was an issue in the past too, right? You can always use virtualenv: $ virtualenv w3af-venv $ . w3af-venv/bin/activate (w3af-venv)$ cd w3af-repo (w3af-venv)/w3af-repo$ ./w3af_console (w3af-venv)/w3af-repo$ pip install ... All the packages are installed inside the w3af-venv directory, and while your prompt says w3af-venv you're using that specific python Regards, В письме от 29 января 2014 19:03:23 пользователь Andres Riancho написал: Taras, Added that because it is the best thing to do. Search the mailing list for the issue we had with pdfminer, what happen there was:
Re: [W3af-develop] W3af Ubuntu 13.10
I was wrong...I have working **master** branch :(
Andres, why did you add requirement for **exact** match of versions in
'feature/module' branch?
$ grep -B5 'version matches'
w3af/core/controllers/dependency_check/dependency_check.py
for w3af_req in pip_packages:
if USE_PIP_MODULE:
dependency_specs = w3af_req.package_name, w3af_req.package_version
for dist in pip_distributions:
if (dist.project_name, dist.version) == dependency_specs:
# It's installed and the version matches!
...
В письме от 26 января 2014 14:39:14 пользователь Taras написал:
Israel, I have working feature/module version of w3af on 13.10
What problems do you have?
В письме от 22 января 2014 21:53:48 пользователь Andres Riancho написал:
Israel,
Haven't tried with that specific version, but what's wrong with:
git clone g...@github.com:andresriancho/w3af.git
cd w3af
git checkout feature/module
./w3af_console
On Wed, Jan 22, 2014 at 6:00 PM, Israel Duvdavan
israelzero...@gmail.com wrote:
Hi, does anyone have a working way to install W3af on 13.10?
--
Israel
--
CenturyLink Cloud: The Leader in Enterprise Cloud Services.
Learn Why More Businesses Are Choosing CenturyLink Cloud For
Critical Workloads, Development Environments Everything In Between.
Get a Quote or Start a Free Trial Today.
http://pubads.g.doubleclick.net/gampad/clk?id=119420431iu=/4140/ostg.cl
kt
rk ___
W3af-develop mailing list
W3af-develop@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/w3af-develop
--
Taras
https://www.oxdef.info
--
WatchGuard Dimension instantly turns raw network data into actionable
security intelligence. It gives you real-time visual feedback on key
security issues and trends. Skip the complicated setup - simply import
a virtual appliance and go from zero to informed in seconds.
http://pubads.g.doubleclick.net/gampad/clk?id=123612991iu=/4140/ostg.clktrk
___
W3af-develop mailing list
W3af-develop@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/w3af-develop
Re: [W3af-develop] W3af Ubuntu 13.10
Andres, Thanks for description of the reason. There are at least two issues with such requirements: 1. It makes impossible to packageinstall w3af, e.g. into deb package, doesn't it? 2. If w3af requires 3rd party A version 1 and another application on the system also requires 3rd party A but version 1.1, how it will be solved by the user? В письме от 29 января 2014 19:03:23 пользователь Andres Riancho написал: Taras, Added that because it is the best thing to do. Search the mailing list for the issue we had with pdfminer, what happen there was: * w3af had a requirement for pdfminer, any version * w3af worked without issues with version 1 of that library * The pdfminer developers released version 2 of that library * People trying to install w3af, and because the requirement didn't had any specific version installed pdfminer like pip install pdfminer * w3af stopped working because pdfminer changed its API, and one of the functions we were calling wasn't there anymore * Fix Add specific version matching for pip packages On Wed, Jan 29, 2014 at 5:46 PM, Taras ox...@oxdef.info wrote: I was wrong...I have working **master** branch :( Andres, why did you add requirement for **exact** match of versions in 'feature/module' branch? $ grep -B5 'version matches' w3af/core/controllers/dependency_check/dependency_check.py for w3af_req in pip_packages: if USE_PIP_MODULE: dependency_specs = w3af_req.package_name, w3af_req.package_version for dist in pip_distributions: if (dist.project_name, dist.version) == dependency_specs: # It's installed and the version matches! ... В письме от 26 января 2014 14:39:14 пользователь Taras написал: Israel, I have working feature/module version of w3af on 13.10 What problems do you have? В письме от 22 января 2014 21:53:48 пользователь Andres Riancho написал: Israel, Haven't tried with that specific version, but what's wrong with: git clone g...@github.com:andresriancho/w3af.git cd w3af git checkout feature/module ./w3af_console On Wed, Jan 22, 2014 at 6:00 PM, Israel Duvdavan israelzero...@gmail.com wrote: Hi, does anyone have a working way to install W3af on 13.10? -- Israel - --- -- CenturyLink Cloud: The Leader in Enterprise Cloud Services. Learn Why More Businesses Are Choosing CenturyLink Cloud For Critical Workloads, Development Environments Everything In Between. Get a Quote or Start a Free Trial Today. http://pubads.g.doubleclick.net/gampad/clk?id=119420431iu=/4140/ostg .cl kt rk ___ W3af-develop mailing list W3af-develop@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/w3af-develop -- Taras https://www.oxdef.info -- Taras https://www.oxdef.info -- WatchGuard Dimension instantly turns raw network data into actionable security intelligence. It gives you real-time visual feedback on key security issues and trends. Skip the complicated setup - simply import a virtual appliance and go from zero to informed in seconds. http://pubads.g.doubleclick.net/gampad/clk?id=123612991iu=/4140/ostg.clktrk ___ W3af-develop mailing list W3af-develop@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/w3af-develop
Re: [W3af-develop] W3af Ubuntu 13.10
Israel, Haven't tried with that specific version, but what's wrong with: git clone g...@github.com:andresriancho/w3af.git cd w3af git checkout feature/module ./w3af_console On Wed, Jan 22, 2014 at 6:00 PM, Israel Duvdavan israelzero...@gmail.com wrote: Hi, does anyone have a working way to install W3af on 13.10? -- Israel -- CenturyLink Cloud: The Leader in Enterprise Cloud Services. Learn Why More Businesses Are Choosing CenturyLink Cloud For Critical Workloads, Development Environments Everything In Between. Get a Quote or Start a Free Trial Today. http://pubads.g.doubleclick.net/gampad/clk?id=119420431iu=/4140/ostg.clktrk ___ W3af-develop mailing list W3af-develop@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/w3af-develop -- Andrés Riancho Project Leader at w3af - http://w3af.org/ Web Application Attack and Audit Framework Twitter: @w3af GPG: 0x93C344F3 -- CenturyLink Cloud: The Leader in Enterprise Cloud Services. Learn Why More Businesses Are Choosing CenturyLink Cloud For Critical Workloads, Development Environments Everything In Between. Get a Quote or Start a Free Trial Today. http://pubads.g.doubleclick.net/gampad/clk?id=119420431iu=/4140/ostg.clktrk ___ W3af-develop mailing list W3af-develop@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/w3af-develop
