I found the issue. The upgrade is looking for config to have [database] section and connection definition. If I use /etc/neutron/neutron.conf, then the neutron-db-manage runs.
On Tue, Jul 7, 2015 at 3:38 PM Paul Michali <[email protected]> wrote: > I have that change in the neutron repo that is being used with by this > neutron-vpnaas repo. > > On Tue, Jul 7, 2015 at 3:12 PM Mike Bayer <[email protected]> wrote: > >> >> >> On 7/7/15 1:28 PM, Paul Michali wrote: >> >> HEAD, head, 24f28869838b (my new file) all say the same thing. :( >> >> >> On Tue, Jul 7, 2015 at 12:34 PM Salvatore Orlando <[email protected]> >> wrote: >> >>> possibly I was wrong in mixing up git & alembic. >>> It should be "upgrade head" - lowercase. >>> >>> If that doesn't work there might some other issue lurking. >>> >>> Salvatore >>> >>> On 7 July 2015 at 17:44, Paul Michali <[email protected]> wrote: >>> >>>> Salvatore, >>>> >>>> I changed head to the version before my new one, and then tried to >>>> upgrade and I see this: >>>> neutron-db-manage --config-file /opt/stack/neutron/etc/neutron.conf >>>> --service vpnaas upgrade HEAD >>>> Traceback (most recent call last): >>>> File "/usr/local/bin/neutron-db-manage", line 10, in <module> >>>> sys.exit(main()) >>>> File "/opt/stack/neutron/neutron/db/migration/cli.py", line 238, in >>>> main >>>> CONF.command.func(config, CONF.command.name) >>>> File "/opt/stack/neutron/neutron/db/migration/cli.py", line 105, in >>>> do_upgrade >>>> run_sanity_checks(config, revision) >>>> File "/opt/stack/neutron/neutron/db/migration/cli.py", line 229, in >>>> run_sanity_checks >>>> script_dir.run_env() >>>> File "/usr/local/lib/python2.7/dist-packages/alembic/script.py", line >>>> 390, in run_env >>>> util.load_python_file(self.dir, 'env.py') >>>> File "/usr/local/lib/python2.7/dist-packages/alembic/util.py", line >>>> 243, in load_python_file >>>> module = load_module_py(module_id, path) >>>> File "/usr/local/lib/python2.7/dist-packages/alembic/compat.py", line >>>> 79, in load_module_py >>>> mod = imp.load_source(module_id, path, fp) >>>> File >>>> "/opt/stack/neutron-vpnaas/neutron_vpnaas/db/migration/alembic_migrations/env.py", >>>> line 86, in <module> >>>> run_migrations_online() >>>> File >>>> "/opt/stack/neutron-vpnaas/neutron_vpnaas/db/migration/alembic_migrations/env.py", >>>> line 67, in run_migrations_online >>>> engine = session.create_engine(neutron_config.database.connection) >>>> File >>>> "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py", >>>> line 112, in create_engine >>>> url = sqlalchemy.engine.url.make_url(sql_connection) >>>> File >>>> "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/url.py", line >>>> 186, in make_url >>>> return _parse_rfc1738_args(name_or_url) >>>> File >>>> "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/url.py", line >>>> 235, in _parse_rfc1738_args >>>> "Could not parse rfc1738 URL from string '%s'" % name) >>>> sqlalchemy.exc.ArgumentError: Could not parse rfc1738 URL from string '' >>>> >>>> Any ideas what is wrong here? >>>> >>> >> I'm going to guess this is the issue fixed by >> https://review.openstack.org/#/c/194197/ >> >> >> >> >> >> >>>> On Tue, Jul 7, 2015 at 10:05 AM Paul Michali <[email protected]> wrote: >>>> >>>>> >>>>> Yes, I wasn't using the --service option, so I suspect that is why >>>>> my down_version was wrong. In talking with Akihiro, I added a check to >>>>> PEP8 and made sure that it fails if head is wrong. It is: >>>>> https://review.openstack.org/#/c/199082/ (of course that failed py27 >>>>> - I've got to see if there was some recent breakage in vpn repo, again). >>>>> >>>>> Regarding the migration, one of the new columns may be None, but >>>>> there must be at least one IP version entry (there is an existing test in >>>>> VPN for using a router w/o an external IP set). Since the new code will >>>>> rely on these new fields, I'd like to populate them as part of the >>>>> migration. I think it would be more complicated to handle during >>>>> operation. >>>>> >>>>> Does anyone have examples of how to do queries of objects, from the >>>>> migration upgrade() code? >>>>> >>>>> >>>>> Regards, >>>>> >>>>> PCM >>>>> >>>>> On Tue, Jul 7, 2015 at 9:02 AM Akihiro Motoki <[email protected]> >>>>> wrote: >>>>> >>>>>> 2015-07-07 21:39 GMT+09:00 Henry Gessau <[email protected]>: >>>>>> >>>>>>> On Tue, Jul 07, 2015, Paul Michali <[email protected]> >>>>>>> <[email protected]> <[email protected]> wrote: >>>>>>> >>>>>>> Thanks Salvatore for the responses. See @PCM in-line... >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Tue, Jul 7, 2015 at 6:14 AM Salvatore Orlando < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Some comments inline. >>>>>>>> >>>>>>>> Salvatore >>>>>>>> >>>>>>>> On 6 July 2015 at 20:00, Paul Michali <[email protected]> wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> I have some urgent requests about migration that I'm hoping to >>>>>>>>> get some info on. I'm working on a bug where I need to add two >>>>>>>>> (related) >>>>>>>>> fields to a table for VPNaaS. Here's the objectives related to >>>>>>>>> migration... >>>>>>>>> >>>>>>>>> 1) create local_v4_ip and lcoal_v6_ip fields in the vpnservice >>>>>>>>> table >>>>>>>>> 2) for each entry in the vpnservice table: >>>>>>>>> 2.1) Get the router.gw_port.fixed_ips list >>>>>>>>> 2.2) Determine the version of each fixed IP and store the >>>>>>>>> first of each version (if any) into the appropriate new field. >>>>>>>>> >>>>>>>>> I have created a migration file, and I changed the down_revision >>>>>>>>> to be the number of the revision that is the first in the migration >>>>>>>>> chain >>>>>>>>> in the VPN repo. >>>>>>>>> >>>>>>>>> Here are the many questions I have... >>>>>>>>> >>>>>>>>> When I look in the VPN repo, the HEAD file has the version >>>>>>>>> 'kilo', which is not the current head. >>>>>>>>> >>>>>>>> >>>>>>>>> Shouldn't it the version number of the first file in the >>>>>>>>> migration chain? >>>>>>>>> >>>>>>>> >>>>>>>> It should indeed. How are you generating the revision script? >>>>>>>> Using neutron-db-manage it should be updated automatically [1] >>>>>>>> >>>>>>> >>>>>>> @PCM I ran neutron-db-manage, when in the neutron repo, and it >>>>>>> assigned some version, but it was not the latest in the neutron-vpnaas >>>>>>> repo. >>>>>>> >>>>>>> neutron-db-manage does not handle alembic branches in separate >>>>>>> repos very well at all yet. I am working on updating it with >>>>>>> https://review.openstack.org/198524 but I have quite a lot left to >>>>>>> do. >>>>>>> >>>>>> >>>>>> Yes, at now we have implicit order of running alembic migrations. >>>>>> First run neutron db migration and then advanced service migrations. >>>>>> >>>>>> I do not fully understand how alembic branch mechanism works, but >>>>>> I think we can have a common ancestor and have multiple branches, >>>>>> and each branch can evolve independently. >>>>>> >>>>>> >>>>>>> >>>>>>> I checked the VPN repo and there were a chain of versions, which I >>>>>>> used to determine what the head should be and have set the version >>>>>>> accordingly. However, in the current repo, head is set to "kilo", which >>>>>>> appears to be incorrect. The versions are: >>>>>>> >>>>>>> 56893333aa52 >>>>>>> kilo <<< HEAD >>>>>>> 3ea02b2a773e >>>>>>> start_neutron_vpnaas >>>>>>> None >>>>>>> >>>>>>> Ouch. That is an error, because https://review.openstack.org/190569 >>>>>>> should have updated HEAD but didn't. >>>>>>> >>>>>>> The version sequence (you can see it in any devstack run) is: >>>>>>> >>>>>>> INFO [alembic.migration] Running upgrade -> start_neutron_vpnaas, >>>>>>> start neutron-vpnaas chain >>>>>>> INFO [alembic.migration] Running upgrade start_neutron_vpnaas -> >>>>>>> 3ea02b2a773e, add_index_tenant_id >>>>>>> INFO [alembic.migration] Running upgrade 3ea02b2a773e -> kilo, kilo >>>>>>> INFO [alembic.migration] Running upgrade kilo -> 56893333aa52, fix >>>>>>> identifier map fk >>>>>>> >>>>>> >>>>>> It seems we don't have an appropriate check for HEAD revision in >>>>>> at least VPNaaS repo. >>>>>> Paul and I just discussed it. We need to improve the check too. >>>>>> >>>>>> >>>>>>> Should I do a separate commit that fixes the HEAD file, or just >>>>>>> fix it as part of the bug fix I'm working on. >>>>>>> >>>>>>> Yes, you should immediately submit a patch to change HEAD to >>>>>>> 56893333aa52. >>>>>>> >>>>>>> >>>>>>> BTW, at one point, after having correctly set the HEAD and >>>>>>> versions in my new migration file, I think I ran neutron-db-manage >>>>>>> check_migration, and I think it set the HEAD to my version, but it did >>>>>>> that >>>>>>> in the neutron repo, and not the VPN repo. I might have been running >>>>>>> from >>>>>>> the wrong repo? >>>>>>> >>>>>>> I working on updating the devref docs for this process. Things have >>>>>>> changed quite a bit with the alembic branches in separate repos. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> For my commit, I'm assuming I change the HEAD file to use my >>>>>>>>> migration file's version? >>>>>>>>> >>>>>>>> >>>>>>>> You can do that manually too, yes. >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> I set HEAD to my migration file, and my file has a down revision >>>>>>>>> of the previous head's revision. If I run 'neutron-db-manage >>>>>>>>> --config-file >>>>>>>>> ../neutron/etc/neutron.conf --config-file >>>>>>>>> ../neutron/etc/neutron/plugins/ml2/ml2_conf.ini check_migration' >>>>>>>>> there is >>>>>>>>> no output so I guess that is OK. >>>>>>>>> >>>>>>>>> As I develop my new migration file, is there a way that I can >>>>>>>>> test it (running neutron-db-migration, maybe)? >>>>>>>>> >>>>>>>> >>>>>>>> When I test migrations I usually dump the database, run the >>>>>>>> migration with neutron-db-manage upgrade HEAD (I think it's not >>>>>>>> necessary >>>>>>>> to specify HEAD), and restore the db from the dump if the migration >>>>>>>> fails. >>>>>>>> >>>>>>>> >>>>>>>>> Is there a way to run the migration file under the debugger, as >>>>>>>>> well (importing pdb, for example)? >>>>>>>>> >>>>>>>> >>>>>>>> The migration process is just like any python application, so >>>>>>>> I guess you can debug it with pdb. >>>>>>>> >>>>>>> >>>>>>> @PCM Ah, so use "neutron-db-manage upgrade HEAD". That was the >>>>>>> piece that was missing. I take it there are no specific unit tests of >>>>>>> the >>>>>>> migration files? >>>>>>> >>>>>>> >>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> In the migration, I can add the columns needed. What's the best >>>>>>>>> way to fill out those fields - using raw SQL queries or create a >>>>>>>>> Session >>>>>>>>> object and access the VpnService object's router object? >>>>>>>>> >>>>>>>> >>>>>>>> If the default value for the column is not enough, and you >>>>>>>> need to specify a value which depends on other values in the same row I >>>>>>>> would prefer plain SQL statements, but if that become cumbersome I >>>>>>>> guess >>>>>>>> it's ok to use sqlalchemy's session. >>>>>>>> >>>>>>>> >>>>>>>>> I see there is some op.bind() call and then engine.execute(), >>>>>>>>> but could use some help on the best way to extract the needed queries >>>>>>>>> (I >>>>>>>>> need to access the vpnservice's router, and then access the (Port) >>>>>>>>> gw_port >>>>>>>>> relationship, and from that access the (IPAllocation) fixed_ips list). >>>>>>>>> >>>>>>>> >>>>>>>> Perhaps you can point us to the review pages on gerrit, and >>>>>>>> we can provide detailed comments there. >>>>>>>> >>>>>>> >>>>>>> @PCM Yeah, I haven't pushed it up yet. I have a few more changes >>>>>>> to make, and should be able to get it up in a few days. The LP bug is >>>>>>> 1464387. >>>>>>> >>>>>>> Essentially, in the vpnservices table, I'm adding an IPv4 and/or >>>>>>> IPv6 addresses for the "local" end of VPN connections that will be >>>>>>> established. This is to allow alternative VPN implementation >>>>>>> (appliances, >>>>>>> separate S/W, H/W, VM based VPN, etc) to specify addresses different >>>>>>> than >>>>>>> what is available on the Neutron router. >>>>>>> >>>>>>> However, for the reference implementation, we'll use the Neutron >>>>>>> router's fixed_ips list (as is done today), and to handle the migration, >>>>>>> I'm thinking the following is needed: >>>>>>> >>>>>>> 1) create the new columns. >>>>>>> 2) Identify the router for that service and obtain it's GW fixed_ips >>>>>>> list. >>>>>>> 3) Pick first IPv4 address (if any) and IPv6 address (if any), and >>>>>>> store in new columns. >>>>>>> >>>>>>> So I need to form a query and code to do this. >>>>>>> >>>>>>> >>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> Appreciate any advise here on how to debug the migration stuff... >>>>>>>>> >>>>>>>>> Paul Michali (pc_m) >>>>>>>>> >>>>>>>> >>>>>>>> [1] >>>>>>>> http://git.openstack.org/cgit/openstack/neutron/tree/neutron/db/migration/cli.py#n124 >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>> >>>>>>> >>>>>>> __________________________________________________________________________ >>>>>>> OpenStack Development Mailing List (not for usage questions) >>>>>>> Unsubscribe: >>>>>>> [email protected]?subject:unsubscribe >>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>>>> >>>>>>> >>>>>> __________________________________________________________________________ >>>>>> OpenStack Development Mailing List (not for usage questions) >>>>>> Unsubscribe: >>>>>> [email protected]?subject:unsubscribe >>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>>> >>>>> >>>> >>>> __________________________________________________________________________ >>>> OpenStack Development Mailing List (not for usage questions) >>>> Unsubscribe: >>>> [email protected]?subject:unsubscribe >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>> >>>> >>> >>> __________________________________________________________________________ >>> OpenStack Development Mailing List (not for usage questions) >>> Unsubscribe: >>> [email protected]?subject:unsubscribe >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >> >> >> __________________________________________________________________________ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: >> [email protected]?subject:unsubscribehttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> >> >> __________________________________________________________________________ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: >> [email protected]?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
