I don't think jobs are ever programmatically removed from the jobs table nor are their statuses changed. They always remain in a pending state.
SELECT COUNT(*) FROM job WHERE status NOT IN ( SELECT id FROM job_status WHERE name = 'PENDING' ) 0 Like I mentioned earlier, I think there was a lot more planned for jobs but as it stands today, the jobs table is currently for "invalidate content" (purge) jobs ONLY and the jobs go in to that table and never come out. It's kind of like the roach motel :) https://www.youtube.com/watch?v=jKhGHxO-woc&feature=youtu.be&t=27 Anyhow, it's important to remember that when initiating an "invalidate content" request for delivery service X, the important thing is that the upd_pending flag for all servers that serve that delivery service is flipped to true that way each one of those servers will fetch a new regex_revalidate.config file that ATS needs to "invalidate" the asset. Whether or not a job is created really has nothing to do with how content is invalidated. It's currently just a way to record who did what and when. I'm going to create an issue to remove those unused tables (job_status, job_result, job_agent) if nobody is opposed to it. Maybe I'll send out a different email specifically about that. Jeremy On Wed, Feb 1, 2017 at 2:07 AM, Oren Shemesh <[email protected]> wrote: > Thanks for your replies and the information :-) > > Dave, Is there a place I can document a work-around for being included in > the release notes of 1.8 ? > My suggested work-around would be fixing the create_tables.sql script > before running 'postinstall'. (I have a nice awk command that does it). > > And a related question: Can someone comment please about the process which > jobs are removed from the job table (i.e. Actually delete the records) ? I > could not find any code that does this. > Or maybe content invalidation jobs are supposed to remain in the DB forever > ? > > Thanks, Oren. > > On Wed, Feb 1, 2017 at 1:26 AM, Steve Malenfant <[email protected]> > wrote: > > > Please look at the release notes of 1.3.0, might solve your problem. > > > > https://github.com/Comcast/traffic_control/releases/tag/RELEASE-1.3.0 > > > > The revalidation feature doesn't work out of the box. The entry id=1 in > the > > job_agent table MUST exist as well as name=PURGE must be present in the > > job_status table. > > > > On Tue, Jan 31, 2017 at 3:31 PM, Jeremy Mitchell <[email protected]> > > wrote: > > > > > In my opinion, the job agent id should have been passed to the newjob() > > > subroutine rather than depending on a hard-coded value...actually, it > > looks > > > like that was the intent but the $agent variable never got used. > > > > > > https://github.com/apache/incubator-trafficcontrol/blob/ > > > master/traffic_ops/app/lib/UI/Job.pm#L42 > > > > > > Also, I don't think the concept of jobs, job agents, etc was ever fully > > > fleshed out (somebody correct me if i'm wrong), therefore, it would be > my > > > opinion that we eventually kill these tables: > > > > > > - job_agent > > > - job_status > > > - job_result > > > > > > this would eliminate the job_agent FK from the job table and then this > > > problem of potentially using an job agent ID that doesn't exist goes > bye > > > bye... > > > > > > also, fyi, i believe the job table really only contains "purge" aka > > > "invalidate content" jobs...(defined by keyword=purge). > > > > > > Jeremy > > > > > > On Tue, Jan 31, 2017 at 12:42 PM, Dave Neuman <[email protected]> > wrote: > > > > > > > Thanks for the investigation. Sounds like there is definitely a bug > > > > there. I would vote don't worry about fixing the issue in 1.8 since > it > > > is > > > > fixed in master. We should, however, make a release note or > something > > > > explaining the issue and the work around. > > > > > > > > --Dave > > > > > > > > On Tue, Jan 31, 2017 at 12:32 PM, Oren Shemesh <[email protected]> > > wrote: > > > > > > > > > Continuing the investigation of this issue, I have found what I > > believe > > > > to > > > > > be an inconsistency in the code that handles content invalidation. > > > > > To the best of my understanding, the bug exists in 1.7.x, and in > the > > > > latest > > > > > 1.8.x as well, and was only fixed in master, as part of the move to > > > > > postgresql. > > > > > > > > > > The bug prevents you from adding a job, you get an immediate error > in > > > the > > > > > UI. > > > > > > > > > > Question: Can anyone attest to Content Invalidation working in a > > fresh > > > > > install of TC 1.7 or 1.8 ? > > > > > > > > > > Another question: Given that the problem probably does not exist in > > > > > 'master', do we care about fixing it in the 1.8.x train (Or a > > following > > > > > train, non-postgresql) ? > > > > > If so, I will issue a JIRA. > > > > > > > > > > The details: > > > > > > > > > > I believe there is an inconsistency between the code that creates > the > > > TO > > > > DB > > > > > during a clean install, and the code that depends on it. > > > > > > > > > > Creating the TO DB is done by traffic_ops/app/db/create_ > tables.sql. > > In > > > > the > > > > > mysql version (1.7 and 1.8), it creates the 'job_agent' table and > the > > > > > 'job_status' table with table options AUTO_INCREMENT=2 > > > > > and AUTO_INCREMENT=5, respectively: > > > > > > > > > > *CREATE TABLE `job_agent`* ( > > > > > `id` int(11) NOT NULL AUTO_INCREMENT, > > > > > ... > > > > > ) ENGINE=InnoDB *AUTO_INCREMENT=2* DEFAULT CHARSET=latin1; > > > > > > > > > > *CREATE TABLE `job_status`* ( > > > > > `id` int(11) NOT NULL AUTO_INCREMENT, > > > > > ... > > > > > ) ENGINE=InnoDB *AUTO_INCREMENT=5* DEFAULT CHARSET=latin1; > > > > > > > > > > This means that when the script that populates these tables > > > > > (traffic_ops/app/db/migrations/20151208000000_add_job_status.sql) > is > > > > run, > > > > > it creates entries with an id that begins with 2 and 5, > respectively. > > > > > > > > > > However, the code that attempts to add an entry to the 'job' table, > > > > assumes > > > > > that the id numbers in both tables, begins at 1. > > > > > > > > > > From traffic_ops/app/lib/UI/Job.pm: > > > > > > > > > > sub newjob { > > > > > ... > > > > > my $*status = 1*; > > > > > ... > > > > > my $insert = $self->db->resultset('Job')->create( > > > > > { > > > > > *agent => 1,* > > > > > object_type => $object_type, > > > > > object_name => $object_name, > > > > > entered_time => $entered_time, > > > > > keyword => $keyword, > > > > > parameters => $parameters, > > > > > asset_url => $org_server_fqdn, > > > > > asset_type => $asset_type, > > > > > *status => $status*, > > > > > job_user => $user, > > > > > start_time => $start_time_gmt, > > > > > job_deliveryservice => $ds->id, > > > > > } > > > > > > > > > > As you can see, both the 'agent' and the 'status' fields are set > to a > > > > > hard-coded value of 1, which cannot exist in tables that are set to > > > have > > > > an > > > > > auto-incremented value of 'id' that begins with 2 or 5. > > > > > > > > > > When looking at the traffic_ops/app/db/create_tables.sql script in > > > > > 'master', which is now adapted to postgresql, it seems that it is > > > totally > > > > > different code, but the inconsistency is now gone, because the > > > numbering > > > > > starts at 1: > > > > > > > > > > CREATE TABLE job_agent ( > > > > > id bigint NOT NULL, > > > > > name text, > > > > > description text, > > > > > active integer DEFAULT 0 NOT NULL, > > > > > last_updated timestamp with time zone DEFAULT now() > > > > > ); > > > > > > > > > > CREATE SEQUENCE job_agent_id_seq > > > > > * START WITH 1* > > > > > INCREMENT BY 1 > > > > > NO MINVALUE > > > > > NO MAXVALUE > > > > > CACHE 1; > > > > > > > > > > ALTER SEQUENCE job_agent_id_seq OWNED BY job_agent.id; > > > > > > > > > > Regards, Oren. > > > > > > > > > > On Thu, Jan 26, 2017 at 5:36 PM, Dave Neuman <[email protected]> > > > wrote: > > > > > > > > > > > Good to hear. It's too bad that we have some routes requiring a > > > > specific > > > > > > ID from the database, when you come across those can you open a > > Jira > > > > > issue > > > > > > so we know they need to be fixed? > > > > > > Thanks, > > > > > > Dave > > > > > > > > > > > > On Thu, Jan 26, 2017 at 5:17 AM, Naama Shoresh <[email protected] > > > > > > wrote: > > > > > > > > > > > > > Hi Dave, > > > > > > > > > > > > > > This is the next error I got. Already fixed the DB accordingly. > > > > > > > Traffic Ops fatal error occurred while processing your request. > > > > > > > ------------------------------ > > > > > > > Error at line 146 ( my $insert = $self->db->resultset('Job')-> > > > > create() > > > > > > > ------------------------------ > > > > > > > DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: > > > > > DBD::mysql::st > > > > > > > execute failed: Cannot add or update a child row: a foreign key > > > > > > constraint > > > > > > > fails (`traffic_ops_db`.`job`, CONSTRAINT `fk_job_status_id1` > > > FOREIGN > > > > > KEY > > > > > > > (`status`) REFERENCES `job_status` (`id`) ON DELETE NO ACTION > ON > > > > UPDATE > > > > > > NO > > > > > > > ACTION) [for Statement "INSERT INTO job ( agent, asset_type, > > > > asset_url, > > > > > > > entered_time, job_deliveryservice, job_user, keyword, > > object_name, > > > > > > > object_type, parameters, start_time, status) VALUES ( ?, ?, ?, > ?, > > > ?, > > > > ?, > > > > > > ?, > > > > > > > ?, ?, ?, ?, ? )" with ParamValues: 0=1, 1="file", 2=" > > > > > > > http://images1.ynet.co.il/foo/.*";, 3='2017-01-24 09:16:46', > > > 4='311', > > > > > > > 5='59', 6="PURGE", 7=undef, 8=undef, 9="TTL:48h", > 10='2017-01-24 > > > > > > 09:05:26', > > > > > > > 11=1] at /opt/traffic_ops/app/lib/UI/Job.pm line 146 > > > > > > > > > > > > > > What happened was that TO post_install script failed the first > > > time I > > > > > ran > > > > > > > it, and I ran it again. As a result, the records in job_status > & > > > > > > job_agent > > > > > > > were added, removed, and added again, and their ids were > offset. > > I > > > > > guess > > > > > > > there are several routes in the code counting on a specific > > record > > > id > > > > > in > > > > > > > the DB. > > > > > > > > > > > > > > After fixing the DB, there was another error, the parameter > > > > > snapshot_dir > > > > > > > was missing from regex_revalidate.config profile. > > > > > > > When I added the missing parameter,"Invalidate content" worked > as > > > > > > expected. > > > > > > > > > > > > > > Thanks a lot for your help. > > > > > > > > > > > > > > Naama > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Jan 24, 2017 at 5:16 PM, Dave Neuman < > [email protected]> > > > > > wrote: > > > > > > > > > > > > > > > can you send the new error? > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Jan 24, 2017 at 2:18 AM, Naama Shoresh < > > [email protected] > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > actually, I didn't look closely enough. It is now > complaining > > > > > about a > > > > > > > > > different table. > > > > > > > > > Thanks! > > > > > > > > > > > > > > > > > > On Tue, Jan 24, 2017 at 11:13 AM, Naama Shoresh < > > > > [email protected]> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > I checked. The exact same error. > > > > > > > > > > It seemed weird to me, too. > > > > > > > > > > > > > > > > > > > > On Tue, Jan 24, 2017 at 11:12 AM, Oren Shemesh < > > > > [email protected]> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > >> [1:1] > > > > > > > > > >> > > > > > > > > > >> R U sure that the exact same error repeats ? > > > > > > > > > >> It seems that now the constraint is satisfied, so if > there > > > is > > > > an > > > > > > > > error, > > > > > > > > > it > > > > > > > > > >> will be a different one ... > > > > > > > > > >> > > > > > > > > > >> (Not that I know SQL, but my sqenglish is good enough > :-) > > > > > > > > > >> > > > > > > > > > >> On Tue, Jan 24, 2017 at 10:57 AM, Naama Shoresh < > > > > > [email protected] > > > > > > > > > > > > > > > > wrote: > > > > > > > > > >> > > > > > > > > > >> > I tried adding a row to the job_agent table: > > > > > > > > > >> > mysql> select * from job_agent; > > > > > > > > > >> > +----+-------+---------------- > > > > ------------+--------+-------- > > > > > > > > > >> -------------+ > > > > > > > > > >> > | id | name | description | active | > > > > > > last_updated > > > > > > > > > >> | > > > > > > > > > >> > +----+-------+---------------- > > > > ------------+--------+-------- > > > > > > > > > >> -------------+ > > > > > > > > > >> > | 1 | PURGE | Description of Purge Agent | 1 | > > > > > 2017-01-24 > > > > > > > > > >> 08:50:22 | > > > > > > > > > >> > | 2 | dummy | Description of Purge Agent | 1 | > > > > > 2017-01-01 > > > > > > > > > >> 10:28:43 | > > > > > > > > > >> > +----+-------+---------------- > > > > ------------+--------+-------- > > > > > > > > > >> -------------+ > > > > > > > > > >> > 2 rows in set (0.00 sec) > > > > > > > > > >> > > > > > > > > > > >> > But the error repeats. > > > > > > > > > >> > Any ideas? > > > > > > > > > >> > > > > > > > > > > >> > Thanks, > > > > > > > > > >> > Naama > > > > > > > > > >> > > > > > > > > > > >> > On Tue, Jan 24, 2017 at 12:17 AM, Dave Neuman < > > > > > > [email protected]> > > > > > > > > > >> wrote: > > > > > > > > > >> > > > > > > > > > > >> > > The error message says that you are trying to > submit a > > > job > > > > > > with > > > > > > > an > > > > > > > > > >> Agent > > > > > > > > > >> > id > > > > > > > > > >> > > of 1, which doesn't exist. > > > > > > > > > >> > > ```INSERT INTO job ( agent, asset_type, asset_url, > > > > > > > > > >> > > entered_time, job_deliveryservice, job_user, > keyword, > > > > > > > object_name, > > > > > > > > > >> > > object_type, parameters, start_time, status) VALUES > ( > > ?, > > > > ?, > > > > > ?, > > > > > > > ?, > > > > > > > > ?, > > > > > > > > > >> ?, > > > > > > > > > >> > ?, > > > > > > > > > >> > > ?, ?, ?, ?, ? )" with ParamValues: 0=1, 1="file", > 2=" > > > > > > > > > >> > > http://s2.ipcamlive.com/.*/ > > 7464537492280autoresize.jpg > > > ", > > > > > > > > > >> 3='2017-01-22 > > > > > > > > > >> > > 13:04:53', 4='312', 5='59', 6="PURGE", 7=undef, > > 8=undef, > > > > > > > > > 9="TTL:48h", > > > > > > > > > >> > > 10='2017-01-22 13:05:22', 11=1] at > > > > > > /opt/traffic_ops/app/lib/UI/ > > > > > > > > > Job.pm > > > > > > > > > >> > line > > > > > > > > > >> > > 146``` > > > > > > > > > >> > > I am not sure how that agent is supposed to be > > created, > > > > this > > > > > > > might > > > > > > > > > be > > > > > > > > > >> a > > > > > > > > > >> > > bug. Can you please submit an issue? > > > > > > > > > >> > > The workaround would be to create an agent with id = > > 1. > > > > > > > > > >> > > > > > > > > > > > >> > > Thanks, > > > > > > > > > >> > > Dave > > > > > > > > > >> > > > > > > > > > > > >> > > On Mon, Jan 23, 2017 at 3:05 PM, Naama Shoresh < > > > > > > > [email protected]> > > > > > > > > > >> wrote: > > > > > > > > > >> > > > > > > > > > > > >> > > > No, only this one: > > > > > > > > > >> > > > (2,'dummy','Description of Purge > > Agent',1,'2017-01-04 > > > > > > > 13:16:30') > > > > > > > > > >> > > > > > > > > > > > > >> > > > What does that mean? > > > > > > > > > >> > > > > > > > > > > > > >> > > > Thanks, > > > > > > > > > >> > > > Naama > > > > > > > > > >> > > > > > > > > > > > > >> > > > On Mon, Jan 23, 2017 at 5:27 PM, Dave Neuman < > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > >> > wrote: > > > > > > > > > >> > > > > > > > > > > > > >> > > > > Do you have an agent in your job_agent table > with > > > the > > > > id > > > > > > of > > > > > > > 1? > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > On Mon, Jan 23, 2017 at 5:42 AM, Naama Shoresh < > > > > > > > > > [email protected]> > > > > > > > > > >> > > wrote: > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > Hi, > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > When trying to invalidate content from the TO > > UI, > > > I > > > > > get > > > > > > > the > > > > > > > > > >> > following > > > > > > > > > >> > > > > fatal > > > > > > > > > >> > > > > > error message: > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > Traffic Ops fatal error occurred while > > processing > > > > your > > > > > > > > > request. > > > > > > > > > >> > > > > > ------------------------------ > > > > > > > > > >> > > > > > Error at line 146 ( my $insert = > > > > > > > > $self->db->resultset('Job')-> > > > > > > > > > >> > > create() > > > > > > > > > >> > > > > > ------------------------------ > > > > > > > > > >> > > > > > DBIx::Class::Storage::DBI::_dbh_execute(): > DBI > > > > > > Exception: > > > > > > > > > >> > > > DBD::mysql::st > > > > > > > > > >> > > > > > execute failed: Cannot add or update a child > > row: > > > a > > > > > > > foreign > > > > > > > > > key > > > > > > > > > >> > > > > constraint > > > > > > > > > >> > > > > > fails (`traffic_ops_db`.`job`, CONSTRAINT > > > > > > > `fk_job_agent_id1` > > > > > > > > > >> > FOREIGN > > > > > > > > > >> > > > KEY > > > > > > > > > >> > > > > > (`agent`) REFERENCES `job_agent` (`id`) ON > > DELETE > > > > > > CASCADE > > > > > > > ON > > > > > > > > > >> UPDATE > > > > > > > > > >> > > NO > > > > > > > > > >> > > > > > ACTION) [for Statement "INSERT INTO job ( > agent, > > > > > > > asset_type, > > > > > > > > > >> > > asset_url, > > > > > > > > > >> > > > > > entered_time, job_deliveryservice, job_user, > > > > keyword, > > > > > > > > > >> object_name, > > > > > > > > > >> > > > > > object_type, parameters, start_time, status) > > > VALUES > > > > ( > > > > > ?, > > > > > > > ?, > > > > > > > > ?, > > > > > > > > > >> ?, > > > > > > > > > >> > ?, > > > > > > > > > >> > > ?, > > > > > > > > > >> > > > > ?, > > > > > > > > > >> > > > > > ?, ?, ?, ?, ? )" with ParamValues: 0=1, > > 1="file", > > > > 2=" > > > > > > > > > >> > > > > > http://s2.ipcamlive.com/.*/ > > > > > 7464537492280autoresize.jpg > > > > > > ", > > > > > > > > > >> > > 3='2017-01-22 > > > > > > > > > >> > > > > > 13:04:53', 4='312', 5='59', 6="PURGE", > 7=undef, > > > > > 8=undef, > > > > > > > > > >> > 9="TTL:48h", > > > > > > > > > >> > > > > > 10='2017-01-22 13:05:22', 11=1] at > > > > > > > > > /opt/traffic_ops/app/lib/UI/ > > > > > > > > > >> > > Job.pm > > > > > > > > > >> > > > > line > > > > > > > > > >> > > > > > 146 > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > In traffic_ops log file I see this message: > > > > > > > > > >> > > > > > [2017-01-23 12:31:47,834] [ERROR] > > > > > > > > DBIx::Class::Storage::DBI::_ > > > > > > > > > >> > > > > > dbh_execute(): > > > > > > > > > >> > > > > > DBI Exception: DBD::mysql::st execute failed: > > > Cannot > > > > > add > > > > > > > or > > > > > > > > > >> update > > > > > > > > > >> > a > > > > > > > > > >> > > > > child > > > > > > > > > >> > > > > > row: a foreign key constraint fails > > > > > > > (`traffic_ops_db`.`job`, > > > > > > > > > >> > > CONSTRAINT > > > > > > > > > >> > > > > > `fk_job_agent_id1` FOREIGN KEY (`agent`) > > > REFERENCES > > > > > > > > > `job_agent` > > > > > > > > > >> > > (`id`) > > > > > > > > > >> > > > ON > > > > > > > > > >> > > > > > DELETE CASCADE ON UPDATE NO ACTION) [for > > Statement > > > > > > "INSERT > > > > > > > > > INTO > > > > > > > > > >> > job ( > > > > > > > > > >> > > > > > agent, asset_type, asset_url, entered_time, > > > > > > > > > job_deliveryservice, > > > > > > > > > >> > > > > job_user, > > > > > > > > > >> > > > > > keyword, object_name, object_type, parameters, > > > > > > start_time, > > > > > > > > > >> status) > > > > > > > > > >> > > > > VALUES ( > > > > > > > > > >> > > > > > ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ? )" with > > > > > ParamValues: > > > > > > > 0=1, > > > > > > > > > >> > > 1="file", > > > > > > > > > >> > > > > 2=" > > > > > > > > > >> > > > > > http://images1.ynet.co.il/foo/.*";, > > 3='2017-01-23 > > > > > > > 12:31:47', > > > > > > > > > >> > 4='311', > > > > > > > > > >> > > > > > 5='59', 6="PURGE", 7=undef, 8=undef, > > 9="TTL:48h", > > > > > > > > > 10='2017-01-23 > > > > > > > > > >> > > > > 12:32:22', > > > > > > > > > >> > > > > > 11=1] at /opt/traffic_ops/app/lib/UI/Job.pm > > line > > > > 146 > > > > > > > > > >> > > > > > [2017-01-23 12:31:47,835] [ERROR] > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > Any idea why this happens? > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > Thanks, > > > > > > > > > >> > > > > > Naama > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > -- > > > > > > > > > >> > > > *Naama Shoresh* > > > > > > > > > >> > > > Qwilt | Work: +972-72-2221706 > <+972%2072-222-1706> | > > > > > Mobile: > > > > > > > > > >> +972-52-3401999 <+972%2052-340-1999> | > > > > > > > > > >> > > > [email protected] > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > -- > > > > > > > > > >> > *Naama Shoresh* > > > > > > > > > >> > Qwilt | Work: +972-72-2221706 | Mobile: > +972-52-3401999 > > | > > > > > > > > > >> > [email protected] > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> -- > > > > > > > > > >> > > > > > > > > > >> *Oren Shemesh* > > > > > > > > > >> Qwilt | Work: +972-72-2221637| Mobile: +972-50-2281168 > | > > > > > > > > > [email protected] > > > > > > > > > >> <[email protected]> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > *Naama Shoresh* > > > > > > > > > > Qwilt | Work: +972-72-2221706 <+972%2072-222-1706> | > > Mobile: > > > > > > > > > > +972-52-3401999 <+972%2052-340-1999> | [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > *Naama Shoresh* > > > > > > > > > Qwilt | Work: +972-72-2221706 | Mobile: +972-52-3401999 | > > > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > *Naama Shoresh* > > > > > > > Qwilt | Work: +972-72-2221706 | Mobile: +972-52-3401999 | > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > *Oren Shemesh* > > > > > Qwilt | Work: +972-72-2221637| Mobile: +972-50-2281168 | > > > [email protected] > > > > > <[email protected]> > > > > > > > > > > > > > > > > > > -- > > *Oren Shemesh* > Qwilt | Work: +972-72-2221637| Mobile: +972-50-2281168 | [email protected] > <[email protected]> >
