[foreman-users] 2.4-3.0 upgrade stuck

[sinux shen]

Hey guys,

when I upgrade katello from 2.4 to 3.0, I found it stuck at 
"update_puppet_repository_distributors (this may take a while)"
Upgrade Step: import_puppet_modules (this may take a while) ...
Upgrade Step: import_subscriptions (this may take a while) ...
Upgrade Step: elasticsearch_message...
Elasticsearch has been removed as a dependency.  The database files can be 
removed manually with #rm -rf /var/lib/elasticsearch.  Some packages are no 
longer needed and can be removed:  #rpm -e ruby193-rubygem-tire 
tfm-rubygem-tire elasticsearch sigar-java sigar snappy-java lucene4-contrib 
lucene4
Upgrade Step: add_export_distributor (this may take a while) ...
Upgrade Step: remove_docker_v1_content (this may take a while) ...
Upgrade Step: update_puppet_repository_distributors (this may take a while) 
...

the whole system still works though, but I don't know if any impact it will 
has, is there any way that we can manually run to trouble shoot or proceed.?

Thanks
-Sinux

-- 
You received this message because you are subscribed to the Google Groups 
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-users+unsubscr...@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

[foreman-users] can not download the RHEL repos

[sinux shen]

Hey there,

I have exported and uploaded the RHEL subscriptions to foreman and make it 
as an 3rd party SAM, now I can see all the RHEL products, I have done this 
before by using our previous RHEL subscriptions and can download RHEL 
6server repos successfully, after our subscription expire, (I would say, I 
didn't refresh the subscriptions in Katello/foreman), I deleted these RHEL 
6server repos by unchecking them from "RedHat repositories" menu, and now, 
after I refreshed our subscription, I want to enable these repos again, but 
after I enabled them, I cannot sync/download them, it returned with "no new 
packages", is there anything wrong with the settings? Where I can check the 
log?

Any help is appreciated.

Thanks
-Sinux

-- 
You received this message because you are subscribed to the Google Groups 
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-users+unsubscr...@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

[foreman-users] Re: can not download the RHEL repos

[sinux shen]

The product I was trying to sync was "Red Hat Enterprise Linux 6 Server 
RPMs x86_64 6Server" which is one of the repos from RHEL6 product, it 
always ended up with "No New Packages", but when I was syncing another repo 
"Red Hat Enterprise Linux 6 Server - Optional RPMs x86_64 6Server", I can 
sync all the contents though.

On Friday, February 10, 2017 at 11:26:28 AM UTC+8, sinux shen wrote:
>
> Hey there,
>
> I have exported and uploaded the RHEL subscriptions to foreman and make it 
> as an 3rd party SAM, now I can see all the RHEL products, I have done this 
> before by using our previous RHEL subscriptions and can download RHEL 
> 6server repos successfully, after our subscription expire, (I would say, I 
> didn't refresh the subscriptions in Katello/foreman), I deleted these RHEL 
> 6server repos by unchecking them from "RedHat repositories" menu, and now, 
> after I refreshed our subscription, I want to enable these repos again, but 
> after I enabled them, I cannot sync/download them, it returned with "no new 
> packages", is there anything wrong with the settings? Where I can check the 
> log?
>
> Any help is appreciated.
>
> Thanks
> -Sinux
>

-- 
You received this message because you are subscribed to the Google Groups 
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-users+unsubscr...@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

[foreman-users] search host very slow in foreman

[sinux shen]

Hey guys,

I have been suffered for this for quite a while, we are using katello 3.0 
with foreman 1.11.3, we have around 400+ hosts added in the system, but 
every time when I search a specific host in the search bar, it response me 
very very slow, still work though, just took me like 30 secs or even 
minutes to get the result, I suspect it was something wrong with elastic 
search or something else, is there any way that we can trouble shoot or 
some system tuning we can do to improve this.

Thanks
-Sinux

-- 
You received this message because you are subscribed to the Google Groups 
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-users+unsubscr...@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-users] search host very slow in foreman

[sinux shen]

we actually did what you suggested, it was pretty much like the following 
two join that caused the problem:

> LEFT OUTER JOIN "katello_installed_packages" ON 
 "katello_installed_packages"."id" 
= "katello_host_installed_packages"."installed_package_id" 
> LEFT OUTER JOIN "katello_content_facets" 
ON "katello_content_facets"."host_id" = "hosts"."id" 

each table has more than 500,000+ rows of records. I will try to get the 
output when I get back to work tomorrow.

Thanks again for the help.

On Wednesday, September 21, 2016 at 9:47:02 PM UTC+8, Chris Duryee wrote:
>
>
>
> On 09/21/2016 09:24 AM, sinux shen wrote: 
> > Hi, Lukas & Ohad, 
> > 
> > Thanks for the help, we do turned the slow log on and we found the heavy 
> > SQL query while search a specific host and now we realized that we 
> should 
> > use auto completer to make the query more smart and lite, and here is 
> the 
> > SQL that almost join all the other tables that was associated to that 
> > specific host, and in two of the table (katello_host_installed_packages 
> and 
> > katello_installed_packages), we have 500,000+ rows for each, and this 
> > caused heavy load and slow query, just FYI: 
> > 
> > SELECT DISTINCT "hosts".id FROM "hosts" 
> > LEFT OUTER JOIN "models" ON "models"."id" = "hosts"."model_id" 
> > LEFT OUTER JOIN "hostgroups" ON "hostgroups"."id" = 
> "hosts"."hostgroup_id" 
> > AND "hostgroups"."id" IN (14, 27, 19, 1, 2, 21, 9, 3, 10, 23, 11, 4, 18, 
> 22) 
> > LEFT OUTER JOIN "nics" ON "nics"."host_id" = "hosts"."id" AND 
> > "nics"."primary" = 't' 
> > LEFT OUTER JOIN "domains" ON "domains"."id" = "nics"."domain_id" AND 
> > "domains"."id" IN (1) 
> > LEFT OUTER JOIN "realms" ON "realms"."id" = "hosts"."realm_id" AND 1=0 
> > LEFT OUTER JOIN "environments" ON "environments"."id" = 
> > "hosts"."environment_id" AND "environments"."id" IN (2, 3, 4, 8, 5) 
> > LEFT OUTER JOIN "architectures" ON "architectures"."id" = 
> > "hosts"."architecture_id" 
> > LEFT OUTER JOIN "compute_resources" ON "compute_resources"."id" = 
> > "hosts"."compute_resource_id" AND 1=0 
> > LEFT OUTER JOIN "images" ON "images"."id" = "hosts"."image_id" 
> > LEFT OUTER JOIN "operatingsystems" ON "operatingsystems"."id" = 
> > "hosts"."operatingsystem_id" 
> > LEFT OUTER JOIN "nics" "primary_interfaces_hosts" ON 
> > "primary_interfaces_hosts"."host_id" = "hosts"."id" AND 
> > "primary_interfaces_hosts"."primary" = 't' 
> > LEFT OUTER JOIN "nics" "interfaces_hosts" ON 
> "interfaces_hosts"."host_id" = 
> > "hosts"."id" 
> > LEFT OUTER JOIN "taxonomies" ON "taxonomies"."id" = 
> "hosts"."location_id" 
> > AND "taxonomies"."type" IN ('Location') 
> > LEFT OUTER JOIN "nics" "primary_interfaces_hosts_join" ON 
> > "primary_interfaces_hosts_join"."host_id" = "hosts"."id" AND 
> > "primary_interfaces_hosts_join"."primary" = 't' 
> > LEFT OUTER JOIN "subnets" ON "subnets"."id" = 
> > "primary_interfaces_hosts_join"."subnet_id" AND "subnets"."id" IN (5, 2, 
> 1) 
> > LEFT OUTER JOIN "nics" "provision_interfaces_hosts" ON 
> > "provision_interfaces_hosts"."host_id" = "hosts"."id" AND 
> > "provision_interfaces_hosts"."provision" = 't' 
> > LEFT OUTER JOIN "smart_proxies" ON "smart_proxies"."id" = 
> > "hosts"."content_source_id" AND "smart_proxies"."id" IN (7, 11, 1) 
> > LEFT OUTER JOIN "katello_host_collection_hosts" ON 
> > "katello_host_collection_hosts"."host_id" = "hosts"."id" 
> > LEFT OUTER

Re: [foreman-users] search host very slow in foreman

[sinux shen]

Hi ohad & Lukas,

I checked the database, when I do a host query, one of the postgres thread 
used 100% of CPU, and I am surprised that the sql has so many join:
SELECT DISTINCT "hosts"."id", "hosts"."name" AS alias_0 FROM "hosts" \
 LEFT OUTER JOIN "models" ON "models"."id" = "hosts"."model_id" \
LEFT OUTER JOIN "hostgroups" ON "hostgroups"."id" = "hosts"."hostgroup_id" 
AND "hostgroups"."id" IN (14, 27, 19, 1, 2, 21, 9, 3, 10, 23, 11, 4, 18, 
22) \
LEFT OUTER JOIN "nics" ON "nics"."host_id" = "hosts"."id" AND 
"nics"."primary" = 't' \
LEFT OUTER JOIN "domains" ON "domains"."id" = "nics"."domain_id" AND 
"domains"."id" IN (1) \
LEFT OUTER JOIN "realms" ON "realms"."id" = "hosts"."realm_id" AND 1=0 \
LEFT OUTER JOIN "environments" ON "environments"."id" = 
"hosts"."environment_id" AND "environments"."id" IN (2, 3, 4, 8, 5) \
LEFT OUTER JOIN "architectures" ON "architectures"."id" = 
"hosts"."architecture_id" \
LEFT OUTER JOIN "compute_resources" ON "compute_resources"."id" = 
"hosts"."compute_resource_id" AND 1=0 \
LEFT OUTER JOIN "images" ON "images"."id" = "hosts"."image_id" \
LEFT OUTER JOIN "operatingsystems" ON "operatingsystems"."id" = 
"hosts"."operatingsystem_id" \
LEFT OUTER JOIN "nics"

I think we probably need to tune the sql so it can run a little faster


On Tuesday, September 20, 2016 at 8:16:12 AM UTC+8, sinux shen wrote:
>
> I just simply search a specific hostname in the search bar, for example 
> “foo”, and then have to wait for a long time, normally 20~30 secs, I will 
> try to turn the slow query on and see what happened. 
>
> Thanks 
> -Sinux 
> > On Sep 14, 2016, at 3:59 PM, Lukas Zapletal <l...@redhat.com> wrote: 
> > 
> >> what was the search query that you used? 
> > 
> > And do you experience this when loading "intelligent completion" or when 
> > doing actual query? 
> > 
> > Are you able to enable slow queries logging in your postgres and send us 
> > the output? 
> > 
> > https://wiki.postgresql.org/wiki/Logging_Difficult_Queries 
> > 
> > Easiest is to set log_min_duration_statement to reasonable value (1 
> > second). 
> > 
> > -- 
> > Later, 
> > Lukas #lzap Zapletal 
> > 
> > -- 
> > You received this message because you are subscribed to the Google 
> Groups "Foreman users" group. 
> > To unsubscribe from this group and stop receiving emails from it, send 
> an email to foreman-users+unsubscr...@googlegroups.com. 
> > To post to this group, send email to foreman-users@googlegroups.com. 
> > Visit this group at https://groups.google.com/group/foreman-users. 
> > For more options, visit https://groups.google.com/d/optout. 
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-users+unsubscr...@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-users] search host very slow in foreman

[sinux shen]

 Scan on katello_host_collection_hosts  (cost=4.30..14.45 
rows=7 width=8) (actual time=0.011..0.011 rows=0 loops=1)
Recheck Cond: (host_id = 934)
->  Bitmap Index Scan on index_katello_host_collection_hosts_on_host_id  
(cost=0.00..4.30 rows=7 width=0) (actual time=0.011..0.011 rows=0 loops=1)
Index Cond: (host_id = 934)
->  Hash  (cost=11.30..11.30 rows=130 width=577) (never executed)
->  Seq Scan on katello_host_collections  (cost=0.00..11.30 rows=130 
width=577) (never executed)
->  Materialize  (cost=16594.42..16617.17 rows=2275 width=53) (actual 
time=0.031..2.017 rows=2296 loops=6)
->  Nested Loop Left Join  (cost=53.96..16592.15 rows=2275 width=53) 
(actual time=0.181..7.464 rows=2296 loops=1)
->  Bitmap Heap Scan on katello_host_installed_packages  
(cost=53.96..2986.55 rows=2275 width=8) (actual time=0.172..0.713 rows=2296 
loops=1)
Recheck Cond: (host_id = 934)
->  Bitmap Index Scan on index_katello_host_installed_packages_on_host_id  
(cost=0.00..53.39 rows=2275 width=0) (actual time=0.163..0.163 rows=2350 
loops=1)
Index Cond: (host_id = 934)
->  Index Scan using katello_installed_packages_pkey on 
katello_installed_packages  (cost=0.00..5.97 rows=1 width=49) (actual 
time=0.002..0.002 rows=1 loops=2296)
Index Cond: (katello_installed_packages.id = 
katello_host_installed_packages.installed_package_id)
->  Materialize  (cost=16.55..16.56 rows=1 width=1093) (actual 
time=0.000..0.000 rows=1 loops=13776)
->  Nested Loop Left Join  (cost=0.00..16.55 rows=1 width=1093) (actual 
time=0.008..0.009 rows=1 loops=1)
->  Index Scan using katello_content_facets_host_id on 
katello_content_facets content_facets_hosts_join  (cost=0.00..8.27 rows=1 
width=8) (actual time=0.003..0.004 rows=1 loops=1)
Index Cond: (host_id = 934)
->  Index Scan using katello_environments_pkey on katello_environments  
(cost=0.00..8.27 rows=1 width=1089) (actual time=0.003..0.003 rows=1 
loops=1)
Index Cond: (katello_environments.id = 
content_facets_hosts_join.lifecycle_environment_id)
->  Materialize  (cost=16.55..16.56 rows=1 width=1098) (actual 
time=0.000..0.000 rows=1 loops=13776)
->  Nested Loop Left Join  (cost=0.00..16.55 rows=1 width=1098) (actual 
time=0.006..0.007 rows=1 loops=1)
->  Index Scan using katello_content_facets_host_id on 
katello_content_facets  (cost=0.00..8.27 rows=1 width=8) (actual 
time=0.001..0.002 rows=1 loops=1)
Index Cond: (host_id = 934)
->  Index Scan using katello_content_views_pkey on katello_content_views  
(cost=0.00..8.27 rows=1 width=1094) (actual time=0.004..0.004 rows=1 
loops=1)
Index Cond: (katello_content_views.id = 
katello_content_facets.content_view_id)
->  Index Scan using katello_subscription_facets_host_id on 
katello_subscription_facets  (cost=0.00..0.27 rows=1 width=1094) (actual 
time=0.001..0.001 rows=1 loops=13776)
Index Cond: ((katello_subscription_facets.host_id = 934) AND 
(katello_subscription_facets.host_id = hosts.id))
->  Result  (cost=0.00..0.01 rows=1 width=0) (actual time=0.000..0.000 
rows=0 loops=13776)One-Time Filter: false
->  Index Scan using index_host_status_on_host_id on host_status  
(cost=0.00..0.34 rows=4 width=50) (actual time=0.001..0.002 rows=4 
loops=13776)
Index Cond: ((host_status.host_id = 934) AND (host_status.host_id = 
hosts.id))
Total runtime: 252.073 ms
(122 rows)

On Thursday, September 22, 2016 at 5:59:00 AM UTC+8, jsherril wrote:
>
> On 09/21/2016 10:48 AM, sinux shen wrote:
>
> we actually did what you suggested, it was pretty much like the following 
> two join that caused the problem: 
>
> > LEFT OUTER JOIN "katello_installed_packages" ON 
>  "katello_installed_packages"."id" 
> = "katello_host_installed_packages"."installed_package_id" 
> > LEFT OUTER JOIN "katello_content_facets" 
> ON "katello_content_facets"."host_id" = "hosts"."id" 
>
> each table has more than 500,000+ rows of records. I will try to get the 
> output when I get back to work tomorrow.
>
> Went ahead and opened an issue here:  
> http://projects.theforeman.org/issues/16647
> and a PR here: https://github.com/Katello/katello/pull/6338
>
> -Justin
>
> Thanks again for the help.
>
> On Wednesday, September 21, 2016 at 9:47:02 PM UTC+8, Chris Duryee wrote: 
>>
>>
>>
>> On 09/21/2016 09:24 AM, sinux shen wrote: 
>> > Hi, Lukas & Ohad, 
>> > 
>> > Thanks for the help, we do turned the slow log on and we found the 
>> heavy 
>> > SQL query while search a specific host and now we realized that we 
>> should 
>> > use auto completer to make the query more smart and lite, and here is 
>> the 
>> > SQL that almost join all the other tables that was associated to that 
>> > specific host, and in two of the table (katello_host_installed_packages 
>> and 
>> > kate

Re: [foreman-users] search host very slow in foreman

[sinux shen]

I just simply search a specific hostname in the search bar, for example “foo”, 
and then have to wait for a long time, normally 20~30 secs, I will try to turn 
the slow query on and see what happened.

Thanks
-Sinux
> On Sep 14, 2016, at 3:59 PM, Lukas Zapletal  wrote:
> 
>> what was the search query that you used?
> 
> And do you experience this when loading "intelligent completion" or when
> doing actual query?
> 
> Are you able to enable slow queries logging in your postgres and send us
> the output?
> 
> https://wiki.postgresql.org/wiki/Logging_Difficult_Queries
> 
> Easiest is to set log_min_duration_statement to reasonable value (1
> second).
> 
> -- 
> Later,
> Lukas #lzap Zapletal
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Foreman users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to foreman-users+unsubscr...@googlegroups.com.
> To post to this group, send email to foreman-users@googlegroups.com.
> Visit this group at https://groups.google.com/group/foreman-users.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-users+unsubscr...@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.