Re: [foreman-users] 500 Internal Server Error when subscribing hosts

[Justin Sherrill]

On 09/14/2016 10:17 AM, James Evans wrote:
I ran foreman-rake katello:reimport as that seems to be the 
replacement for the functionality of katello:reindex, restarted all 
katello services, and tried again. It's still failing. I've attached 
the requested log file.


I recognize an error:


Error opening CA Certificate /etc/pki/pulp/ca.crt
140466849802144:error:02001002:system library:fopen:No such file or 
directory:bss_file.c:398:fopen('/etc/pki/pulp/ca.crt','r')
 140466849802144:error:20074002:BIO routines:FILE_CTRL:system 
lib:bss_file.c:400:

 unable to load certificate
 unable to write 'random state'

 can you try running:

pulp-gen-ca-certificate
katello-service restart


-Justin



Thanks,

james

On Tuesday, September 13, 2016 at 4:46:24 PM UTC-5, John Mitsch wrote:

James,

Try a couple things first:

foreman-rake katello:reindex
systemctl restart tomcat

then
 'tail -f -n 0 /var/log/candlepin/* /var/log/messages/
/var/log/foreman/production.log' on your machine and retry the
activation key.
Share that output with us (if its still erroring). It will be best
to pipe this to a file and attach it as it makes it easier to parse.

Let me know if you have any questions.

Thanks,

John Mitsch
Red Hat Engineering
(860)-967-7285
irc: jomitsch

On Tue, Sep 13, 2016 at 4:54 PM, James Evans
 wrote:

I've been prototyping Foreman for a while now, and everything
has been working perfectly in testing. Now I've built out our
production deployment and imported content, and started trying
to move hosts to the Foreman and Katello server. When ever I
run subscription-manager register, from any host, I get 500
Internal Server Error, and the task hangs. I have to go into
the dynaflow console and skip the broken step, and resume the
task to clear locks.

I've never had this happen in any of my tests where I've built
a number of test servers to test out processes and
configurations. All the test systems were able to register
clients with no issues at all. The server and client are both
fully updated RHEL 7.2 boxes, the server is
running foreman-1.12.2-1.el7.noarch
and katello-3.1.0-5.el7.noarch.

The only other difference between the production and test is
that I never imported RHEL content into the test systems.

I've spent a week building out the products, repositories,
content views, etc. and would really like to recover this
without having to start over, but this is obviously a major
problem!

I've googled the errors, with out finding any thing. Can
anyone help me?

On the client I run:

# subscription-manager register --org="myOrg"
--activationkey="rhel-72-x86_64-production"
Task 1ac48d97-a570-4a43-9a09-b5cb0be7cf29:
RestClient::InternalServerError: 500 Internal Server Error

And the production.log contains (abbreviated):

2016-09-13 15:39:15 [app] [I] Started GET "/rhsm/" for
XX.XX.XX.XX at 2016-09-13 15:39:15 -0500
2016-09-13 15:39:15 [app] [I] Processing by
Katello::Api::V2::RootController#rhsm_resource_list as JSON
2016-09-13 15:39:15 [app] [I]   Parameters: {"root"=>{}}
2016-09-13 15:39:15 [app] [I]   Rendered

/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0.1/app/views/katello/api/v2/root/resource_list.json.rabl
within katello/api/v2/layouts/collection (2.4ms)
2016-09-13 15:39:15 [app] [I] Completed 200 OK in 21ms (Views:
3.8ms | ActiveRecord: 0.0ms)
2016-09-13 15:39:15 [app] [I] Started GET "/rhsm/status" for
XX.XX.XX.XX at 2016-09-13 15:39:15 -0500
2016-09-13 15:39:15 [app] [I] Processing by
Katello::Api::Rhsm::CandlepinProxiesController#server_status
as JSON
2016-09-13 15:39:15 [app] [I] Completed 200 OK in 308ms
(Views: 0.9ms | ActiveRecord: 0.0ms)
2016-09-13 15:39:16 [app] [I] Started POST
"/rhsm/consumers?owner=myOrg_keys=rhel-72-x86_64-production"
for XX.XX.XX.XX at 2016-09-13 15:39:16 -0500
2016-09-13 15:39:16 [app] [I] Processing by
Katello::Api::Rhsm::CandlepinProxiesController#consumer_activate
as JSON
2016-09-13 15:39:16 [app] [I]   Parameters:
{"facts"=>"[FILTERED]",
"installedProducts"=>[{"productName"=>"Red Hat Enterprise
Linux Server", "arch"=>"x86_64", "version"=>"7.2",
"productId"=>"69"}], "type"=>"system", "name"=>"client.com
", "contentTags"=>["rhel-7",
"rhel-7-server"], "owner"=>"myOrg",
"activation_keys"=>"rhel-72-x86_64-production"}
2016-09-13 15:39:18 [app] [I] Import facts for
'client.example.com ' completed.
Added: 0, Updated: 1, 

Re: [foreman-users] 500 Internal Server Error when subscribing hosts

[James Evans]

I ran foreman-rake katello:reimport as that seems to be the replacement for 
the functionality of katello:reindex, restarted all katello services, and 
tried again. It's still failing. I've attached the requested log file.

Thanks,

james

On Tuesday, September 13, 2016 at 4:46:24 PM UTC-5, John Mitsch wrote:
>
> James,
>
> Try a couple things first:
>
> foreman-rake katello:reindex
> systemctl restart tomcat
>
> then
>  'tail -f -n 0 /var/log/candlepin/* /var/log/messages/ 
> /var/log/foreman/production.log' on your machine and retry the activation 
> key.
> Share that output with us (if its still erroring). It will be best to pipe 
> this to a file and attach it as it makes it easier to parse.
>
> Let me know if you have any questions.
>
> Thanks,
>
> John Mitsch
> Red Hat Engineering
> (860)-967-7285
> irc: jomitsch
>
> On Tue, Sep 13, 2016 at 4:54 PM, James Evans  > wrote:
>
>> I've been prototyping Foreman for a while now, and everything has been 
>> working perfectly in testing. Now I've built out our production deployment 
>> and imported content, and started trying to move hosts to the Foreman and 
>> Katello server. When ever I run subscription-manager register, from any 
>> host, I get 500 Internal Server Error, and the task hangs. I have to go 
>> into the dynaflow console and skip the broken step, and resume the task to 
>> clear locks.
>>
>> I've never had this happen in any of my tests where I've built a number 
>> of test servers to test out processes and configurations. All the test 
>> systems were able to register clients with no issues at all. The server and 
>> client are both fully updated RHEL 7.2 boxes, the server is 
>> running foreman-1.12.2-1.el7.noarch and katello-3.1.0-5.el7.noarch.
>>
>> The only other difference between the production and test is that I never 
>> imported RHEL content into the test systems.
>>
>> I've spent a week building out the products, repositories, content views, 
>> etc. and would really like to recover this without having to start over, 
>> but this is obviously a major problem!
>>
>> I've googled the errors, with out finding any thing. Can anyone help me?
>>
>> On the client I run:
>>
>> # subscription-manager register --org="myOrg" 
>> --activationkey="rhel-72-x86_64-production"
>> Task 1ac48d97-a570-4a43-9a09-b5cb0be7cf29: 
>> RestClient::InternalServerError: 500 Internal Server Error
>>
>> And the production.log contains (abbreviated):
>>
>> 2016-09-13 15:39:15 [app] [I] Started GET "/rhsm/" for XX.XX.XX.XX at 
>> 2016-09-13 15:39:15 -0500
>> 2016-09-13 15:39:15 [app] [I] Processing by 
>> Katello::Api::V2::RootController#rhsm_resource_list as JSON
>> 2016-09-13 15:39:15 [app] [I]   Parameters: {"root"=>{}}
>> 2016-09-13 15:39:15 [app] [I]   Rendered 
>> /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0.1/app/views/katello/api/v2/root/resource_list.json.rabl
>>  
>> within katello/api/v2/layouts/collection (2.4ms)
>> 2016-09-13 15:39:15 [app] [I] Completed 200 OK in 21ms (Views: 3.8ms | 
>> ActiveRecord: 0.0ms)
>> 2016-09-13 15:39:15 [app] [I] Started GET "/rhsm/status" for XX.XX.XX.XX 
>> at 2016-09-13 15:39:15 -0500
>> 2016-09-13 15:39:15 [app] [I] Processing by 
>> Katello::Api::Rhsm::CandlepinProxiesController#server_status as JSON
>> 2016-09-13 15:39:15 [app] [I] Completed 200 OK in 308ms (Views: 0.9ms | 
>> ActiveRecord: 0.0ms)
>> 2016-09-13 15:39:16 [app] [I] Started POST 
>> "/rhsm/consumers?owner=myOrg_keys=rhel-72-x86_64-production" for 
>> XX.XX.XX.XX at 2016-09-13 15:39:16 -0500
>> 2016-09-13 15:39:16 [app] [I] Processing by 
>> Katello::Api::Rhsm::CandlepinProxiesController#consumer_activate as JSON
>> 2016-09-13 15:39:16 [app] [I]   Parameters: {"facts"=>"[FILTERED]", 
>> "installedProducts"=>[{"productName"=>"Red Hat Enterprise Linux Server", 
>> "arch"=>"x86_64", "version"=>"7.2", "productId"=>"69"}], "type"=>"system", 
>> "name"=>"client.com", "contentTags"=>["rhel-7", "rhel-7-server"], 
>> "owner"=>"myOrg", "activation_keys"=>"rhel-72-x86_64-production"}
>> 2016-09-13 15:39:18 [app] [I] Import facts for 'client.example.com' 
>> completed. Added: 0, Updated: 1, Deleted 21 facts
>> 2016-09-13 15:39:21 [katello/pulp_rest] [E] RestClient.post "
>> https://foreman.example.com/pulp/api/v2/consumers/;, 
>> "{\"id\":\"1c796228-efa5-4190-b5de-a0f7605e3074\",\"display_name\":\"
>> client.example.com\"}", "Accept"=>"*/*; q=0.5, application/xml", 
>> "Accept-Encoding"=>"gzip, deflate", "Content-Length"=>"100", 
>> "accept"=>"application/json", "content_type"=>"application/json"
>>  | \n# => 500 InternalServerError | application/json 2551 bytes
>>  | 
>> 2016-09-13 15:39:21 [foreman-tasks/action] [E] 500 Internal Server Error 
>> (RestClient::InternalServerError)
>>  | 
>> /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.8.0/lib/restclient/abstract_response.rb:74:in
>>  
>> `return!'
>>  | 
>> /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.7.2/lib/runcible/base.rb:79:in
>>  
>> `block in 

Re: [foreman-users] 500 Internal Server Error when subscribing hosts

[James Evans]

Thank you for the response. I've seen several references to 
katello:reindex, but I don't seem to have that task on my production 
machine:

# foreman-rake katello:reindex
rake aborted!
Don't know how to build task 'katello:reindex'

(See full trace by running task with --trace)

Interestingly enough, I do seem to have the task on my test server. This is 
pretty confusing as I scripted out the install on both machines, so they 
*should* be identical. 
A quick look at rpm -qVa doesn't so anything obvious, what package provides 
this task?

james

On Tuesday, September 13, 2016 at 4:46:24 PM UTC-5, John Mitsch wrote:
>
> James,
>
> Try a couple things first:
>
> foreman-rake katello:reindex
> systemctl restart tomcat
>
> then
>  'tail -f -n 0 /var/log/candlepin/* /var/log/messages/ 
> /var/log/foreman/production.log' on your machine and retry the activation 
> key.
> Share that output with us (if its still erroring). It will be best to pipe 
> this to a file and attach it as it makes it easier to parse.
>
> Let me know if you have any questions.
>
> Thanks,
>
> John Mitsch
> Red Hat Engineering
> (860)-967-7285
> irc: jomitsch
>
> On Tue, Sep 13, 2016 at 4:54 PM, James Evans  > wrote:
>
>> I've been prototyping Foreman for a while now, and everything has been 
>> working perfectly in testing. Now I've built out our production deployment 
>> and imported content, and started trying to move hosts to the Foreman and 
>> Katello server. When ever I run subscription-manager register, from any 
>> host, I get 500 Internal Server Error, and the task hangs. I have to go 
>> into the dynaflow console and skip the broken step, and resume the task to 
>> clear locks.
>>
>> I've never had this happen in any of my tests where I've built a number 
>> of test servers to test out processes and configurations. All the test 
>> systems were able to register clients with no issues at all. The server and 
>> client are both fully updated RHEL 7.2 boxes, the server is 
>> running foreman-1.12.2-1.el7.noarch and katello-3.1.0-5.el7.noarch.
>>
>> The only other difference between the production and test is that I never 
>> imported RHEL content into the test systems.
>>
>> I've spent a week building out the products, repositories, content views, 
>> etc. and would really like to recover this without having to start over, 
>> but this is obviously a major problem!
>>
>> I've googled the errors, with out finding any thing. Can anyone help me?
>>
>> On the client I run:
>>
>> # subscription-manager register --org="myOrg" 
>> --activationkey="rhel-72-x86_64-production"
>> Task 1ac48d97-a570-4a43-9a09-b5cb0be7cf29: 
>> RestClient::InternalServerError: 500 Internal Server Error
>>
>> And the production.log contains (abbreviated):
>>
>> 2016-09-13 15:39:15 [app] [I] Started GET "/rhsm/" for XX.XX.XX.XX at 
>> 2016-09-13 15:39:15 -0500
>> 2016-09-13 15:39:15 [app] [I] Processing by 
>> Katello::Api::V2::RootController#rhsm_resource_list as JSON
>> 2016-09-13 15:39:15 [app] [I]   Parameters: {"root"=>{}}
>> 2016-09-13 15:39:15 [app] [I]   Rendered 
>> /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0.1/app/views/katello/api/v2/root/resource_list.json.rabl
>>  
>> within katello/api/v2/layouts/collection (2.4ms)
>> 2016-09-13 15:39:15 [app] [I] Completed 200 OK in 21ms (Views: 3.8ms | 
>> ActiveRecord: 0.0ms)
>> 2016-09-13 15:39:15 [app] [I] Started GET "/rhsm/status" for XX.XX.XX.XX 
>> at 2016-09-13 15:39:15 -0500
>> 2016-09-13 15:39:15 [app] [I] Processing by 
>> Katello::Api::Rhsm::CandlepinProxiesController#server_status as JSON
>> 2016-09-13 15:39:15 [app] [I] Completed 200 OK in 308ms (Views: 0.9ms | 
>> ActiveRecord: 0.0ms)
>> 2016-09-13 15:39:16 [app] [I] Started POST 
>> "/rhsm/consumers?owner=myOrg_keys=rhel-72-x86_64-production" for 
>> XX.XX.XX.XX at 2016-09-13 15:39:16 -0500
>> 2016-09-13 15:39:16 [app] [I] Processing by 
>> Katello::Api::Rhsm::CandlepinProxiesController#consumer_activate as JSON
>> 2016-09-13 15:39:16 [app] [I]   Parameters: {"facts"=>"[FILTERED]", 
>> "installedProducts"=>[{"productName"=>"Red Hat Enterprise Linux Server", 
>> "arch"=>"x86_64", "version"=>"7.2", "productId"=>"69"}], "type"=>"system", 
>> "name"=>"client.com", "contentTags"=>["rhel-7", "rhel-7-server"], 
>> "owner"=>"myOrg", "activation_keys"=>"rhel-72-x86_64-production"}
>> 2016-09-13 15:39:18 [app] [I] Import facts for 'client.example.com' 
>> completed. Added: 0, Updated: 1, Deleted 21 facts
>> 2016-09-13 15:39:21 [katello/pulp_rest] [E] RestClient.post "
>> https://foreman.example.com/pulp/api/v2/consumers/;, 
>> "{\"id\":\"1c796228-efa5-4190-b5de-a0f7605e3074\",\"display_name\":\"
>> client.example.com\"}", "Accept"=>"*/*; q=0.5, application/xml", 
>> "Accept-Encoding"=>"gzip, deflate", "Content-Length"=>"100", 
>> "accept"=>"application/json", "content_type"=>"application/json"
>>  | \n# => 500 InternalServerError | application/json 2551 bytes
>>  | 
>> 2016-09-13 15:39:21 [foreman-tasks/action] [E] 

Re: [foreman-users] 500 Internal Server Error when subscribing hosts

[John Mitsch]

Just to avoid any confusion, that tail command is going to be on your main
katello server

Thanks,

John Mitsch
Red Hat Engineering
(860)-967-7285
irc: jomitsch

On Tue, Sep 13, 2016 at 5:46 PM, John Mitsch  wrote:

> James,
>
> Try a couple things first:
>
> foreman-rake katello:reindex
> systemctl restart tomcat
>
> then
>  'tail -f -n 0 /var/log/candlepin/* /var/log/messages/
> /var/log/foreman/production.log' on your machine and retry the activation
> key.
> Share that output with us (if its still erroring). It will be best to pipe
> this to a file and attach it as it makes it easier to parse.
>
> Let me know if you have any questions.
>
> Thanks,
>
> John Mitsch
> Red Hat Engineering
> (860)-967-7285
> irc: jomitsch
>
> On Tue, Sep 13, 2016 at 4:54 PM, James Evans 
> wrote:
>
>> I've been prototyping Foreman for a while now, and everything has been
>> working perfectly in testing. Now I've built out our production deployment
>> and imported content, and started trying to move hosts to the Foreman and
>> Katello server. When ever I run subscription-manager register, from any
>> host, I get 500 Internal Server Error, and the task hangs. I have to go
>> into the dynaflow console and skip the broken step, and resume the task to
>> clear locks.
>>
>> I've never had this happen in any of my tests where I've built a number
>> of test servers to test out processes and configurations. All the test
>> systems were able to register clients with no issues at all. The server and
>> client are both fully updated RHEL 7.2 boxes, the server is
>> running foreman-1.12.2-1.el7.noarch and katello-3.1.0-5.el7.noarch.
>>
>> The only other difference between the production and test is that I never
>> imported RHEL content into the test systems.
>>
>> I've spent a week building out the products, repositories, content views,
>> etc. and would really like to recover this without having to start over,
>> but this is obviously a major problem!
>>
>> I've googled the errors, with out finding any thing. Can anyone help me?
>>
>> On the client I run:
>>
>> # subscription-manager register --org="myOrg"
>> --activationkey="rhel-72-x86_64-production"
>> Task 1ac48d97-a570-4a43-9a09-b5cb0be7cf29: RestClient::InternalServerError:
>> 500 Internal Server Error
>>
>> And the production.log contains (abbreviated):
>>
>> 2016-09-13 15:39:15 [app] [I] Started GET "/rhsm/" for XX.XX.XX.XX at
>> 2016-09-13 15:39:15 -0500
>> 2016-09-13 15:39:15 [app] [I] Processing by 
>> Katello::Api::V2::RootController#rhsm_resource_list
>> as JSON
>> 2016-09-13 15:39:15 [app] [I]   Parameters: {"root"=>{}}
>> 2016-09-13 15:39:15 [app] [I]   Rendered /opt/theforeman/tfm/root/usr/s
>> hare/gems/gems/katello-3.1.0.1/app/views/katello/api/v2/root/resource_list.json.rabl
>> within katello/api/v2/layouts/collection (2.4ms)
>> 2016-09-13 15:39:15 [app] [I] Completed 200 OK in 21ms (Views: 3.8ms |
>> ActiveRecord: 0.0ms)
>> 2016-09-13 15:39:15 [app] [I] Started GET "/rhsm/status" for XX.XX.XX.XX
>> at 2016-09-13 15:39:15 -0500
>> 2016-09-13 15:39:15 [app] [I] Processing by 
>> Katello::Api::Rhsm::CandlepinProxiesController#server_status
>> as JSON
>> 2016-09-13 15:39:15 [app] [I] Completed 200 OK in 308ms (Views: 0.9ms |
>> ActiveRecord: 0.0ms)
>> 2016-09-13 15:39:16 [app] [I] Started POST "/rhsm/consumers?owner=myOrg
>> ctivation_keys=rhel-72-x86_64-production" for XX.XX.XX.XX at 2016-09-13
>> 15:39:16 -0500
>> 2016-09-13 15:39:16 [app] [I] Processing by Katello::Api::Rhsm::CandlepinP
>> roxiesController#consumer_activate as JSON
>> 2016-09-13 15:39:16 [app] [I]   Parameters: {"facts"=>"[FILTERED]",
>> "installedProducts"=>[{"productName"=>"Red Hat Enterprise Linux Server",
>> "arch"=>"x86_64", "version"=>"7.2", "productId"=>"69"}], "type"=>"system",
>> "name"=>"client.com", "contentTags"=>["rhel-7", "rhel-7-server"],
>> "owner"=>"myOrg", "activation_keys"=>"rhel-72-x86_64-production"}
>> 2016-09-13 15:39:18 [app] [I] Import facts for 'client.example.com'
>> completed. Added: 0, Updated: 1, Deleted 21 facts
>> 2016-09-13 15:39:21 [katello/pulp_rest] [E] RestClient.post "
>> https://foreman.example.com/pulp/api/v2/consumers/;,
>> "{\"id\":\"1c796228-efa5-4190-b5de-a0f7605e3074\",\"display_name\":\"
>> client.example.com\"}", "Accept"=>"*/*; q=0.5, application/xml",
>> "Accept-Encoding"=>"gzip, deflate", "Content-Length"=>"100",
>> "accept"=>"application/json", "content_type"=>"application/json"
>>  | \n# => 500 InternalServerError | application/json 2551 bytes
>>  |
>> 2016-09-13 15:39:21 [foreman-tasks/action] [E] 500 Internal Server Error
>> (RestClient::InternalServerError)
>>  | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.8
>> .0/lib/restclient/abstract_response.rb:74:in `return!'
>>  | 
>> /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.7.2/lib/runcible/base.rb:79:in
>> `block in get_response'
>>  | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.8
>>