Re: [FW-1] R75.40 Dashboard giving Operation Error message
Finally got the chance to check on this, in fact cpca has over 1000 files opened, far more than any other process in the firewall. I still don't have a solution for the problem yet, so far have not find any documentation about the problem. Thanks again for the help David... anything else to suggest... anybody? On Fri, Sep 20, 2013 at 10:54 PM, David DeSimone f...@verio.net wrote: You can debug this using Linux debug techniques, since Splat and Gaia are just Linux under the hood. The message Too many open files either means that the process cpca has opened too many files and overflowed its local file table, or it means that some other process on the system has opened too many files and overflowed the global file table. You can see which processes have a large number of files open with and expert command like this: ls /proc/*/fd It should show a series of numbers for each process running, for example: /proc/522/fd: 0 1 2 3 4 5 6 7 8 9 That shows a process with only 10 files open. /proc/5196/fd: 0109 12 130 141 152 163 174 185 196 206 26 37 48 59 7 80 91 111 120 131 142 153 164 175 186 197 207 27 38 49 6 70 81 92 10 110 121 132 143 154 165 176 187 198 208 28 39 5 60 71 82 93 100 111 122 133 144 155 166 177 188 199 209 29 4 50 61 72 83 94 101 112 123 134 145 156 167 178 189 221 3 40 51 62 73 84 95 102 113 124 135 146 157 168 179 19 20 210 30 41 52 63 74 85 96 103 114 125 136 147 158 169 18 190 200 211 31 42 53 64 75 86 97 104 115 126 137 148 159 17 180 191 201 213 32 43 54 65 76 87 98 105 116 127 138 149 16 170 181 192 202 22 33 44 55 66 77 88 99 106 117 128 139 15 160 171 182 193 203 23 34 45 56 67 78 89 107 118 129 14 150 161 172 183 194 204 24 35 46 57 68 79 9 108 119 13 140 151 162 173 184 195 205 25 36 47 58 69 8 90 That shows a process that has 214 files open. If you see a process which has hundreds or thousands of files open, then you can find the name of the process with ps: [Expert@f020102]# ps 5196 PID TTY STAT TIME COMMAND 5196 ?Ssl 234:21 cpd So my cpd process in this example has 214 files open. If your system was really out of files in its global file table, it would probably be crashing pretty badly. So it seems more likely that your cpca process has forgotten to close some of its files and it eventually used up all of its file table entries. If so, then you probably could fix the situation by killing the cpca process to force it to restart, but that's getting into guesswork. :) Sergio Alvarez seral...@gmail.com wrote: Hello. This customer is running a stand alone appliance running SPLAT R75.40, a few weeks ago, noticed opening any objects' properties in the Dashboard was taking far longer than usual and from time to time an error message popped up: Operation Error. If the problem persists contact customer support. the issue went away with no changes done, and then returned a few days later. I checked the box for resource issues and even when CPU won't go higher than 40%, I noticed top shows cpca as consuming 100% of that 40%, I don't think I have seen something like this before, so looked in the $FWDIR/log directory and found a bunch of cpca.elg files (around 10), all of them are filled with hundreds of lines like this; SvcSk_new_connection: accept(12) failed: Too many open files SvcSk_new_connection: accept(11) failed: Too many open files SvcSk_new_connection: accept(12) failed: Too many open files SvcSk_new_connection: accept(11) failed: Too many open files SK search did not help but I did find an SK on how to troubleshoot cpca which basically says to run debugs like this: *fw debug cpca on TDERROR_ALL_ALL=5* * * *Which I did, but I don't get anything new in the elg files, just the same lines above.* A search in SK and even Google did not help. Only SK doc mentioning the exact error message seen in Dashboard regards a problem trying to remove an ICA cert from the gateway, which off course we are not doing here. Any ideas? All suggestions would be appreciated. -- David DeSimone == Network Admin == f...@verio.net I don't like spinach, and I'm glad I don't, because if I liked it I'd eat it, and I just hate it. -- Clarence Darrow This email message is intended for the use of the person to whom it has been sent, and may contain information that is confidential or legally protected. If you are not the intended recipient or have received this message in error, you are not authorized to copy, distribute, or otherwise use this message or its attachments. Please notify the sender immediately by return e-mail
Re: [FW-1] R75.40 Dashboard giving Operation Error message
Thank you David. On Fri, Sep 20, 2013 at 10:54 PM, David DeSimone f...@verio.net wrote: You can debug this using Linux debug techniques, since Splat and Gaia are just Linux under the hood. The message Too many open files either means that the process cpca has opened too many files and overflowed its local file table, or it means that some other process on the system has opened too many files and overflowed the global file table. You can see which processes have a large number of files open with and expert command like this: ls /proc/*/fd It should show a series of numbers for each process running, for example: /proc/522/fd: 0 1 2 3 4 5 6 7 8 9 That shows a process with only 10 files open. /proc/5196/fd: 0109 12 130 141 152 163 174 185 196 206 26 37 48 59 7 80 91 111 120 131 142 153 164 175 186 197 207 27 38 49 6 70 81 92 10 110 121 132 143 154 165 176 187 198 208 28 39 5 60 71 82 93 100 111 122 133 144 155 166 177 188 199 209 29 4 50 61 72 83 94 101 112 123 134 145 156 167 178 189 221 3 40 51 62 73 84 95 102 113 124 135 146 157 168 179 19 20 210 30 41 52 63 74 85 96 103 114 125 136 147 158 169 18 190 200 211 31 42 53 64 75 86 97 104 115 126 137 148 159 17 180 191 201 213 32 43 54 65 76 87 98 105 116 127 138 149 16 170 181 192 202 22 33 44 55 66 77 88 99 106 117 128 139 15 160 171 182 193 203 23 34 45 56 67 78 89 107 118 129 14 150 161 172 183 194 204 24 35 46 57 68 79 9 108 119 13 140 151 162 173 184 195 205 25 36 47 58 69 8 90 That shows a process that has 214 files open. If you see a process which has hundreds or thousands of files open, then you can find the name of the process with ps: [Expert@f020102]# ps 5196 PID TTY STAT TIME COMMAND 5196 ?Ssl 234:21 cpd So my cpd process in this example has 214 files open. If your system was really out of files in its global file table, it would probably be crashing pretty badly. So it seems more likely that your cpca process has forgotten to close some of its files and it eventually used up all of its file table entries. If so, then you probably could fix the situation by killing the cpca process to force it to restart, but that's getting into guesswork. :) Sergio Alvarez seral...@gmail.com wrote: Hello. This customer is running a stand alone appliance running SPLAT R75.40, a few weeks ago, noticed opening any objects' properties in the Dashboard was taking far longer than usual and from time to time an error message popped up: Operation Error. If the problem persists contact customer support. the issue went away with no changes done, and then returned a few days later. I checked the box for resource issues and even when CPU won't go higher than 40%, I noticed top shows cpca as consuming 100% of that 40%, I don't think I have seen something like this before, so looked in the $FWDIR/log directory and found a bunch of cpca.elg files (around 10), all of them are filled with hundreds of lines like this; SvcSk_new_connection: accept(12) failed: Too many open files SvcSk_new_connection: accept(11) failed: Too many open files SvcSk_new_connection: accept(12) failed: Too many open files SvcSk_new_connection: accept(11) failed: Too many open files SK search did not help but I did find an SK on how to troubleshoot cpca which basically says to run debugs like this: *fw debug cpca on TDERROR_ALL_ALL=5* * * *Which I did, but I don't get anything new in the elg files, just the same lines above.* A search in SK and even Google did not help. Only SK doc mentioning the exact error message seen in Dashboard regards a problem trying to remove an ICA cert from the gateway, which off course we are not doing here. Any ideas? All suggestions would be appreciated. -- David DeSimone == Network Admin == f...@verio.net I don't like spinach, and I'm glad I don't, because if I liked it I'd eat it, and I just hate it. -- Clarence Darrow This email message is intended for the use of the person to whom it has been sent, and may contain information that is confidential or legally protected. If you are not the intended recipient or have received this message in error, you are not authorized to copy, distribute, or otherwise use this message or its attachments. Please notify the sender immediately by return e-mail and permanently delete this message and any attachments. Verio Inc. makes no warranty that this email is error or virus free. Thank you. Email secured by Check Point = To set vacation, Out-Of-Office, or away messages, send an email to
Re: [FW-1] R75.40 Dashboard giving Operation Error message
You can debug this using Linux debug techniques, since Splat and Gaia are just Linux under the hood. The message Too many open files either means that the process cpca has opened too many files and overflowed its local file table, or it means that some other process on the system has opened too many files and overflowed the global file table. You can see which processes have a large number of files open with and expert command like this: ls /proc/*/fd It should show a series of numbers for each process running, for example: /proc/522/fd: 0 1 2 3 4 5 6 7 8 9 That shows a process with only 10 files open. /proc/5196/fd: 0109 12 130 141 152 163 174 185 196 206 26 37 48 59 7 80 91 111 120 131 142 153 164 175 186 197 207 27 38 49 6 70 81 92 10 110 121 132 143 154 165 176 187 198 208 28 39 5 60 71 82 93 100 111 122 133 144 155 166 177 188 199 209 29 4 50 61 72 83 94 101 112 123 134 145 156 167 178 189 221 3 40 51 62 73 84 95 102 113 124 135 146 157 168 179 19 20 210 30 41 52 63 74 85 96 103 114 125 136 147 158 169 18 190 200 211 31 42 53 64 75 86 97 104 115 126 137 148 159 17 180 191 201 213 32 43 54 65 76 87 98 105 116 127 138 149 16 170 181 192 202 22 33 44 55 66 77 88 99 106 117 128 139 15 160 171 182 193 203 23 34 45 56 67 78 89 107 118 129 14 150 161 172 183 194 204 24 35 46 57 68 79 9 108 119 13 140 151 162 173 184 195 205 25 36 47 58 69 8 90 That shows a process that has 214 files open. If you see a process which has hundreds or thousands of files open, then you can find the name of the process with ps: [Expert@f020102]# ps 5196 PID TTY STAT TIME COMMAND 5196 ?Ssl 234:21 cpd So my cpd process in this example has 214 files open. If your system was really out of files in its global file table, it would probably be crashing pretty badly. So it seems more likely that your cpca process has forgotten to close some of its files and it eventually used up all of its file table entries. If so, then you probably could fix the situation by killing the cpca process to force it to restart, but that's getting into guesswork. :) Sergio Alvarez seral...@gmail.com wrote: Hello. This customer is running a stand alone appliance running SPLAT R75.40, a few weeks ago, noticed opening any objects' properties in the Dashboard was taking far longer than usual and from time to time an error message popped up: Operation Error. If the problem persists contact customer support. the issue went away with no changes done, and then returned a few days later. I checked the box for resource issues and even when CPU won't go higher than 40%, I noticed top shows cpca as consuming 100% of that 40%, I don't think I have seen something like this before, so looked in the $FWDIR/log directory and found a bunch of cpca.elg files (around 10), all of them are filled with hundreds of lines like this; SvcSk_new_connection: accept(12) failed: Too many open files SvcSk_new_connection: accept(11) failed: Too many open files SvcSk_new_connection: accept(12) failed: Too many open files SvcSk_new_connection: accept(11) failed: Too many open files SK search did not help but I did find an SK on how to troubleshoot cpca which basically says to run debugs like this: *fw debug cpca on TDERROR_ALL_ALL=5* * * *Which I did, but I don't get anything new in the elg files, just the same lines above.* A search in SK and even Google did not help. Only SK doc mentioning the exact error message seen in Dashboard regards a problem trying to remove an ICA cert from the gateway, which off course we are not doing here. Any ideas? All suggestions would be appreciated. -- David DeSimone == Network Admin == f...@verio.net I don't like spinach, and I'm glad I don't, because if I liked it I'd eat it, and I just hate it. -- Clarence Darrow This email message is intended for the use of the person to whom it has been sent, and may contain information that is confidential or legally protected. If you are not the intended recipient or have received this message in error, you are not authorized to copy, distribute, or otherwise use this message or its attachments. Please notify the sender immediately by return e-mail and permanently delete this message and any attachments. Verio Inc. makes no warranty that this email is error or virus free. Thank you. Email secured by Check Point = To set vacation, Out-Of-Office, or away messages, send an email to lists...@amadeus.us.checkpoint.com in the BODY of the email add: set fw-1-mailinglist nomail = To unsubscribe from this mailing
