[jira] [Commented] (MESOS-8985) Posting to the operator api with 'accept recordio' header can crash the agent
[ https://issues.apache.org/jira/browse/MESOS-8985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16533496#comment-16533496 ] Alexander Rukletsov commented on MESOS-8985: [~bmahler] This does not look like a critical issue to me, hence no back port. > Posting to the operator api with 'accept recordio' header can crash the agent > - > > Key: MESOS-8985 > URL: https://issues.apache.org/jira/browse/MESOS-8985 > Project: Mesos > Issue Type: Bug > Components: HTTP API >Affects Versions: 1.4.1, 1.5.1 >Reporter: Philip Norman >Assignee: Benno Evers >Priority: Major > Labels: mesosphere > Fix For: 1.7.0 > > Attachments: mesos-slave-crash.log > > > It's possible to crash the mesos agent by posting a reasonable request to the > operator API. > h3. Background: > Sending a request to the v1 api endpoint with an unsupported 'accept' header: > {code:java} > curl -X POST http://10.0.3.27:5051/api/v1 \ > -H 'accept: application/atom+xml' \ > -H 'content-type: application/json' \ > -d '{"type":"GET_CONTAINERS","get_containers":{"show_nested": > true,"show_standalone": true}}'{code} > Results in the following friendly error message: > {code:java} > Expecting 'Accept' to allow application/json or application/x-protobuf or > application/recordio{code} > h3. Reproducible crash: > However, sending the same request with 'application/recordio' 'accept' header: > {code:java} > curl -X POST \ > http://10.0.3.27:5051/api/v1 \ > -H 'accept: application/recordio' \ > -H 'content-type: application/json' \ > -d '{"type":"GET_CONTAINERS","get_containers":{"show_nested": > true,"show_standalone": true}}'{code} > causes the agent to crash (no response is received). > Crash log is shown below, full log from the agent is attached here: > {code:java} > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: > I0607 22:30:32.397320 3743 logfmt.cpp:178] type=audit timestamp=2018-06-07 > 22:30:32.397243904+00:00 reason="Error in token 'Missing 'Authorization' > header from HTTP request'. Allowing anonymous connection" > object="/slave(1)/api/v1" agent="Mozilla/5.0 (Macintosh; Intel Mac OS X > 10_13_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 > Safari/537.36" authorizer="mesos-agent" action="POST" result=allow > srcip=10.0.6.99 dstport=5051 srcport=42084 dstip=10.0.3.27 > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: > W0607 22:30:32.397434 3743 authenticator.cpp:289] Error in token on request > from '10.0.6.99:42084': Missing 'Authorization' header from HTTP request > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: > W0607 22:30:32.397466 3743 authenticator.cpp:291] Falling back to anonymous > connection using user 'dcos_anonymous' > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: > I0607 22:30:32.397629 3748 http.cpp:1099] HTTP POST for /slave(1)/api/v1 from > 10.0.6.99:42084 with User-Agent='Mozilla/5.0 (Macintosh; Intel Mac OS X > 10_13_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 > Safari/537.36' > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: > I0607 22:30:32.397784 3748 http.cpp:2030] Processing GET_CONTAINERS call > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: > F0607 22:30:32.398736 3747 http.cpp:121] Serializing a RecordIO stream is not > supported > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: > *** Check failure stack trace: *** > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ > 0x7f619478636d google::LogMessage::Fail() > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ > 0x7f619478819d google::LogMessage::SendToLog() > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ > 0x7f6194785f5c google::LogMessage::Flush() > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ > 0x7f6194788a99 google::LogMessageFatal::~LogMessageFatal() > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ > 0x7f61935e2b9d mesos::internal::serialize() > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ > 0x7f6193a4c0ef > _ZNO6lambda12CallableOnceIFN7process6FutureINS1_4http8ResponseEEERKN4JSON5ArrayEEE10CallableFnIZNK5mesos8internal5slave4Http13getContainersERKNSD_5agent4CallENSD_11ContentTypeERK6OptionINS3_14authentication9PrincipalEEEUlRKNS2_IS7_EEE0_EclES9_ > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ > 0x7f6193a81d61 process::internal::thenf<>() > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal
[jira] [Commented] (MESOS-8985) Posting to the operator api with 'accept recordio' header can crash the agent
[ https://issues.apache.org/jira/browse/MESOS-8985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16532107#comment-16532107 ] Benjamin Mahler commented on MESOS-8985: [~bennoe] [~alexr] any reason not to backport to the supported releases? > Posting to the operator api with 'accept recordio' header can crash the agent > - > > Key: MESOS-8985 > URL: https://issues.apache.org/jira/browse/MESOS-8985 > Project: Mesos > Issue Type: Bug > Components: HTTP API >Affects Versions: 1.4.1, 1.5.1 >Reporter: Philip Norman >Assignee: Benno Evers >Priority: Major > Labels: mesosphere > Fix For: 1.7.0 > > Attachments: mesos-slave-crash.log > > > It's possible to crash the mesos agent by posting a reasonable request to the > operator API. > h3. Background: > Sending a request to the v1 api endpoint with an unsupported 'accept' header: > {code:java} > curl -X POST http://10.0.3.27:5051/api/v1 \ > -H 'accept: application/atom+xml' \ > -H 'content-type: application/json' \ > -d '{"type":"GET_CONTAINERS","get_containers":{"show_nested": > true,"show_standalone": true}}'{code} > Results in the following friendly error message: > {code:java} > Expecting 'Accept' to allow application/json or application/x-protobuf or > application/recordio{code} > h3. Reproducible crash: > However, sending the same request with 'application/recordio' 'accept' header: > {code:java} > curl -X POST \ > http://10.0.3.27:5051/api/v1 \ > -H 'accept: application/recordio' \ > -H 'content-type: application/json' \ > -d '{"type":"GET_CONTAINERS","get_containers":{"show_nested": > true,"show_standalone": true}}'{code} > causes the agent to crash (no response is received). > Crash log is shown below, full log from the agent is attached here: > {code:java} > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: > I0607 22:30:32.397320 3743 logfmt.cpp:178] type=audit timestamp=2018-06-07 > 22:30:32.397243904+00:00 reason="Error in token 'Missing 'Authorization' > header from HTTP request'. Allowing anonymous connection" > object="/slave(1)/api/v1" agent="Mozilla/5.0 (Macintosh; Intel Mac OS X > 10_13_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 > Safari/537.36" authorizer="mesos-agent" action="POST" result=allow > srcip=10.0.6.99 dstport=5051 srcport=42084 dstip=10.0.3.27 > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: > W0607 22:30:32.397434 3743 authenticator.cpp:289] Error in token on request > from '10.0.6.99:42084': Missing 'Authorization' header from HTTP request > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: > W0607 22:30:32.397466 3743 authenticator.cpp:291] Falling back to anonymous > connection using user 'dcos_anonymous' > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: > I0607 22:30:32.397629 3748 http.cpp:1099] HTTP POST for /slave(1)/api/v1 from > 10.0.6.99:42084 with User-Agent='Mozilla/5.0 (Macintosh; Intel Mac OS X > 10_13_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 > Safari/537.36' > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: > I0607 22:30:32.397784 3748 http.cpp:2030] Processing GET_CONTAINERS call > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: > F0607 22:30:32.398736 3747 http.cpp:121] Serializing a RecordIO stream is not > supported > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: > *** Check failure stack trace: *** > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ > 0x7f619478636d google::LogMessage::Fail() > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ > 0x7f619478819d google::LogMessage::SendToLog() > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ > 0x7f6194785f5c google::LogMessage::Flush() > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ > 0x7f6194788a99 google::LogMessageFatal::~LogMessageFatal() > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ > 0x7f61935e2b9d mesos::internal::serialize() > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ > 0x7f6193a4c0ef > _ZNO6lambda12CallableOnceIFN7process6FutureINS1_4http8ResponseEEERKN4JSON5ArrayEEE10CallableFnIZNK5mesos8internal5slave4Http13getContainersERKNSD_5agent4CallENSD_11ContentTypeERK6OptionINS3_14authentication9PrincipalEEEUlRKNS2_IS7_EEE0_EclES9_ > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ > 0x7f6193a81d61 process::internal::thenf<>() > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ >
[jira] [Commented] (MESOS-8985) Posting to the operator api with 'accept recordio' header can crash the agent
[ https://issues.apache.org/jira/browse/MESOS-8985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509812#comment-16509812 ] Benno Evers commented on MESOS-8985: Review: https://reviews.apache.org/r/67547/ > Posting to the operator api with 'accept recordio' header can crash the agent > - > > Key: MESOS-8985 > URL: https://issues.apache.org/jira/browse/MESOS-8985 > Project: Mesos > Issue Type: Bug >Affects Versions: 1.4.1, 1.5.1 >Reporter: Philip Norman >Assignee: Benno Evers >Priority: Major > Attachments: mesos-slave-crash.log > > > It's possible to crash the mesos agent by posting a reasonable request to the > operator API. > h3. Background: > Sending a request to the v1 api endpoint with an unsupported 'accept' header: > {code:java} > curl -X POST http://10.0.3.27:5051/api/v1 \ > -H 'accept: application/atom+xml' \ > -H 'content-type: application/json' \ > -d '{"type":"GET_CONTAINERS","get_containers":{"show_nested": > true,"show_standalone": true}}'{code} > Results in the following friendly error message: > {code:java} > Expecting 'Accept' to allow application/json or application/x-protobuf or > application/recordio{code} > h3. Reproducible crash: > However, sending the same request with 'application/recordio' 'accept' header: > {code:java} > curl -X POST \ > http://10.0.3.27:5051/api/v1 \ > -H 'accept: application/recordio' \ > -H 'content-type: application/json' \ > -d '{"type":"GET_CONTAINERS","get_containers":{"show_nested": > true,"show_standalone": true}}'{code} > causes the agent to crash (no response is received). > Crash log is shown below, full log from the agent is attached here: > {code:java} > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: > I0607 22:30:32.397320 3743 logfmt.cpp:178] type=audit timestamp=2018-06-07 > 22:30:32.397243904+00:00 reason="Error in token 'Missing 'Authorization' > header from HTTP request'. Allowing anonymous connection" > object="/slave(1)/api/v1" agent="Mozilla/5.0 (Macintosh; Intel Mac OS X > 10_13_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 > Safari/537.36" authorizer="mesos-agent" action="POST" result=allow > srcip=10.0.6.99 dstport=5051 srcport=42084 dstip=10.0.3.27 > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: > W0607 22:30:32.397434 3743 authenticator.cpp:289] Error in token on request > from '10.0.6.99:42084': Missing 'Authorization' header from HTTP request > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: > W0607 22:30:32.397466 3743 authenticator.cpp:291] Falling back to anonymous > connection using user 'dcos_anonymous' > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: > I0607 22:30:32.397629 3748 http.cpp:1099] HTTP POST for /slave(1)/api/v1 from > 10.0.6.99:42084 with User-Agent='Mozilla/5.0 (Macintosh; Intel Mac OS X > 10_13_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 > Safari/537.36' > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: > I0607 22:30:32.397784 3748 http.cpp:2030] Processing GET_CONTAINERS call > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: > F0607 22:30:32.398736 3747 http.cpp:121] Serializing a RecordIO stream is not > supported > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: > *** Check failure stack trace: *** > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ > 0x7f619478636d google::LogMessage::Fail() > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ > 0x7f619478819d google::LogMessage::SendToLog() > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ > 0x7f6194785f5c google::LogMessage::Flush() > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ > 0x7f6194788a99 google::LogMessageFatal::~LogMessageFatal() > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ > 0x7f61935e2b9d mesos::internal::serialize() > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ > 0x7f6193a4c0ef > _ZNO6lambda12CallableOnceIFN7process6FutureINS1_4http8ResponseEEERKN4JSON5ArrayEEE10CallableFnIZNK5mesos8internal5slave4Http13getContainersERKNSD_5agent4CallENSD_11ContentTypeERK6OptionINS3_14authentication9PrincipalEEEUlRKNS2_IS7_EEE0_EclES9_ > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ > 0x7f6193a81d61 process::internal::thenf<>() > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ > 0x7f6193a59b15 >
[jira] [Commented] (MESOS-8985) Posting to the operator api with 'accept recordio' header can crash the agent
[ https://issues.apache.org/jira/browse/MESOS-8985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507953#comment-16507953 ] Jan Schlicht commented on MESOS-8985: - This is caused by {{Content-Type}} being (in Mesos terms) non-streaming type, while {{Accept}} indicates a streaming type. We don't cover this case in the current code, make some wrong assumptions and finally erroneously try to serialize to RecordIO which isn't supported. > Posting to the operator api with 'accept recordio' header can crash the agent > - > > Key: MESOS-8985 > URL: https://issues.apache.org/jira/browse/MESOS-8985 > Project: Mesos > Issue Type: Bug >Affects Versions: 1.4.1, 1.5.1 >Reporter: Philip Norman >Assignee: Gilbert Song >Priority: Major > Attachments: mesos-slave-crash.log > > > It's possible to crash the mesos agent by posting a reasonable request to the > operator API. > h3. Background: > Sending a request to the v1 api endpoint with an unsupported 'accept' header: > {code:java} > curl -X POST http://10.0.3.27:5051/api/v1 \ > -H 'accept: application/atom+xml' \ > -H 'content-type: application/json' \ > -d '{"type":"GET_CONTAINERS","get_containers":{"show_nested": > true,"show_standalone": true}}'{code} > Results in the following friendly error message: > {code:java} > Expecting 'Accept' to allow application/json or application/x-protobuf or > application/recordio{code} > h3. Reproducible crash: > However, sending the same request with 'application/recordio' 'accept' header: > {code:java} > curl -X POST \ > http://10.0.3.27:5051/api/v1 \ > -H 'accept: application/recordio' \ > -H 'content-type: application/json' \ > -d '{"type":"GET_CONTAINERS","get_containers":{"show_nested": > true,"show_standalone": true}}'{code} > causes the agent to crash (no response is received). > Crash log is shown below, full log from the agent is attached here: > {code:java} > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: > I0607 22:30:32.397320 3743 logfmt.cpp:178] type=audit timestamp=2018-06-07 > 22:30:32.397243904+00:00 reason="Error in token 'Missing 'Authorization' > header from HTTP request'. Allowing anonymous connection" > object="/slave(1)/api/v1" agent="Mozilla/5.0 (Macintosh; Intel Mac OS X > 10_13_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 > Safari/537.36" authorizer="mesos-agent" action="POST" result=allow > srcip=10.0.6.99 dstport=5051 srcport=42084 dstip=10.0.3.27 > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: > W0607 22:30:32.397434 3743 authenticator.cpp:289] Error in token on request > from '10.0.6.99:42084': Missing 'Authorization' header from HTTP request > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: > W0607 22:30:32.397466 3743 authenticator.cpp:291] Falling back to anonymous > connection using user 'dcos_anonymous' > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: > I0607 22:30:32.397629 3748 http.cpp:1099] HTTP POST for /slave(1)/api/v1 from > 10.0.6.99:42084 with User-Agent='Mozilla/5.0 (Macintosh; Intel Mac OS X > 10_13_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 > Safari/537.36' > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: > I0607 22:30:32.397784 3748 http.cpp:2030] Processing GET_CONTAINERS call > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: > F0607 22:30:32.398736 3747 http.cpp:121] Serializing a RecordIO stream is not > supported > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: > *** Check failure stack trace: *** > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ > 0x7f619478636d google::LogMessage::Fail() > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ > 0x7f619478819d google::LogMessage::SendToLog() > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ > 0x7f6194785f5c google::LogMessage::Flush() > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ > 0x7f6194788a99 google::LogMessageFatal::~LogMessageFatal() > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ > 0x7f61935e2b9d mesos::internal::serialize() > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ > 0x7f6193a4c0ef > _ZNO6lambda12CallableOnceIFN7process6FutureINS1_4http8ResponseEEERKN4JSON5ArrayEEE10CallableFnIZNK5mesos8internal5slave4Http13getContainersERKNSD_5agent4CallENSD_11ContentTypeERK6OptionINS3_14authentication9PrincipalEEEUlRKNS2_IS7_EEE0_EclES9_ > Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ > 0x7f6193a81d61