[
https://issues.apache.org/jira/browse/MESOS-4642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15319175#comment-15319175
]
Steven Schlansker commented on MESOS-4642:
------------------------------------------
I really think a documentation "fix" is a bad solution for this issue. This is
an API that can be broken solely based on user controlled data. Allowing a
(potentially malicious) "isolated" process to cause Mesos APIs to produce
semantically invalid responses is a bad behavior that is worthy of a breaking
change IMO.
Even if the cluster admin understands the gotcha, end users can break it
unwittingly. We never intended to make non-UTF8 data, that was an accident, so
any existing documentation would have helped us understand and recover but
could not have prevented this outage.
> Mesos Agent Json API can dump binary data from log files out as invalid JSON
> ----------------------------------------------------------------------------
>
> Key: MESOS-4642
> URL: https://issues.apache.org/jira/browse/MESOS-4642
> Project: Mesos
> Issue Type: Bug
> Components: json api, slave
> Affects Versions: 0.27.0
> Reporter: Steven Schlansker
> Priority: Critical
>
> One of our tasks accidentally started logging binary data to stderr. This
> was not intentional and generally should not happen -- however, it causes
> severe problems with the Mesos Agent "files/read.json" API, since it gladly
> dumps this binary data out as invalid JSON.
> {code}
> # hexdump -C /path/to/task/stderr | tail
> 0003d1f0 6f 6e 6e 65 63 74 69 6f 6e 0a 4e 45 54 3a 20 31 |onnection.NET: 1|
> 0003d200 20 6f 6e 72 65 61 64 20 45 4e 4f 45 4e 54 20 32 | onread ENOENT 2|
> 0003d210 39 35 34 35 36 20 32 35 31 20 32 39 35 37 30 37 |95456 251 295707|
> 0003d220 0a 01 00 00 00 00 00 00 ac 57 65 64 2c 20 31 30 |.........Wed, 10|
> 0003d230 20 55 6e 72 65 63 6f 67 6e 69 7a 65 64 20 69 6e | Unrecognized in|
> 0003d240 70 75 74 20 68 65 61 64 65 72 0a |put header.|
> {code}
> {code}
> # curl
> 'http://agent-host:5051/files/read.json?path=/path/to/task/stderr&offset=220443&length=90000&grep='
> | hexdump -C
> 00007970 6e 65 63 74 69 6f 6e 5c 6e 4e 45 54 3a 20 31 20 |nection\nNET: 1 |
> 00007980 6f 6e 72 65 61 64 20 45 4e 4f 45 4e 54 20 32 39 |onread ENOENT 29|
> 00007990 35 34 35 36 20 32 35 31 20 32 39 35 37 30 37 5c |5456 251 295707\|
> 000079a0 6e 5c 75 30 30 30 31 5c 75 30 30 30 30 5c 75 30 |n\u0001\u0000\u0|
> 000079b0 30 30 30 5c 75 30 30 30 30 5c 75 30 30 30 30 5c |000\u0000\u0000\|
> 000079c0 75 30 30 30 30 5c 75 30 30 30 30 ac 57 65 64 2c |u0000\u0000.Wed,|
> 000079d0 20 31 30 20 55 6e 72 65 63 6f 67 6e 69 7a 65 64 | 10 Unrecognized|
> 000079e0 20 69 6e 70 75 74 20 68 65 61 64 65 72 5c 6e 22 | input header\n"|
> 000079f0 2c 22 6f 66 66 73 65 74 22 3a 32 32 30 34 34 33 |,"offset":220443|
> 00007a00 7d |}|
> {code}
> This causes downstream sadness:
> {code}
> ERROR [2016-02-10 18:55:12,303]
> io.dropwizard.jersey.errors.LoggingExceptionMapper: Error handling a request:
> 0ee749630f8b26f1
> ! com.fasterxml.jackson.core.JsonParseException: Invalid UTF-8 start byte 0xac
> ! at [Source: org.jboss.netty.buffer.ChannelBufferInputStream@6d69ee8; line:
> 1, column: 31181]
> ! at
> com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1487)
> ~[singularity-0.4.9.jar:0.4.9]
> ! at
> com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:518)
> ~[singularity-0.4.9.jar:0.4.9]
> ! at
> com.fasterxml.jackson.core.json.UTF8StreamJsonParser._reportInvalidInitial(UTF8StreamJsonParser.java:3339)
> ~[singularity-0.4.9.jar:0.4.9]
> ! at
> com.fasterxml.jackson.core.json.UTF8StreamJsonParser._reportInvalidChar(UTF8StreamJsonParser.java:3333)
> ~[singularity-0.4.9.jar:0.4.9]
> ! at
> com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishString2(UTF8StreamJsonParser.java:2360)
> ~[singularity-0.4.9.jar:0.4.9]
> ! at
> com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishString(UTF8StreamJsonParser.java:2287)
> ~[singularity-0.4.9.jar:0.4.9]
> ! at
> com.fasterxml.jackson.core.json.UTF8StreamJsonParser.getText(UTF8StreamJsonParser.java:286)
> ~[singularity-0.4.9.jar:0.4.9]
> ! at
> com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:29)
> ~[singularity-0.4.9.jar:0.4.9]
> ! at
> com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:12)
> ~[singularity-0.4.9.jar:0.4.9]
> ! at
> com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:523)
> ~[singularity-0.4.9.jar:0.4.9]
> ! at
> com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:381)
> ~[singularity-0.4.9.jar:0.4.9]
> ! at
> com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1073)
> ~[singularity-0.4.9.jar:0.4.9]
> ! at
> com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.deserializeFromObject(SuperSonicBeanDeserializer.java:196)
> ~[singularity-0.4.9.jar:0.4.9]
> ! at
> com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:142)
> ~[singularity-0.4.9.jar:0.4.9]
> ! at
> com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.deserialize(SuperSonicBeanDeserializer.java:117)
> ~[singularity-0.4.9.jar:0.4.9]
> ! at
> com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3562)
> ~[singularity-0.4.9.jar:0.4.9]
> ! at
> com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2648)
> ~[singularity-0.4.9.jar:0.4.9]
> ! at com.hubspot.singularity.data.SandboxManager.read(SandboxManager.java:97)
> ~[singularity-0.4.9.jar:0.4.9]
> {code}
> This is extremely similar to https://issues.apache.org/jira/browse/MESOS-3771
> Since this is now the second major issue caused by this, is there any
> possibility of using a JSON processing library that actually guarantees
> spec-compliant output? I know we can fix the point problem again here, but
> it is frustrating that this keeps happening, and I'm sure it will happen
> again in the future.
> Failing that, maybe we should audit all JSON objects produced to ensure they
> cannot contain binary data.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
