[jira] [Assigned] (AVRO-1156) Avro responder swallows thrown Errors
[ https://issues.apache.org/jira/browse/AVRO-1156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Percy reassigned AVRO-1156: Assignee: (was: Mike Percy) > Avro responder swallows thrown Errors > - > > Key: AVRO-1156 > URL: https://issues.apache.org/jira/browse/AVRO-1156 > Project: Avro > Issue Type: Bug >Reporter: Mike Percy >Priority: Major > Attachments: AVRO-1156-1.patch > > > The Avro responder wraps caught Errors, such as OutOfMemoryErrors, in > Exceptions and rethrows them. That's problematic because an Error should be > allowed to crash the JVM, since it's often irrecoverable. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-1197) Avro Java mapreduce pom depends on IPC test jar which is not exposed
[ https://issues.apache.org/jira/browse/AVRO-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13493457#comment-13493457 ] Mike Percy commented on AVRO-1197: -- I was running mvn clean install -DskipTests ... I can try looking @ it again later. Avro Java mapreduce pom depends on IPC test jar which is not exposed Key: AVRO-1197 URL: https://issues.apache.org/jira/browse/AVRO-1197 Project: Avro Issue Type: Bug Components: build, java Reporter: Mike Percy Assignee: Mike Percy Fix For: 1.7.3 Attachments: AVRO-1197.patch I am getting a build issue on trunk: [INFO] [INFO] Reactor Summary: [INFO] [INFO] Apache Avro Toplevel .. SUCCESS [1.202s] [INFO] Apache Avro Java .. SUCCESS [0.755s] [INFO] Apache Avro ... SUCCESS [7.015s] [INFO] Apache Avro Compiler .. SUCCESS [1.554s] [INFO] Apache Avro Maven Plugin .. SUCCESS [1.279s] [INFO] Apache Avro IPC ... SUCCESS [2.914s] [INFO] Trevni Java ... SUCCESS [0.195s] [INFO] Trevni Java Core .. SUCCESS [0.690s] [INFO] Apache Avro Mapred API FAILURE [0.427s] [INFO] Trevni Java Avro .. SKIPPED [INFO] Trevni Specification .. SKIPPED [INFO] Apache Avro Tools . SKIPPED [INFO] Apache Avro Protobuf Compatibility SKIPPED [INFO] Apache Avro Thrift Compatibility .. SKIPPED [INFO] Apache Avro Maven Archetypes .. SKIPPED [INFO] Apache Avro Maven Service Archetype ... SKIPPED [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 17.124s [INFO] Finished at: Wed Nov 07 16:30:55 PST 2012 [INFO] Final Memory: 27M/81M [INFO] [ERROR] Failed to execute goal on project avro-mapred: Could not resolve dependencies for project org.apache.avro:avro-mapred:jar:1.7.3-SNAPSHOT: Could not find artifact org.apache.avro:avro-ipc:jar:tests:1.7.3-SNAPSHOT in apache.snapshots (http://repository.apache.org/snapshots) - [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn goals -rf :avro-mapred -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1198) Malformed Avro data may cause confusing ArrayIndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/AVRO-1198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13493803#comment-13493803 ] Mike Percy commented on AVRO-1198: -- Doug, this looks good to me. I suppose we should add a unit test for this scenario as well. Malformed Avro data may cause confusing ArrayIndexOutOfBoundsException -- Key: AVRO-1198 URL: https://issues.apache.org/jira/browse/AVRO-1198 Project: Avro Issue Type: Bug Components: java Affects Versions: 1.7.2 Reporter: Mike Percy Assignee: Mike Percy Fix For: 1.7.3 Attachments: AVRO-1198-1.patch, AVRO-1198.patch I am currently debugging an issue where I am getting an ArrayIndexOutOfBoundsException from the decoder while reading some Avro data. Turns out that the integer indicating number of bytes to read next is negative. It would be better if a more helpful error message were provided. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-1197) Avro Java mapreduce pom depends on IPC test jar which is not exposed
[ https://issues.apache.org/jira/browse/AVRO-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Percy updated AVRO-1197: - Attachment: AVRO-1197.patch This seems to fix the build issue for me. Avro Java mapreduce pom depends on IPC test jar which is not exposed Key: AVRO-1197 URL: https://issues.apache.org/jira/browse/AVRO-1197 Project: Avro Issue Type: Bug Components: build, java Reporter: Mike Percy Fix For: 1.7.3 Attachments: AVRO-1197.patch I am getting a build issue on trunk: [INFO] [INFO] Reactor Summary: [INFO] [INFO] Apache Avro Toplevel .. SUCCESS [1.202s] [INFO] Apache Avro Java .. SUCCESS [0.755s] [INFO] Apache Avro ... SUCCESS [7.015s] [INFO] Apache Avro Compiler .. SUCCESS [1.554s] [INFO] Apache Avro Maven Plugin .. SUCCESS [1.279s] [INFO] Apache Avro IPC ... SUCCESS [2.914s] [INFO] Trevni Java ... SUCCESS [0.195s] [INFO] Trevni Java Core .. SUCCESS [0.690s] [INFO] Apache Avro Mapred API FAILURE [0.427s] [INFO] Trevni Java Avro .. SKIPPED [INFO] Trevni Specification .. SKIPPED [INFO] Apache Avro Tools . SKIPPED [INFO] Apache Avro Protobuf Compatibility SKIPPED [INFO] Apache Avro Thrift Compatibility .. SKIPPED [INFO] Apache Avro Maven Archetypes .. SKIPPED [INFO] Apache Avro Maven Service Archetype ... SKIPPED [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 17.124s [INFO] Finished at: Wed Nov 07 16:30:55 PST 2012 [INFO] Final Memory: 27M/81M [INFO] [ERROR] Failed to execute goal on project avro-mapred: Could not resolve dependencies for project org.apache.avro:avro-mapred:jar:1.7.3-SNAPSHOT: Could not find artifact org.apache.avro:avro-ipc:jar:tests:1.7.3-SNAPSHOT in apache.snapshots (http://repository.apache.org/snapshots) - [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn goals -rf :avro-mapred -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-1197) Avro Java mapreduce pom depends on IPC test jar which is not exposed
[ https://issues.apache.org/jira/browse/AVRO-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Percy updated AVRO-1197: - Assignee: Mike Percy Status: Patch Available (was: Open) Avro Java mapreduce pom depends on IPC test jar which is not exposed Key: AVRO-1197 URL: https://issues.apache.org/jira/browse/AVRO-1197 Project: Avro Issue Type: Bug Components: build, java Reporter: Mike Percy Assignee: Mike Percy Fix For: 1.7.3 Attachments: AVRO-1197.patch I am getting a build issue on trunk: [INFO] [INFO] Reactor Summary: [INFO] [INFO] Apache Avro Toplevel .. SUCCESS [1.202s] [INFO] Apache Avro Java .. SUCCESS [0.755s] [INFO] Apache Avro ... SUCCESS [7.015s] [INFO] Apache Avro Compiler .. SUCCESS [1.554s] [INFO] Apache Avro Maven Plugin .. SUCCESS [1.279s] [INFO] Apache Avro IPC ... SUCCESS [2.914s] [INFO] Trevni Java ... SUCCESS [0.195s] [INFO] Trevni Java Core .. SUCCESS [0.690s] [INFO] Apache Avro Mapred API FAILURE [0.427s] [INFO] Trevni Java Avro .. SKIPPED [INFO] Trevni Specification .. SKIPPED [INFO] Apache Avro Tools . SKIPPED [INFO] Apache Avro Protobuf Compatibility SKIPPED [INFO] Apache Avro Thrift Compatibility .. SKIPPED [INFO] Apache Avro Maven Archetypes .. SKIPPED [INFO] Apache Avro Maven Service Archetype ... SKIPPED [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 17.124s [INFO] Finished at: Wed Nov 07 16:30:55 PST 2012 [INFO] Final Memory: 27M/81M [INFO] [ERROR] Failed to execute goal on project avro-mapred: Could not resolve dependencies for project org.apache.avro:avro-mapred:jar:1.7.3-SNAPSHOT: Could not find artifact org.apache.avro:avro-ipc:jar:tests:1.7.3-SNAPSHOT in apache.snapshots (http://repository.apache.org/snapshots) - [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn goals -rf :avro-mapred -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (AVRO-1197) Avro Java mapreduce pom depends on IPC test jar which is not exposed
Mike Percy created AVRO-1197: Summary: Avro Java mapreduce pom depends on IPC test jar which is not exposed Key: AVRO-1197 URL: https://issues.apache.org/jira/browse/AVRO-1197 Project: Avro Issue Type: Bug Components: build, java Reporter: Mike Percy Fix For: 1.7.3 Attachments: AVRO-1197.patch I am getting a build issue on trunk: [INFO] [INFO] Reactor Summary: [INFO] [INFO] Apache Avro Toplevel .. SUCCESS [1.202s] [INFO] Apache Avro Java .. SUCCESS [0.755s] [INFO] Apache Avro ... SUCCESS [7.015s] [INFO] Apache Avro Compiler .. SUCCESS [1.554s] [INFO] Apache Avro Maven Plugin .. SUCCESS [1.279s] [INFO] Apache Avro IPC ... SUCCESS [2.914s] [INFO] Trevni Java ... SUCCESS [0.195s] [INFO] Trevni Java Core .. SUCCESS [0.690s] [INFO] Apache Avro Mapred API FAILURE [0.427s] [INFO] Trevni Java Avro .. SKIPPED [INFO] Trevni Specification .. SKIPPED [INFO] Apache Avro Tools . SKIPPED [INFO] Apache Avro Protobuf Compatibility SKIPPED [INFO] Apache Avro Thrift Compatibility .. SKIPPED [INFO] Apache Avro Maven Archetypes .. SKIPPED [INFO] Apache Avro Maven Service Archetype ... SKIPPED [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 17.124s [INFO] Finished at: Wed Nov 07 16:30:55 PST 2012 [INFO] Final Memory: 27M/81M [INFO] [ERROR] Failed to execute goal on project avro-mapred: Could not resolve dependencies for project org.apache.avro:avro-mapred:jar:1.7.3-SNAPSHOT: Could not find artifact org.apache.avro:avro-ipc:jar:tests:1.7.3-SNAPSHOT in apache.snapshots (http://repository.apache.org/snapshots) - [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn goals -rf :avro-mapred -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (AVRO-1198) Malformed Avro data may cause confusing ArrayIndexOutOfBoundsException
Mike Percy created AVRO-1198: Summary: Malformed Avro data may cause confusing ArrayIndexOutOfBoundsException Key: AVRO-1198 URL: https://issues.apache.org/jira/browse/AVRO-1198 Project: Avro Issue Type: Bug Components: java Affects Versions: 1.7.2 Reporter: Mike Percy Fix For: 1.7.3 I am currently debugging an issue where I am getting an ArrayIndexOutOfBoundsException from the decoder while reading some Avro data. Turns out that the integer indicating number of bytes to read next is negative. It would be better if a more helpful error message were provided. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-1198) Malformed Avro data may cause confusing ArrayIndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/AVRO-1198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Percy updated AVRO-1198: - Attachment: AVRO-1198-1.patch Simple patch to throw an IllegalArgumentException if this occurs. I don't know if this is acceptable since I am not super familiar with this code, so just starting with something. Before: java.lang.ArrayIndexOutOfBoundsException at org.apache.avro.io.BinaryDecoder.doReadBytes(BinaryDecoder.java:329) at org.apache.avro.io.BinaryDecoder.readString(BinaryDecoder.java:255) at org.apache.avro.io.BinaryDecoder.readString(BinaryDecoder.java:264) at org.apache.avro.io.ValidatingDecoder.readString(ValidatingDecoder.java:113) ... After: java.lang.IllegalArgumentException: Attempt to read negative length: -50 at org.apache.avro.io.BinaryDecoder.doReadBytes(BinaryDecoder.java:329) at org.apache.avro.io.BinaryDecoder.readString(BinaryDecoder.java:255) at org.apache.avro.io.BinaryDecoder.readString(BinaryDecoder.java:264) at org.apache.avro.io.ValidatingDecoder.readString(ValidatingDecoder.java:113) ... Malformed Avro data may cause confusing ArrayIndexOutOfBoundsException -- Key: AVRO-1198 URL: https://issues.apache.org/jira/browse/AVRO-1198 Project: Avro Issue Type: Bug Components: java Affects Versions: 1.7.2 Reporter: Mike Percy Fix For: 1.7.3 Attachments: AVRO-1198-1.patch I am currently debugging an issue where I am getting an ArrayIndexOutOfBoundsException from the decoder while reading some Avro data. Turns out that the integer indicating number of bytes to read next is negative. It would be better if a more helpful error message were provided. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-1198) Malformed Avro data may cause confusing ArrayIndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/AVRO-1198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Percy updated AVRO-1198: - Assignee: Mike Percy Status: Patch Available (was: Open) Malformed Avro data may cause confusing ArrayIndexOutOfBoundsException -- Key: AVRO-1198 URL: https://issues.apache.org/jira/browse/AVRO-1198 Project: Avro Issue Type: Bug Components: java Affects Versions: 1.7.2 Reporter: Mike Percy Assignee: Mike Percy Fix For: 1.7.3 Attachments: AVRO-1198-1.patch I am currently debugging an issue where I am getting an ArrayIndexOutOfBoundsException from the decoder while reading some Avro data. Turns out that the integer indicating number of bytes to read next is negative. It would be better if a more helpful error message were provided. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1122) Java: Avro RPC Requestor can block during handshake in async mode
[ https://issues.apache.org/jira/browse/AVRO-1122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13486585#comment-13486585 ] Mike Percy commented on AVRO-1122: -- Hi James, yep now I know it pretty much has to block based on the Transceiver API and the way the Proxy is implemented. It's a little tricky, since at the moment you can't perform the handshake before the proxy object is instantiated AFAICT. Based on my experience with Avro RPC in Flume, I'd like to be able to use a client API that provides something along the lines of: 1. set address port of endpoint (NettyTransceiver does this today) 2. set interface (SpecificRequestor.getClient() does this today) 3. async TCP connection using a CallFuture (not possible today) - throws if address/port not set properly 4. async handshake using a CallFuture (not possible today) - throws if not connected, or if interface not set properly - alternatively, specify a boolean flag to connect if needed 5. async RPC calls (possible using the CallFuture / Callback APIs) - throws if handshake not completed or connection is not open I'm not sure how to weave that into the existing framework yet, though. Java: Avro RPC Requestor can block during handshake in async mode - Key: AVRO-1122 URL: https://issues.apache.org/jira/browse/AVRO-1122 Project: Avro Issue Type: Bug Affects Versions: 1.6.3 Reporter: Mike Percy Attachments: Screen Shot 2012-06-27 at 12.43.32 AM.png We are seeing an issue in Flume where the Avro RPC Requestor is blocking for long periods of time waiting for the Avro handshake to complete. Since we are using the API with Futures, this should not block. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1111) Malformed data can cause OutOfMemoryError in Avro IPC
[
https://issues.apache.org/jira/browse/AVRO-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13453281#comment-13453281
]
Mike Percy commented on AVRO-:
--
Thanks Tom!
Malformed data can cause OutOfMemoryError in Avro IPC
-
Key: AVRO-
URL: https://issues.apache.org/jira/browse/AVRO-
Project: Avro
Issue Type: Bug
Components: java
Affects Versions: 1.6.3
Reporter: Hari Shreedharan
Assignee: Mike Percy
Fix For: 1.7.2
Attachments: AVRO--1.patch, AVRO--2.patch
If the data that comes in through the Netty channel buffer is not framed
correctly/is not valid Avro data, then the incoming data can cause
arbitrarily large array lists to be created, causing OutOfMemoryError.
The relevant code(org.apache.avro.ipc.NettyTransportCodec):
private boolean decodePackHeader(ChannelHandlerContext ctx, Channel channel,
ChannelBuffer buffer) throws Exception {
if (buffer.readableBytes()8) { return false; }
int serial = buffer.readInt();
listSize = buffer.readInt();
dataPack = new NettyDataPack(serial, new ArrayListByteBuffer(listSize));
return true;
}
If the buffer does not have valid Avro data, the listSize variable can have
arbitrary values, causing massive ArrayLists to be created, leading to
OutOfMemoryErrors.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (AVRO-1156) Avro responder swallows thrown Errors
Mike Percy created AVRO-1156: Summary: Avro responder swallows thrown Errors Key: AVRO-1156 URL: https://issues.apache.org/jira/browse/AVRO-1156 Project: Avro Issue Type: Bug Reporter: Mike Percy Fix For: 1.7.2 The Avro responder wraps caught Errors, such as OutOfMemoryErrors, in Exceptions and rethrows them. That's problematic because an Error should be allowed to crash the JVM, since it's often irrecoverable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-1156) Avro responder swallows thrown Errors
[ https://issues.apache.org/jira/browse/AVRO-1156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Percy updated AVRO-1156: - Attachment: AVRO-1156-1.patch Small patch so that if the Throwable is an Error, throw it as an Error. Avro responder swallows thrown Errors - Key: AVRO-1156 URL: https://issues.apache.org/jira/browse/AVRO-1156 Project: Avro Issue Type: Bug Reporter: Mike Percy Fix For: 1.7.2 Attachments: AVRO-1156-1.patch The Avro responder wraps caught Errors, such as OutOfMemoryErrors, in Exceptions and rethrows them. That's problematic because an Error should be allowed to crash the JVM, since it's often irrecoverable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-1156) Avro responder swallows thrown Errors
[ https://issues.apache.org/jira/browse/AVRO-1156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Percy updated AVRO-1156: - Assignee: Mike Percy Status: Patch Available (was: Open) Avro responder swallows thrown Errors - Key: AVRO-1156 URL: https://issues.apache.org/jira/browse/AVRO-1156 Project: Avro Issue Type: Bug Reporter: Mike Percy Assignee: Mike Percy Fix For: 1.7.2 Attachments: AVRO-1156-1.patch The Avro responder wraps caught Errors, such as OutOfMemoryErrors, in Exceptions and rethrows them. That's problematic because an Error should be allowed to crash the JVM, since it's often irrecoverable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1156) Avro responder swallows thrown Errors
[ https://issues.apache.org/jira/browse/AVRO-1156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13453462#comment-13453462 ] Mike Percy commented on AVRO-1156: -- Thanks for the feedback guys! Re. the zombie detection, makes sense... there appear to be many benefits to doing it that way. The issue I think I'm seeing is that an OOM error gets thrown while trying to service the request, so we wrap it in an Exception and try to handle it, so the Responder tries to allocate memory to serialize the error response and at that point another OOM error gets thrown by the JVM and we lose the original error. I think trying to handle certain Errors in general is a tough proposition... maybe most Errors can be marshalled but OutOfMemoryError or VirtualMachineError should be special cased to just propagate? Or if we really want to try, maybe we should just log the Errors before wrapping them so at least we know if we are hitting a double-throw exception clobbering situation like I am suspecting. Avro responder swallows thrown Errors - Key: AVRO-1156 URL: https://issues.apache.org/jira/browse/AVRO-1156 Project: Avro Issue Type: Bug Reporter: Mike Percy Assignee: Mike Percy Fix For: 1.7.2 Attachments: AVRO-1156-1.patch The Avro responder wraps caught Errors, such as OutOfMemoryErrors, in Exceptions and rethrows them. That's problematic because an Error should be allowed to crash the JVM, since it's often irrecoverable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1156) Avro responder swallows thrown Errors
[ https://issues.apache.org/jira/browse/AVRO-1156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13453473#comment-13453473 ] Mike Percy commented on AVRO-1156: -- To add a bit more information here, I'm seeing stack traces along the lines of the following: WARN ipc.Responder: user error java.lang.Exception: java.lang.OutOfMemoryError: Java heap space at org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:93) at org.apache.avro.ipc.Responder.respond(Responder.java:149) at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:158) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80) at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:143) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:302) at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321) at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299) at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:214) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80) ... snip ... Caused by: java.lang.OutOfMemoryError: Java heap space and WARN ipc.Responder: system error org.apache.avro.AvroRuntimeException: Unknown datum type: java.lang.Exception: java.lang.OutOfMemoryError: Java heap space at org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:574) at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:539) at org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:137) at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:70) at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:57) at org.apache.avro.ipc.specific.SpecificResponder.writeError(SpecificResponder.java:71) at org.apache.avro.ipc.Responder.respond(Responder.java:167) at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:158) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80) at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:143) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:302) at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321) ... snip ... at java.lang.Thread.run(Thread.java:662) Avro responder swallows thrown Errors - Key: AVRO-1156 URL: https://issues.apache.org/jira/browse/AVRO-1156 Project: Avro Issue Type: Bug Reporter: Mike Percy Assignee: Mike Percy Fix For: 1.7.2 Attachments: AVRO-1156-1.patch The Avro responder wraps caught Errors, such as OutOfMemoryErrors, in Exceptions and rethrows them. That's problematic because an Error should be allowed to crash the JVM, since it's often irrecoverable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1156) Avro responder swallows thrown Errors
[ https://issues.apache.org/jira/browse/AVRO-1156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13453639#comment-13453639 ] Mike Percy commented on AVRO-1156: -- Doug I agree with your last comment about improving the logging. The double OOM error is a hunch, I will try to find more evidence of it. Avro responder swallows thrown Errors - Key: AVRO-1156 URL: https://issues.apache.org/jira/browse/AVRO-1156 Project: Avro Issue Type: Bug Reporter: Mike Percy Assignee: Mike Percy Fix For: 1.7.2 Attachments: AVRO-1156-1.patch The Avro responder wraps caught Errors, such as OutOfMemoryErrors, in Exceptions and rethrows them. That's problematic because an Error should be allowed to crash the JVM, since it's often irrecoverable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1111) Malformed data can cause OutOfMemoryError in Avro IPC
[
https://issues.apache.org/jira/browse/AVRO-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13451827#comment-13451827
]
Mike Percy commented on AVRO-:
--
Phil, thank you very much for the ideas! I agree with trying harder to count
the actual received bytes (deep count). Controlling the max-size on the server,
while not requiring the client to provide it, looks more future proof to me.
Malformed data can cause OutOfMemoryError in Avro IPC
-
Key: AVRO-
URL: https://issues.apache.org/jira/browse/AVRO-
Project: Avro
Issue Type: Bug
Components: java
Affects Versions: 1.6.3
Reporter: Hari Shreedharan
Attachments: AVRO--1.patch
If the data that comes in through the Netty channel buffer is not framed
correctly/is not valid Avro data, then the incoming data can cause
arbitrarily large array lists to be created, causing OutOfMemoryError.
The relevant code(org.apache.avro.ipc.NettyTransportCodec):
private boolean decodePackHeader(ChannelHandlerContext ctx, Channel channel,
ChannelBuffer buffer) throws Exception {
if (buffer.readableBytes()8) { return false; }
int serial = buffer.readInt();
listSize = buffer.readInt();
dataPack = new NettyDataPack(serial, new ArrayListByteBuffer(listSize));
return true;
}
If the buffer does not have valid Avro data, the listSize variable can have
arbitrary values, causing massive ArrayLists to be created, leading to
OutOfMemoryErrors.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-1111) Malformed data can cause OutOfMemoryError in Avro IPC
[
https://issues.apache.org/jira/browse/AVRO-?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mike Percy updated AVRO-:
-
Status: Patch Available (was: Open)
Malformed data can cause OutOfMemoryError in Avro IPC
-
Key: AVRO-
URL: https://issues.apache.org/jira/browse/AVRO-
Project: Avro
Issue Type: Bug
Components: java
Affects Versions: 1.6.3
Reporter: Hari Shreedharan
Attachments: AVRO--1.patch
If the data that comes in through the Netty channel buffer is not framed
correctly/is not valid Avro data, then the incoming data can cause
arbitrarily large array lists to be created, causing OutOfMemoryError.
The relevant code(org.apache.avro.ipc.NettyTransportCodec):
private boolean decodePackHeader(ChannelHandlerContext ctx, Channel channel,
ChannelBuffer buffer) throws Exception {
if (buffer.readableBytes()8) { return false; }
int serial = buffer.readInt();
listSize = buffer.readInt();
dataPack = new NettyDataPack(serial, new ArrayListByteBuffer(listSize));
return true;
}
If the buffer does not have valid Avro data, the listSize variable can have
arbitrary values, causing massive ArrayLists to be created, leading to
OutOfMemoryErrors.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-1111) Malformed data can cause OutOfMemoryError in Avro IPC
[
https://issues.apache.org/jira/browse/AVRO-?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mike Percy updated AVRO-:
-
Attachment: AVRO--1.patch
I think this patch is more limited but it's a first pass. What it does it try
to calculate whether an OOM will likely result from the listSize allocation
from the RPC header field, and if so throw an AvroRemoteException. It also
closes the socket.
I think it makes more sense to set an arbitrary limit on the listSize and check
for that since we should throw at closer to 1000 instead of like 7 billion or
something. But I'm not familiar enough with the Responder implementation to
know whether something like that is a bad idea or not... would there be
repercussions to that?
There are also other places where we should do some validation, such as the
length headers before the byte buffers... but that is a little tricky too
(arbitrary limits can be problematic for obvious reasons). I'd like some
suggestions for how to handle this as the current state leaves the protocol
pretty brittle.
Malformed data can cause OutOfMemoryError in Avro IPC
-
Key: AVRO-
URL: https://issues.apache.org/jira/browse/AVRO-
Project: Avro
Issue Type: Bug
Components: java
Affects Versions: 1.6.3
Reporter: Hari Shreedharan
Attachments: AVRO--1.patch
If the data that comes in through the Netty channel buffer is not framed
correctly/is not valid Avro data, then the incoming data can cause
arbitrarily large array lists to be created, causing OutOfMemoryError.
The relevant code(org.apache.avro.ipc.NettyTransportCodec):
private boolean decodePackHeader(ChannelHandlerContext ctx, Channel channel,
ChannelBuffer buffer) throws Exception {
if (buffer.readableBytes()8) { return false; }
int serial = buffer.readInt();
listSize = buffer.readInt();
dataPack = new NettyDataPack(serial, new ArrayListByteBuffer(listSize));
return true;
}
If the buffer does not have valid Avro data, the listSize variable can have
arbitrary values, causing massive ArrayLists to be created, leading to
OutOfMemoryErrors.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1054) AvroRuntimeException: Unknown datum type: java.lang.NullPointerException
[
https://issues.apache.org/jira/browse/AVRO-1054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13416996#comment-13416996
]
Mike Percy commented on AVRO-1054:
--
Worth noting for future reference that when you see this error on the RPC
client, the Avro RPC server has thrown an uncaught NullPointerException inside
an RPC handler method.
AvroRuntimeException: Unknown datum type: java.lang.NullPointerException
Key: AVRO-1054
URL: https://issues.apache.org/jira/browse/AVRO-1054
Project: Avro
Issue Type: Bug
Components: java
Affects Versions: 1.6.3
Environment: Linux, Java 1.6
Reporter: Sushanth
I have a simple messaging protocol, which sends some data, and receives a
response.
I am calling
notification.setStatus(MessageStatus.DELIVERED);client.notify(notification).
I am getting the following error.
org.apache.avro.AvroRuntimeException: Unknown datum type:
java.lang.NullPointerException
org.apache.avro.AvroRuntimeException: org.apache.avro.AvroRuntimeException:
Unknown datum type: java.lang.NullPointerException
at
org.apache.avro.ipc.specific.SpecificRequestor.readError(SpecificRequestor.java:126)
at
org.apache.avro.ipc.Requestor$Response.getResponse(Requestor.java:554)
at
org.apache.avro.ipc.Requestor$TransceiverCallback.handleResult(Requestor.java:359)
at
org.apache.avro.ipc.Requestor$TransceiverCallback.handleResult(Requestor.java:322)
at
org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.messageReceived(NettyTransceiver.java:495)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
at
org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.handleUpstream(NettyTransceiver.java:477)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:302)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:216)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:351)
at
org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
When I do
notification.setStatus(MessageStatus.READ);client.notify(notification);
things are fine.
my avpr is
{namespace: com.company.avro.generated,
protocol: CallResponse,
types: [
{name: Notification, type: record,
fields: [
{name: recipientID, type: string},
{name: message, type: string},
{name: senderID, type: string},
{name: messageID, type: string},
{name: transactionID, type: string},
{name: submitTimestamp, type: string},
{name: msgNotification, type: string},
{name: status, type: {type: enum, name:
MessageStatus,
symbols: [DELIVERED, FAILED, READ]}}
]
},
{name: NotificationAck, type: record,
fields: [{name: response, type: string}]
}
],
messages: {
send: {
request: [{name: notification, type: Notification}],
response: NotificationAck
}
}
}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
[jira] [Created] (AVRO-1122) Java: Avro RPC Requestor can block during handshake in async mode
Mike Percy created AVRO-1122: Summary: Java: Avro RPC Requestor can block during handshake in async mode Key: AVRO-1122 URL: https://issues.apache.org/jira/browse/AVRO-1122 Project: Avro Issue Type: Bug Reporter: Mike Percy We are seeing an issue in Flume where the Avro RPC Requestor is blocking for long periods of time waiting for the Avro handshake to complete. Since we are using the API with Futures, this should not block. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (AVRO-1118) Specifying null as default of a union only works if null is specified as first type in the union
[ https://issues.apache.org/jira/browse/AVRO-1118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Percy resolved AVRO-1118. -- Resolution: Not A Problem Thanks for the quick response, Christophe! In that case, I'll close this JIRA. Regards, Mike Specifying null as default of a union only works if null is specified as first type in the union Key: AVRO-1118 URL: https://issues.apache.org/jira/browse/AVRO-1118 Project: Avro Issue Type: Bug Affects Versions: 1.6.3 Reporter: Mike Percy There is some unexpected behavior I am coming across where if I specify a union as such: type: [string, null], default: null I get an Exception: Exception in thread main org.apache.avro.AvroTypeException: Non-string default value for string: null at org.apache.avro.io.parsing.ResolvingGrammarGenerator.encode(ResolvingGrammarGenerator.java:363) at org.apache.avro.io.parsing.ResolvingGrammarGenerator.encode(ResolvingGrammarGenerator.java:350) at org.apache.avro.io.parsing.ResolvingGrammarGenerator.getBinary(ResolvingGrammarGenerator.java:293) at org.apache.avro.io.parsing.ResolvingGrammarGenerator.resolveRecords(ResolvingGrammarGenerator.java:271) at org.apache.avro.io.parsing.ResolvingGrammarGenerator.generate(ResolvingGrammarGenerator.java:118) at org.apache.avro.io.parsing.ResolvingGrammarGenerator.generate(ResolvingGrammarGenerator.java:50) at org.apache.avro.io.ResolvingDecoder.resolve(ResolvingDecoder.java:82) at org.apache.avro.io.ResolvingDecoder.init(ResolvingDecoder.java:46) at org.apache.avro.io.DecoderFactory.resolvingDecoder(DecoderFactory.java:307) at org.apache.avro.generic.GenericDatumReader.getResolver(GenericDatumReader.java:118) at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:133) at org.apache.avro.file.DataFileStream.next(DataFileStream.java:233) at org.apache.avro.file.DataFileStream.next(DataFileStream.java:220) ... Whereas if I specify the schema as: type: [null, string], default: null It works as expected. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (AVRO-1118) Specifying null as default of a union only works if null is specified as first type in the union
Mike Percy created AVRO-1118: Summary: Specifying null as default of a union only works if null is specified as first type in the union Key: AVRO-1118 URL: https://issues.apache.org/jira/browse/AVRO-1118 Project: Avro Issue Type: Bug Affects Versions: 1.6.3 Reporter: Mike Percy There is some unexpected behavior I am coming across where if I specify a union as such: type: [string, null], default: null I get an Exception: Exception in thread main org.apache.avro.AvroTypeException: Non-string default value for string: null at org.apache.avro.io.parsing.ResolvingGrammarGenerator.encode(ResolvingGrammarGenerator.java:363) at org.apache.avro.io.parsing.ResolvingGrammarGenerator.encode(ResolvingGrammarGenerator.java:350) at org.apache.avro.io.parsing.ResolvingGrammarGenerator.getBinary(ResolvingGrammarGenerator.java:293) at org.apache.avro.io.parsing.ResolvingGrammarGenerator.resolveRecords(ResolvingGrammarGenerator.java:271) at org.apache.avro.io.parsing.ResolvingGrammarGenerator.generate(ResolvingGrammarGenerator.java:118) at org.apache.avro.io.parsing.ResolvingGrammarGenerator.generate(ResolvingGrammarGenerator.java:50) at org.apache.avro.io.ResolvingDecoder.resolve(ResolvingDecoder.java:82) at org.apache.avro.io.ResolvingDecoder.init(ResolvingDecoder.java:46) at org.apache.avro.io.DecoderFactory.resolvingDecoder(DecoderFactory.java:307) at org.apache.avro.generic.GenericDatumReader.getResolver(GenericDatumReader.java:118) at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:133) at org.apache.avro.file.DataFileStream.next(DataFileStream.java:233) at org.apache.avro.file.DataFileStream.next(DataFileStream.java:220) ... Whereas if I specify the schema as: type: [null, string], default: null It works as expected. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1118) Specifying null as default of a union only works if null is specified as first type in the union
[ https://issues.apache.org/jira/browse/AVRO-1118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13397225#comment-13397225 ] Mike Percy commented on AVRO-1118: -- Note that this occurs when trying to read GenericRecords via DataFileReader when the field is not present in the data. Specifying null as default of a union only works if null is specified as first type in the union Key: AVRO-1118 URL: https://issues.apache.org/jira/browse/AVRO-1118 Project: Avro Issue Type: Bug Affects Versions: 1.6.3 Reporter: Mike Percy There is some unexpected behavior I am coming across where if I specify a union as such: type: [string, null], default: null I get an Exception: Exception in thread main org.apache.avro.AvroTypeException: Non-string default value for string: null at org.apache.avro.io.parsing.ResolvingGrammarGenerator.encode(ResolvingGrammarGenerator.java:363) at org.apache.avro.io.parsing.ResolvingGrammarGenerator.encode(ResolvingGrammarGenerator.java:350) at org.apache.avro.io.parsing.ResolvingGrammarGenerator.getBinary(ResolvingGrammarGenerator.java:293) at org.apache.avro.io.parsing.ResolvingGrammarGenerator.resolveRecords(ResolvingGrammarGenerator.java:271) at org.apache.avro.io.parsing.ResolvingGrammarGenerator.generate(ResolvingGrammarGenerator.java:118) at org.apache.avro.io.parsing.ResolvingGrammarGenerator.generate(ResolvingGrammarGenerator.java:50) at org.apache.avro.io.ResolvingDecoder.resolve(ResolvingDecoder.java:82) at org.apache.avro.io.ResolvingDecoder.init(ResolvingDecoder.java:46) at org.apache.avro.io.DecoderFactory.resolvingDecoder(DecoderFactory.java:307) at org.apache.avro.generic.GenericDatumReader.getResolver(GenericDatumReader.java:118) at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:133) at org.apache.avro.file.DataFileStream.next(DataFileStream.java:233) at org.apache.avro.file.DataFileStream.next(DataFileStream.java:220) ... Whereas if I specify the schema as: type: [null, string], default: null It works as expected. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
