[
https://issues.apache.org/jira/browse/MAPREDUCE-685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12725812#action_12725812
]
Aaron Kimball commented on MAPREDUCE-685:
-----------------------------------------
Added patch that fixes this issue. Also includes some other performance
enhancements:
* MySQL now uses "LIMIT 1" when making SELECTs against tables for
metadata-reading purposes.
* Transactions are no longer opened with TRANSACTION_SERIALIZABLE since it's
unnecessary for metadata reads
No new tests for this included since Hadoop testing doesn't mesh well with
MySQL. I tested locally by building a 1.7 GB table in mysql and reading into a
local HDFS instance. This failed before applying the patch, and succeeds
afterwards.
> Sqoop will fail with OutOfMemory on large tables using mysql
> ------------------------------------------------------------
>
> Key: MAPREDUCE-685
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-685
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/sqoop
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-685.patch
>
>
> The default MySQL JDBC client behavior is to buffer the entire ResultSet in
> the client before allowing the user to use the ResultSet object. On large
> SELECTs, this can cause OutOfMemory exceptions, even when the client intends
> to close the ResultSet after reading only a few rows. The MySQL ConnManager
> should configure its connection to use row-at-a-time delivery of results to
> the client.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
