[
https://issues.apache.org/jira/browse/MAPREDUCE-685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742426#action_12742426
]
Aaron Kimball commented on MAPREDUCE-685:
-----------------------------------------
Because it's not actually the same fix ;) Postgresql wants you to do
{{statement.setFetchSize(something_reasonable)}} e.g., 40.
MySQL wants you to do {{statement.setFetchSize(INT_MIN)}}. The only cursor
modes MySQL supports are fully buffered (fetch size = 0) and fully row-wise
cursors (fetch_size = INT_MIN).
That having been said, I have just finished a postgresql patch ready to post up
here this week :) Just waiting for some existing patches to get committed first
so that it applies cleanly.
> Sqoop will fail with OutOfMemory on large tables using mysql
> ------------------------------------------------------------
>
> Key: MAPREDUCE-685
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-685
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/sqoop
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-685.3.patch, MAPREDUCE-685.patch,
> MAPREDUCE-685.patch.2
>
>
> The default MySQL JDBC client behavior is to buffer the entire ResultSet in
> the client before allowing the user to use the ResultSet object. On large
> SELECTs, this can cause OutOfMemory exceptions, even when the client intends
> to close the ResultSet after reading only a few rows. The MySQL ConnManager
> should configure its connection to use row-at-a-time delivery of results to
> the client.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
