[
https://issues.apache.org/jira/browse/MAPREDUCE-685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742345#action_12742345
]
Martin Dittus commented on MAPREDUCE-685:
-----------------------------------------
We just found that PostgreSQL shows the same behaviour. What do you think of
making this a generic fix instead? It seems Postgres has the same mechanism to
enable streaming of ResultSets:
http://jdbc.postgresql.org/documentation/83/query.html -- "Changing code to
cursor mode is as simple as setting the fetch size of the Statement to the
appropriate size. Setting the fetch size back to 0 will cause all rows to be
cached (the default behaviour)."
> Sqoop will fail with OutOfMemory on large tables using mysql
> ------------------------------------------------------------
>
> Key: MAPREDUCE-685
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-685
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/sqoop
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-685.3.patch, MAPREDUCE-685.patch,
> MAPREDUCE-685.patch.2
>
>
> The default MySQL JDBC client behavior is to buffer the entire ResultSet in
> the client before allowing the user to use the ResultSet object. On large
> SELECTs, this can cause OutOfMemory exceptions, even when the client intends
> to close the ResultSet after reading only a few rows. The MySQL ConnManager
> should configure its connection to use row-at-a-time delivery of results to
> the client.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
