Re: Review Request: SQOOP-604 Easy throttling feature for MySQL exports
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/7135/ --- (Updated Oct. 4, 2012, 12:25 p.m.) Review request for Sqoop. Changes --- Sorry, I'm retarded. Description --- Code review for SQOOP-604, see https://issues.apache.org/jira/browse/SQOOP-604 The solution in short: Using the already existing checkpoint feature of the direct (--direct) MySQL exports (the export process is restarted every X bytes written), extending it with a new config value that would simply make the thread sleep for X milliseconds at the checkbpoints. With low enough byte count limit this can be a simple yet powerful throttling mechanism. Diffs (updated) - src/java/org/apache/sqoop/mapreduce/MySQLExportMapper.java a4e8b88 Diff: https://reviews.apache.org/r/7135/diff/ Testing --- Executing with different settings of sqoop.mysql.export.checkpoint.bytes and sqoop.mysql.export.sleep.ms: 33554432B / 0ms: Transferred 4.7579 MB in 8.7175 seconds (558.8826 KB/sec) 102400B / 500ms: Transferred 4.7579 MB in 35.7794 seconds (136.1698 KB/sec) 51200B / 500ms: Transferred 4.758 MB in 57.8675 seconds (84.1959 KB/sec) 51200B / 250ms: Transferred 4.7579 MB in 35.0293 seconds (139.0854 KB/sec) I did not add unit tests yet and as it involves calling to Thread.sleep, I find testing this difficult. Unfortunately there is no machine or environment object that could be injected to these classes as mocks that could take care of time-related fixtures. Thanks, Zoltán Tóth-Czifra
Re: Biweekly sqoop 2 calls
Thank you Abhijeet for your feedback. There weren't any other comments nor objections, so let's cancel the calls. Jarcec On Wed, Oct 03, 2012 at 01:15:58PM +0530, abhijeet gaikwad wrote: Sounds good. +1 Thanks, Abhijeet On 3 Oct 2012 06:13, Jarek Jarcec Cecho jar...@apache.org wrote: Hi Sqoop developers, we're having Sqoop 2 community calls since May and I believe that they significantly helped with kicking off entire project. However it appears that we do not have much topics to agenda lately as the high lever design discussions are already over and we moved our focus to development side. I would like to propose cancelling periodical calls for now (including tomorrow's one) and I would like to get other people opinion on this. I believe that we can always arrange them on demand when there will be important design decision that we did not covered yet. Jarcec signature.asc Description: Digital signature
[jira] [Created] (SQOOP-620) Introduce name to connection and job objects
Jarek Jarcec Cecho created SQOOP-620: Summary: Introduce name to connection and job objects Key: SQOOP-620 URL: https://issues.apache.org/jira/browse/SQOOP-620 Project: Sqoop Issue Type: Improvement Affects Versions: 2.0.0 Reporter: Jarek Jarcec Cecho Assignee: Jarek Jarcec Cecho Fix For: 2.0.0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Review Request: SQOOP-620: Introduce name to connection and job objects
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/7456/ --- Review request for Sqoop. Description --- I've added name to connection and job objects. This addresses bug SQOOP-620. https://issues.apache.org/jira/browse/SQOOP-620 Diffs - client/src/main/java/org/apache/sqoop/client/core/Constants.java PRE-CREATION client/src/main/java/org/apache/sqoop/client/core/Environment.java b496d43effa631d517deb54d36933f9ee389580d client/src/main/java/org/apache/sqoop/client/shell/CloneConnectionFunction.java 9716de6ee540aa1af4f817e373ded17c9a0e710f client/src/main/java/org/apache/sqoop/client/shell/CloneJobFunction.java 6c579d0965d52d29477a23e615099e2ec9093270 client/src/main/java/org/apache/sqoop/client/shell/CreateConnectionFunction.java 904624cf39496feaf28316fc5ed45ec5c1eb6cdf client/src/main/java/org/apache/sqoop/client/shell/CreateJobFunction.java 1a8b901a531852fcb684afaaf0d71f1b18e3bfc6 client/src/main/java/org/apache/sqoop/client/shell/UpdateConnectionFunction.java 1d642968555d41465b78824f536e1191dffa5be6 client/src/main/java/org/apache/sqoop/client/shell/UpdateJobFunction.java 9924cf9a4f0e6c7a226a3237febf8aa200ff1f73 client/src/main/java/org/apache/sqoop/client/utils/FormFiller.java 24045878a4020ea994b4c0069f3bf9fd8e3b437a client/src/main/resources/client-resource.properties PRE-CREATION core/src/main/java/org/apache/sqoop/connector/ConnectorManager.java ea368da53d055579e772e944f56773390ff83389 repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbyRepositoryHandler.java ef3d804588129dd14bf59c8e1693c95cf628f666 repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbySchemaQuery.java 8c0beeb60efbeaf65e89cc0e4d2f289c1ccd9c63 repository/repository-derby/src/test/java/org/apache/sqoop/repository/derby/TestConnectionHandling.java d1897957c938b5f57436379701141802270c4451 repository/repository-derby/src/test/java/org/apache/sqoop/repository/derby/TestJobHandling.java 8ca7e462b61726dc6e7aba3c5d4f4930e9be8963 Diff: https://reviews.apache.org/r/7456/diff/ Testing --- This patch do not include any new unit tests, I've tested it manually. Thanks, Jarek Cecho
[jira] [Updated] (SQOOP-620) Introduce name to connection and job objects
[ https://issues.apache.org/jira/browse/SQOOP-620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Jarcec Cecho updated SQOOP-620: - Attachment: SQOOP-620.patch Introduce name to connection and job objects Key: SQOOP-620 URL: https://issues.apache.org/jira/browse/SQOOP-620 Project: Sqoop Issue Type: Improvement Affects Versions: 2.0.0 Reporter: Jarek Jarcec Cecho Assignee: Jarek Jarcec Cecho Fix For: 2.0.0 Attachments: SQOOP-620.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (SQOOP-621) Requesting support for upsert export with MySQL
Raghav Kumar Gautam created SQOOP-621: - Summary: Requesting support for upsert export with MySQL Key: SQOOP-621 URL: https://issues.apache.org/jira/browse/SQOOP-621 Project: Sqoop Issue Type: Improvement Components: connectors/mysql Affects Versions: 1.4.2 Reporter: Raghav Kumar Gautam Sqoop export with upsert option gives the following error: 12/09/26 11:34:54 ERROR tool.ExportTool: Error during export: Mixed update/insert is not supported against the target database yet From the code it seems to me that upsert is implemented only for Oracle as of now. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-621) Requesting support for upsert export with MySQL
[ https://issues.apache.org/jira/browse/SQOOP-621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13470027#comment-13470027 ] Raghav Kumar Gautam commented on SQOOP-621: --- Also see this thread: http://mail-archives.apache.org/mod_mbox/sqoop-user/201210.mbox/%3C20121004152956.GT16616%40jarcec-thinkpad%3E Requesting support for upsert export with MySQL --- Key: SQOOP-621 URL: https://issues.apache.org/jira/browse/SQOOP-621 Project: Sqoop Issue Type: Improvement Components: connectors/mysql Affects Versions: 1.4.2 Reporter: Raghav Kumar Gautam Labels: newbie Sqoop export with upsert option gives the following error: 12/09/26 11:34:54 ERROR tool.ExportTool: Error during export: Mixed update/insert is not supported against the target database yet From the code it seems to me that upsert is implemented only for Oracle as of now. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
