Hi Pablo, I’m adding Sqoop user mailing list [email protected] - please sing up to this mailing list as instructed in [1].
If you use s3:/ as the underlaying filesystem, Sqoop uses Hadoop’s S3 implementation. I honestly do not know if the S3 implementation uses secured (~SSL) connection underneath even though I would expect that. Looking around internet it seems that these days people are suggesting to use s3a:// instead of s3:// [2]. Jarcec Links: 1: http://sqoop.apache.org/mail-lists.html 2: https://wiki.apache.org/hadoop/AmazonS3 > On Sep 16, 2015, at 2:50 AM, Pablo Torre <[email protected]> wrote: > > Hi Jarek, > > I need your help. > > I would like to ask you a question and see if you can help me. I have this > conf file: > > Hi, > > I have this config file "sqoop.conf": > > # > # Options file for Sqoop > # > # Specifies the tool being invoked > import > # Connect parameter and value > --connect > jdbc:sqlserver://xxx.xxx.x.xx\SQLEXPRESS;database=TEST > # Username parameter and value > --username > xxxx > --password > xxxx > --verbose > --target-dir > s3://...... > --split-by > age > --table > dbo.users > --fields-terminated-by > , > --input-fields-terminated-by > , > --check-column > updated_at > --incremental > lastmodified > --last-value > 2015-02-07 10:33:47 > > > It is working perfectly, I am using Sqoop to import data from a SQL Server > (through a VPN) to Amazon S3. I now that this connection is safe, but what > happen when Sqoop is moving this data to Amazon S3. Is it safe this > connection? Is using Sqoop underneath a security connection? I don't know how > it works. > > Best, > > > > > 2015-05-04 12:53 GMT+02:00 Pablo Torre <[email protected]>: > Hi, > > It works!! Thanks a lot for your help. > > > Best, > Pablo > > 2015-05-01 19:01 GMT+02:00 Jarek Jarcec Cecho <[email protected]>: > I believe that the problem is because you’re explicitly using the —driver > argument Pablo. Could you drop it? > > The —driver option is a bit deceptive in Sqoop and I generally recommend not > to use it at all unless absolutely required. You can read why in the > following blog post: > > http://blog.cloudera.com/blog/2013/09/understanding-connectors-and-drivers-in-the-world-of-sqoop/ > > Jarcec > > > On May 1, 2015, at 9:28 AM, Pablo Torre <[email protected]> wrote: > > > > Hi, > > > > I have this config file "sqoop.conf": > > > > # > > # Options file for Sqoop > > # > > # Specifies the tool being invoked > > import > > # Connect parameter and value > > --connect > > jdbc:sqlserver://xxx.xxx.x.xx\SQLEXPRESS;database=TEST > > # Username parameter and value > > --username > > xxxx > > --password > > xxxx > > --driver > > com.microsoft.sqlserver.jdbc.SQLServerDriver > > --verbose > > --target-dir > > /user/pablotorrerodriguez/sqlserverusers5/ > > --split-by > > age > > --table > > dbo.users > > --fields-terminated-by > > , > > --input-fields-terminated-by > > , > > --check-column > > updated_at > > --incremental > > lastmodified > > --last-value > > 2015-02-07 10:33:47 > > > > > > And I run this command line: > > > > sqoop --options-file sqoop.conf > > > > But I get this error "com.microsoft.sqlserver.jdbc.SQLServerException: > > Incorrent sintaxis near to ')'." I am using hadoop 1.2.1 and sqoop 1.4.5. > > And in the folder sqoop-1.4.5/lib I have the driver sqljdbc4.jar for > > Microsoft SQL Server. > > The incremental imports with Microsoft SQL Server give me problems. > > > > > > Best, > > > > > > 2015-05-01 16:28 GMT+02:00 Jarek Jarcec Cecho <[email protected]>: > > I’m adding [email protected] mailing list as this seems to be normal > > user question Please do register in the mailing list Pablo, you can find > > instructions in [1]. > > > > Could you share with us your entire Sqoop command line? > > > > Jarcec > > > > 1: http://sqoop.apache.org/mail-lists.html > > > > > On May 1, 2015, at 1:10 AM, Pablo Torre <[email protected]> wrote: > > > > > > Hi Jarek. > > > > > > I am taking a look at code in Sqoop 1.4.5 and in the java class > > > Sqlmanager.java I can see that it is using "SELECT TIMESTAMP;" instead of > > > use "SELECT TIMESTAMP()". But I still get the error > > > "com.microsoft.sqlserver.jdbc.SQLServerException: Incorrent sintaxis near > > > to ')'." > > > > > > I was testing it in Microsoft SQL Server Management and If I run the > > > query "SELECT TIMESTAMP()" I get the same error. > > > > > > I don't know why sqoop is trying to execute the wrong query if the source > > > code is right. > > > > > > Please help me with this. > > > > > > > > > > > > 2015-04-20 16:41 GMT+02:00 Jarek Jarcec Cecho <[email protected]>: > > > I believe that this is already fixed in Sqoop 1.4.5. I would suggest to > > > sign up to Sqoop user mailing list (instructions are at [1]) and ask for > > > help :) > > > > > > Jarcec > > > > > > Links: > > > 1: http://sqoop.apache.org/mail-lists.html > > > > > > > On Apr 19, 2015, at 2:12 PM, Pablo Torre <[email protected]> wrote: > > > > > > > > Thanks Jarek, > > > > > > > > I was using Sqoop 1.4.5, maybe I have to install a new version or > > > > update the code. > > > > > > > > Best, > > > > Pablo > > > > > > > > 2015-04-19 20:42 GMT+02:00 Jarek Jarcec Cecho <[email protected]>: > > > > Hi Pablo, > > > > I’m always excited to see a new contributor on Sqoop project. Take a > > > > look at quick “How to contribute” wiki page that contains all the > > > > details about how to contribute to Sqoop: > > > > > > > > https://cwiki.apache.org/confluence/display/SQOOP/How+to+Contribute > > > > > > > > For your particular issue, I feel that it’s long solved as the > > > > Microsoft Sql connector is already using the correct query, check out > > > > the latest source code: > > > > > > > > https://github.com/apache/sqoop/blob/trunk/src/java/org/apache/sqoop/manager/SQLServerManager.java#L251 > > > > > > > > Jarcec > > > > > > > > > On Apr 17, 2015, at 2:24 AM, Pablo Torre <[email protected]> > > > > > wrote: > > > > > > > > > > Hi Jarcec, > > > > > > > > > > My name is Pablo. I am writing this email because I saw that you are > > > > > working on the project sqoop on github. > > > > > https://github.com/apache/sqoop > > > > > > > > > > There is a bug that I would like to share with you and maybe you can > > > > > help me and say me how can I modified the source code or contribute > > > > > to the project: > > > > > > > > > > I am trying to use incremental imports with sqoop using the option > > > > > lastmodified, but with MS SQLSERVER fails. I found this article about > > > > > it > > > > > https://issues.cloudera.org/browse/SQOOP-229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel > > > > > > > > > > "I see in the source code for SqlManager.java that the query being > > > > > used in getCurTimestampQuery() is "select CURRENT_TIMESTAMP()" this > > > > > is not compatible with MSSQL - for MSSQL it would be just "select > > > > > CURRENT_TIMESTAMP" (keyword not a function)." > > > > > > > > > > I would like to modify that class and compile the code to get it > > > > > working. But I don't know how to do it. > > > > > > > > > > Do you know if there is any patch for it? > > > > > If you can help me with this issue. > > > > > > > > > > Thanks. > > > > > > > > > > My Github: https://github.com/sylvinho81 > > > > > > > > > > -- > > > > > Pablo Torre. > > > > > Freelance software engineer and Ruby on Rails developer. > > > > > Oleiros (Coruña) > > > > > Personal site > > > > > My blog > > > > > > > > > > > > > > > > > > > > -- > > > > Pablo Torre. > > > > Freelance software engineer and Ruby on Rails developer. > > > > Oleiros (Coruña) > > > > Personal site > > > > My blog > > > > > > > > > > > > > > > -- > > > Pablo Torre. > > > Freelance software engineer and Ruby on Rails developer. > > > Oleiros (Coruña) > > > Personal site > > > My blog > > > > > > > > > > -- > > Pablo Torre. > > Freelance software engineer and Ruby on Rails developer. > > Oleiros (Coruña) > > Personal site > > My blog > > > > > -- > Pablo Torre. > Freelance software engineer and Ruby on Rails developer. > Oleiros (Coruña) > Personal site > My blog > > > > -- > Pablo Torre. > Freelance software engineer and Ruby on Rails developer. > Oleiros (Coruña) > Personal site > My blog
