Re: Review Request 58466: SQOOP-3158 - Columns added to Mysql after initial sqoop import, export back to table with same schema fails

2017-05-21 Thread Eric Lin 


> On May 9, 2017, 3:09 p.m., Szabolcs Vasas wrote:
> > src/java/org/apache/sqoop/orm/ClassWriter.java
> > Lines 1416 (patched)
> > 
> >
> > I think the hard coded "null" value will not work if 
> > the user overwrites the null string with the input-null-string Sqoop 
> > parameter. You could use this.options.getInNullStringValue() instead of 
> > "null"  like in org.apache.sqoop.orm.ClassWriter#parseNullVal method.

Hi Szabolcs,

Thanks for the review and good pickup. I have updated it and also added a new 
test case to cover it.


- Eric


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58466/#review174317
---


On May 6, 2017, 11:29 a.m., Eric Lin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58466/
> ---
> 
> (Updated May 6, 2017, 11:29 a.m.)
> 
> 
> Review request for Sqoop, Attila Szabo and Szabolcs Vasas.
> 
> 
> Bugs: SQOOP-3158
> https://issues.apache.org/jira/browse/SQOOP-3158
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> I have table in MySQL with 2 columns until yesterday. The columns are id and 
> name.
> 1,Raj
> 2,Jack
> 
> I have imported this data into HDFS yesterday itself as a file. Today we 
> added a new column to the table in MySQL called salary. The table looks like 
> below.
> 
> 1,Raj
> 2,Jack
> 3,Jill,2000
> 4,Nick,3000
> 
> Now I have done Incremental import on this table as a file.
> 
> Part-m-0 file contains
> 1,Raj
> 2,Jack
> 
> Part-m-1 file contains
> 3,Jill,2000
> 4,Nick,3000
> 
> Now I created a new table in MySQL with same schema as Original MySQL table 
> with columns id name and salary.
> 
> Sqoop export will fail with below error:
> 
> java.lang.RuntimeException: Can't parse input data: 'Raj'
> at SQOOP_3158.__loadFromFields(SQOOP_3158.java:316)
> at SQOOP_3158.parse(SQOOP_3158.java:254)
> at 
> org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:89)
> at 
> org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:39)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
> at 
> org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: java.util.NoSuchElementException
> at java.util.ArrayList$Itr.next(ArrayList.java:854)
> at SQOOP_3158.__loadFromFields(SQOOP_3158.java:311)
> ... 12 more
> 
> 
> Diffs
> -
> 
>   src/java/org/apache/sqoop/orm/ClassWriter.java eaa9123 
>   src/test/com/cloudera/sqoop/TestExport.java b2edc53 
> 
> 
> Diff: https://reviews.apache.org/r/58466/diff/3/
> 
> 
> Testing
> ---
> 
> There is no existing test class to cover the path and I am not sure the best 
> way to add test case for this. If you have any suggestion, please let me know.
> 
> I have done manual testing to replicate the issue and confirmed that patch 
> fixed the issue. I have also tried different data types, all working.
> 
> However, if column in MySQL is defined as NOT NULL, then the export will 
> still fail with error, this is expected.
> 
> 
> Thanks,
> 
> Eric Lin
> 
>

Re: Review Request 58466: SQOOP-3158 - Columns added to Mysql after initial sqoop import, export back to table with same schema fails

2017-05-21 Thread Eric Lin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58466/
---

(Updated May 21, 2017, 11:12 a.m.)


Review request for Sqoop, Attila Szabo and Szabolcs Vasas.


Changes
---

update from hardcoded "null" value to this.options.getInNullStringValue() 
function call, and also added a new test case.


Bugs: SQOOP-3158
https://issues.apache.org/jira/browse/SQOOP-3158


Repository: sqoop-trunk


Description
---

I have table in MySQL with 2 columns until yesterday. The columns are id and 
name.
1,Raj
2,Jack

I have imported this data into HDFS yesterday itself as a file. Today we added 
a new column to the table in MySQL called salary. The table looks like below.

1,Raj
2,Jack
3,Jill,2000
4,Nick,3000

Now I have done Incremental import on this table as a file.

Part-m-0 file contains
1,Raj
2,Jack

Part-m-1 file contains
3,Jill,2000
4,Nick,3000

Now I created a new table in MySQL with same schema as Original MySQL table 
with columns id name and salary.

Sqoop export will fail with below error:

java.lang.RuntimeException: Can't parse input data: 'Raj'
at SQOOP_3158.__loadFromFields(SQOOP_3158.java:316)
at SQOOP_3158.parse(SQOOP_3158.java:254)
at 
org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:89)
at 
org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:39)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at 
org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.util.NoSuchElementException
at java.util.ArrayList$Itr.next(ArrayList.java:854)
at SQOOP_3158.__loadFromFields(SQOOP_3158.java:311)
... 12 more


Diffs (updated)
-

  src/java/org/apache/sqoop/orm/ClassWriter.java eaa9123 
  src/test/com/cloudera/sqoop/TestExport.java b2edc53 


Diff: https://reviews.apache.org/r/58466/diff/4/

Changes: https://reviews.apache.org/r/58466/diff/3-4/


Testing
---

There is no existing test class to cover the path and I am not sure the best 
way to add test case for this. If you have any suggestion, please let me know.

I have done manual testing to replicate the issue and confirmed that patch 
fixed the issue. I have also tried different data types, all working.

However, if column in MySQL is defined as NOT NULL, then the export will still 
fail with error, this is expected.


Thanks,

Eric Lin

[jira] [Comment Edited] (SQOOP-3158) Columns added to Mysql after initial sqoop import, export back to table with same schema fails

2017-05-21 Thread Eric Lin (JIRA) 

[ 
https://issues.apache.org/jira/browse/SQOOP-3158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16018783#comment-16018783
 ] 

Eric Lin edited comment on SQOOP-3158 at 5/21/17 11:10 AM:
---

Latest patch based on review: SQOOP-3158.3.patch


was (Author: ericlin):
Latest patch based on review.

> Columns added to Mysql after initial sqoop import, export back to table with 
> same schema fails 
> ---
>
> Key: SQOOP-3158
> URL: https://issues.apache.org/jira/browse/SQOOP-3158
> Project: Sqoop
>  Issue Type: Improvement
>Affects Versions: 1.4.6
>Reporter: viru reddy
>Assignee: Eric Lin
>  Labels: newbie
> Attachments: SQOOP-3158.2.patch, SQOOP-3158.3.patch, SQOOP-3158.patch
>
>
> I have table in MySQL with 2 columns until yesterday. The columns are id and 
> name.
> 1,Raj
> 2,Jack
> I have imported this data into HDFS yesterday itself as a file. Today we 
> added a new column to the table in MySQL called salary. The table looks like 
> below.
> 1,Raj
> 2,Jack
> 3,Jill,2000
> 4,Nick,3000
> Now I have done Incremental import on this table as a file.
> Part-m-0 file contains
> 1,Raj
> 2,Jack
> Part-m-1 file contains
> 3,Jill,2000
> 4,Nick,3000
> Now I created a new table in MySQL with same schema as Original MySQL table 
> with columns id name and salary.
> When I do sqoop export only last 2 rows are getting inserted to the new table 
> in MySQL  and the sqoop export fails
> How can I reflect all the rows to be inserted to the table.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (SQOOP-3158) Columns added to Mysql after initial sqoop import, export back to table with same schema fails

2017-05-21 Thread Eric Lin (JIRA) 

 [ 
https://issues.apache.org/jira/browse/SQOOP-3158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Lin updated SQOOP-3158:

Attachment: SQOOP-3158.3.patch

Latest patch based on review.

> Columns added to Mysql after initial sqoop import, export back to table with 
> same schema fails 
> ---
>
> Key: SQOOP-3158
> URL: https://issues.apache.org/jira/browse/SQOOP-3158
> Project: Sqoop
>  Issue Type: Improvement
>Affects Versions: 1.4.6
>Reporter: viru reddy
>Assignee: Eric Lin
>  Labels: newbie
> Attachments: SQOOP-3158.2.patch, SQOOP-3158.3.patch, SQOOP-3158.patch
>
>
> I have table in MySQL with 2 columns until yesterday. The columns are id and 
> name.
> 1,Raj
> 2,Jack
> I have imported this data into HDFS yesterday itself as a file. Today we 
> added a new column to the table in MySQL called salary. The table looks like 
> below.
> 1,Raj
> 2,Jack
> 3,Jill,2000
> 4,Nick,3000
> Now I have done Incremental import on this table as a file.
> Part-m-0 file contains
> 1,Raj
> 2,Jack
> Part-m-1 file contains
> 3,Jill,2000
> 4,Nick,3000
> Now I created a new table in MySQL with same schema as Original MySQL table 
> with columns id name and salary.
> When I do sqoop export only last 2 rows are getting inserted to the new table 
> in MySQL  and the sqoop export fails
> How can I reflect all the rows to be inserted to the table.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)