[jira] [Updated] (SQOOP-2464) Initializer object is not reused when calling getSchema

[Abraham Fine (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-2464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abraham Fine updated SQOOP-2464:

Attachment: (was: SQOOP-2464.patch)

> Initializer object is not reused when calling getSchema
> ---
>
> Key: SQOOP-2464
> URL: https://issues.apache.org/jira/browse/SQOOP-2464
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.99.6
>Reporter: David Robson
>Assignee: Abraham Fine
> Fix For: 1.99.7
>
>
> In JobManager there is two methods which are called one after the other - 
> "initializeConnector" and "getSchemaForConnector". Both these methods do the 
> same thing as the first step - create a new instance of the initializer class.
> If the same instance of the initializer was shared it means the class could 
> keep resources open (such as a connection to the database) and not have to 
> re-establish the connection. This might mean a close method needs to be added 
> to the initializers as otherwise the getSchema would need to close any 
> resources opened in the initialize call - which might seem a bit confusing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (SQOOP-2464) Initializer object is not reused when calling getSchema

[Abraham Fine (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-2464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abraham Fine updated SQOOP-2464:

Attachment: (was: SQOOP-2464.patch)

> Initializer object is not reused when calling getSchema
> ---
>
> Key: SQOOP-2464
> URL: https://issues.apache.org/jira/browse/SQOOP-2464
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.99.6
>Reporter: David Robson
>Assignee: Abraham Fine
> Fix For: 1.99.7
>
>
> In JobManager there is two methods which are called one after the other - 
> "initializeConnector" and "getSchemaForConnector". Both these methods do the 
> same thing as the first step - create a new instance of the initializer class.
> If the same instance of the initializer was shared it means the class could 
> keep resources open (such as a connection to the database) and not have to 
> re-establish the connection. This might mean a close method needs to be added 
> to the initializers as otherwise the getSchema would need to close any 
> resources opened in the initialize call - which might seem a bit confusing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (SQOOP-2464) Initializer object is not reused when calling getSchema

[Sqoop QA bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-2464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14746635#comment-14746635
 ] 

Sqoop QA bot commented on SQOOP-2464:
-

Testing file 
[SQOOP-2464.patch|https://issues.apache.org/jira/secure/attachment/12756127/SQOOP-2464.patch]
 against branch sqoop2 took 0:53:30.092794.

{color:red}Overall:{color} -1 due to an error(s), see details below:

{color:green}SUCCESS:{color} Clean was successful
{color:green}SUCCESS:{color} Patch applied correctly
{color:red}ERROR:{color} Patch does not add/modify any test case
{color:green}SUCCESS:{color} License check passed
{color:green}SUCCESS:{color} Patch compiled
{color:green}SUCCESS:{color} All unit tests passed (executed 1432 tests)
{color:green}SUCCESS:{color} Test coverage did not decreased 
([report|https://builds.apache.org/job/PreCommit-SQOOP-Build/1691/artifact/patch-process/cobertura_report.txt])
{color:green}SUCCESS:{color} No new findbugs warnings 
([report|https://builds.apache.org/job/PreCommit-SQOOP-Build/1691/artifact/patch-process/findbugs_report.txt])
{color:green}SUCCESS:{color} All integration tests passed (executed 79 tests)

Console output is available 
[here|https://builds.apache.org/job/PreCommit-SQOOP-Build/1691/console].

This message is automatically generated.

> Initializer object is not reused when calling getSchema
> ---
>
> Key: SQOOP-2464
> URL: https://issues.apache.org/jira/browse/SQOOP-2464
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.99.6
>Reporter: David Robson
>Assignee: Abraham Fine
> Fix For: 1.99.7
>
> Attachments: SQOOP-2464.patch
>
>
> In JobManager there is two methods which are called one after the other - 
> "initializeConnector" and "getSchemaForConnector". Both these methods do the 
> same thing as the first step - create a new instance of the initializer class.
> If the same instance of the initializer was shared it means the class could 
> keep resources open (such as a connection to the database) and not have to 
> re-establish the connection. This might mean a close method needs to be added 
> to the initializers as otherwise the getSchema would need to close any 
> resources opened in the initialize call - which might seem a bit confusing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Classpath isolation in Sqoop2

[Fu, Dian]

Hi all,


Currently there is no classpath isolation in Sqoop 2. This will cause problems 
in the following two cases:

1)  if the dependencies of the downstream users of the Sqoop 2 client conflicts 
with the dependencies of Sqoop 2 client

2)  if the dependencies of third-part connectors conflicts with the 
dependencies of Sqoop 2 server or conflicts with other third-part connectors



I'd like to provide classpath isolation in Sqoop 2 and have taken some time to 
investigate the status of classpath isolation in Hadoop. Here is a simple 
summary of the problem and the solution proposed by Sean in 
HADOOP-11656:

The problems HADOOP-11656 tries to solve:

   1) Client side classpath isolation: between Hadoop and its downside 
applications which talk directly with HDFS or submit YARN applications.

  2) Framework level classpath isolation: between YARN server and 
ApplicationMaster or between YARN and user application. There is already a 
solution in Hadoop to solve this issue which uses webapp-style classloader 
named ApplicationClassLoader (parent last).

The solution proposed in HADOOP-11656 by Sean:

   1) For the client side classpath isolation, Sean proposes to use Maven Shade 
Plugin to expose only the public API to clients and use the Maven Shade Plugin 
relocation capacity to relocate other dependencies under the package 
org.apache.hadoop.shaded. (Refer to 
HADOOP-11804 for details)

   2) For the existing webapp-style classloader solution for framework level 
classpath isolation, Sean pointed out it doesn't provide much upgrade help for 
applications that rely on the classes found in the fallback case. That is to 
say, if user code relied on a Hadoop dependency implicitly and Hadoop upgraded 
it to an incompatible version, problems will be caused. Sean proposes to use 
OSGi container to export different set of dependencies in different Hadoop 
versions to solve this issue. (more discussion about this can be found 
here)



Based on the understanding of the above, for the classpath isolation problem in 
Sqoop 2, it can be separated into two parts:

1)  client side classpath isolation

2)  isolation for connectors

And we have three options to consider:

1)  Maven Shade Plugin

2)  Webapp-style classloader

3)  OSGi

I'd like to use Maven Shade Plugin to solve the client side classpath isolation 
problem in the similar way done in 
HADOOP-11804.

For the isolation for connectors, Maven shade plugin won't be an option as it 
isolates the classpath via relocation capacity at build time and it can't 
relocate connectors dependencies at runtime. Between option webapp-style 
classloader and OSGi, we may need to choose OSGi if we want to upgrade Sqoop 2 
dependencies without affecting third-part connectors in the case that 
third-part connectors rely on some Sqoop 2 dependencies implicitly. But if we 
think that requiring third-part connectors to upgrade accordingly is 
acceptable, I would prefer webapp-style classloader as it is easier to 
implement compared to OSGi.



Please feel free to provide your opinions, thanks a lot.



Regards,

Dian

Re: Review Request 38089: SQOOP-2335 : Hcatalog external table support

[Venkat Ranganathan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38089/#review99165
---


Thanks for submitting this.   Please see couple of small issues


src/java/org/apache/sqoop/util/HiveUtil.java (line 39)


None of the following 4 constants are used



src/java/org/apache/sqoop/util/HiveUtil.java (line 61)


Not sure what the long name means - can we have simpler descriptive name - 
may be split into two and probably add a comment?


- Venkat Ranganathan


On Sept. 14, 2015, 11:51 p.m., Rakesh Sharma wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38089/
> ---
> 
> (Updated Sept. 14, 2015, 11:51 p.m.)
> 
> 
> Review request for Sqoop and Venkat Ranganathan.
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> Sqoop currently doesnt support creating external tables while importing data 
> to hcatalog tables. This patch adds support to create hcatalog tables ehile 
> importing data.
> 
> 
> Diffs
> -
> 
>   src/docs/user/hcatalog.txt 99ae4f5 
>   src/java/org/apache/sqoop/SqoopOptions.java ace90fd 
>   src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java 4070c24 
>   src/java/org/apache/sqoop/tool/BaseSqoopTool.java a7f6aec 
>   src/java/org/apache/sqoop/tool/ImportTool.java 39af42c 
>   src/java/org/apache/sqoop/util/HiveUtil.java PRE-CREATION 
>   src/test/org/apache/sqoop/hcat/HCatalogExportTest.java 8aa0725 
>   src/test/org/apache/sqoop/hcat/HCatalogImportTest.java d97b870 
>   src/test/org/apache/sqoop/hcat/HCatalogTestUtils.java 32c267f 
>   src/test/org/apache/sqoop/util/TestHiveUtil.java PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/38089/diff/
> 
> 
> Testing
> ---
> 
> YES
> 
> 
> Thanks,
> 
> Rakesh Sharma
> 
>

[jira] [Commented] (SQOOP-2464) Initializer object is not reused when calling getSchema

[Abraham Fine (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-2464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14746509#comment-14746509
 ] 

Abraham Fine commented on SQOOP-2464:
-

adding patch to prevent re-instantiation of initializer class

> Initializer object is not reused when calling getSchema
> ---
>
> Key: SQOOP-2464
> URL: https://issues.apache.org/jira/browse/SQOOP-2464
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.99.6
>Reporter: David Robson
> Fix For: 1.99.7
>
> Attachments: SQOOP-2464.patch
>
>
> In JobManager there is two methods which are called one after the other - 
> "initializeConnector" and "getSchemaForConnector". Both these methods do the 
> same thing as the first step - create a new instance of the initializer class.
> If the same instance of the initializer was shared it means the class could 
> keep resources open (such as a connection to the database) and not have to 
> re-establish the connection. This might mean a close method needs to be added 
> to the initializers as otherwise the getSchema would need to close any 
> resources opened in the initialize call - which might seem a bit confusing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (SQOOP-2464) Initializer object is not reused when calling getSchema

[Abraham Fine (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-2464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abraham Fine reassigned SQOOP-2464:
---

Assignee: Abraham Fine  (was: Jarek Jarcec Cecho)

> Initializer object is not reused when calling getSchema
> ---
>
> Key: SQOOP-2464
> URL: https://issues.apache.org/jira/browse/SQOOP-2464
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.99.6
>Reporter: David Robson
>Assignee: Abraham Fine
> Fix For: 1.99.7
>
> Attachments: SQOOP-2464.patch
>
>
> In JobManager there is two methods which are called one after the other - 
> "initializeConnector" and "getSchemaForConnector". Both these methods do the 
> same thing as the first step - create a new instance of the initializer class.
> If the same instance of the initializer was shared it means the class could 
> keep resources open (such as a connection to the database) and not have to 
> re-establish the connection. This might mean a close method needs to be added 
> to the initializers as otherwise the getSchema would need to close any 
> resources opened in the initialize call - which might seem a bit confusing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (SQOOP-2464) Initializer object is not reused when calling getSchema

[Abraham Fine (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-2464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abraham Fine updated SQOOP-2464:

Attachment: SQOOP-2464.patch

> Initializer object is not reused when calling getSchema
> ---
>
> Key: SQOOP-2464
> URL: https://issues.apache.org/jira/browse/SQOOP-2464
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.99.6
>Reporter: David Robson
> Fix For: 1.99.7
>
> Attachments: SQOOP-2464.patch
>
>
> In JobManager there is two methods which are called one after the other - 
> "initializeConnector" and "getSchemaForConnector". Both these methods do the 
> same thing as the first step - create a new instance of the initializer class.
> If the same instance of the initializer was shared it means the class could 
> keep resources open (such as a connection to the database) and not have to 
> re-establish the connection. This might mean a close method needs to be added 
> to the initializers as otherwise the getSchema would need to close any 
> resources opened in the initialize call - which might seem a bit confusing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 38400: Review for issue SQOOP-2332: Support dynamic partitions while sqoop import. Also support partition key formats for date partitions

[Venkat Ranganathan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38400/#review99160
---



src/docs/user/hcatalog.txt (line 113)


I think this can be eloborated a bit.   Sqoop already supports dynamic 
partitions - as you can see from the tests that we have.   What is missing 
today is the ability to identify during table creation that some partitions 
should be added to the partition by clause alone (without being in the static 
partition key=value spec list).



src/docs/user/hcatalog.txt (line 129)


I think this is a separate feature by itself that I think should be in a 
separate JIRA?



src/java/org/apache/sqoop/util/DbUtil.java (line 40)


Not sure why these are needed.   Don't we have columninfo queries that 
support most of the needs including support for column precision?


- Venkat Ranganathan


On Sept. 15, 2015, 11:02 a.m., Rakesh Sharma wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38400/
> ---
> 
> (Updated Sept. 15, 2015, 11:02 a.m.)
> 
> 
> Review request for Sqoop and Venkat Ranganathan.
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> Review request for support of Dynamic partitons while importing data. Users 
> can just tell the list of columns based on which partitioning should happen. 
> Also support for date formats is added if user wants to do partitioning based 
> on a date column.
> 
> 
> Diffs
> -
> 
>   src/docs/user/hcatalog.txt 99ae4f5 
>   src/java/org/apache/sqoop/SqoopOptions.java ace90fd 
>   src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatImportHelper.java 878f765 
>   src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java 4070c24 
>   src/java/org/apache/sqoop/metastore/TableMetaData.java PRE-CREATION 
>   src/java/org/apache/sqoop/tool/BaseSqoopTool.java a7f6aec 
>   src/java/org/apache/sqoop/tool/ImportTool.java 39af42c 
>   src/java/org/apache/sqoop/util/DbUtil.java PRE-CREATION 
>   src/test/org/apache/sqoop/hcat/HCatalogImportTest.java d97b870 
> 
> Diff: https://reviews.apache.org/r/38400/diff/
> 
> 
> Testing
> ---
> 
> Yes
> 
> 
> Thanks,
> 
> Rakesh Sharma
> 
>

[jira] [Updated] (SQOOP-2464) Initializer object is not reused when calling getSchema

[Abraham Fine (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-2464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abraham Fine updated SQOOP-2464:

Attachment: SQOOP-2464.patch

> Initializer object is not reused when calling getSchema
> ---
>
> Key: SQOOP-2464
> URL: https://issues.apache.org/jira/browse/SQOOP-2464
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.99.6
>Reporter: David Robson
>Assignee: Abraham Fine
> Fix For: 1.99.7
>
> Attachments: SQOOP-2464.patch, SQOOP-2464.patch
>
>
> In JobManager there is two methods which are called one after the other - 
> "initializeConnector" and "getSchemaForConnector". Both these methods do the 
> same thing as the first step - create a new instance of the initializer class.
> If the same instance of the initializer was shared it means the class could 
> keep resources open (such as a connection to the database) and not have to 
> re-establish the connection. This might mean a close method needs to be added 
> to the initializers as otherwise the getSchema would need to close any 
> resources opened in the initialize call - which might seem a bit confusing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (SQOOP-969) sqoop export with mysql connector uses more than twice as many mappers as the --num-mappers hint

[Senthil Bogana (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14746530#comment-14746530
 ] 

Senthil Bogana commented on SQOOP-969:
--

I tried multiple times , but I am unable to reproduce the issues. 
Could you please reconfirm the issue still exists. 

> sqoop export with mysql connector uses more than twice as many mappers as the 
> --num-mappers hint
> 
>
> Key: SQOOP-969
> URL: https://issues.apache.org/jira/browse/SQOOP-969
> Project: Sqoop
>  Issue Type: Bug
>  Components: connectors
>Affects Versions: 1.4.3
>Reporter: Will McQueen
>  Labels: newbie
>
> sudo -u sqoop sqoop export --connect "jdbc:mysql:///sqoop" 
> --username sqoop --password sqoop --table quicktest --verbose --export-dir 
> /user/sqoop/quicktest --num-mappers 12
> ...results in more than twice the number of mappers (according to the logs) 
> as the specified '12' hint:
> 13/03/26 16:20:35 INFO mapred.JobClient:   Job Counters 
> 13/03/26 16:20:35 INFO mapred.JobClient: Launched map tasks=23
> ...and I couldn't help notice that the logs also say:
> 13/03/26 16:19:03 DEBUG mapreduce.ExportInputFormat: Target numMapTasks=12
> 13/03/26 16:19:03 DEBUG mapreduce.ExportInputFormat: Total input bytes=23
> ...where # input bytes maches # launched map tasks. Coincidence, or relation?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 38323: BigDecimalSplitter java.lang.ArrayIndexOutOfBoundsException

[Venkat Ranganathan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38323/#review99141
---


Thanks Sowmya Ramesh for working on this.   Just one small nit


src/java/org/apache/sqoop/mapreduce/db/BigDecimalSplitter.java (line 144)


Can you add a comment saying when splits.size can be 1?   Essentially if 
the collation and sort orders differ we can hit this


- Venkat Ranganathan


On Sept. 11, 2015, 3:15 p.m., Sowmya Ramesh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38323/
> ---
> 
> (Updated Sept. 11, 2015, 3:15 p.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-2399
> https://issues.apache.org/jira/browse/SQOOP-2399
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> With a relatively small table and a text primary key this error happens:
> 
> 15/06/14 03:00:25 WARN db.BigDecimalSplitter: Set BigDecimal splitSize to 
> MIN_INCREMENT
> 15/06/14 03:00:25 INFO mapred.JobClient: Cleaning up the staging area 
> maprfs:/var/mapr/cluster/mapred/jobTracker/staging/mapr/.staging/job_201404251052_5654
> 15/06/14 03:00:25 ERROR sqoop.Sqoop: Got exception running Sqoop: 
> java.lang.ArrayIndexOutOfBoundsException: -1
> java.lang.ArrayIndexOutOfBoundsException: -1
> at java.util.ArrayList.elementData(ArrayList.java:371)
> at java.util.ArrayList.get(ArrayList.java:384)
> at 
> org.apache.sqoop.mapreduce.db.BigDecimalSplitter.split(BigDecimalSplitter.java:143)
> at org.apache.sqoop.mapreduce.db.TextSplitter.split(TextSplitter.java:151)
> at org.apache.sqoop.mapreduce.db.TextSplitter.split(TextSplitter.java:113)
> 
> 
> Diffs
> -
> 
>   src/java/org/apache/sqoop/mapreduce/db/BigDecimalSplitter.java ebe6c40 
>   src/java/org/apache/sqoop/mapreduce/db/DBSplitter.java b121d4b 
>   src/java/org/apache/sqoop/mapreduce/db/DataDrivenDBInputFormat.java db96e41 
>   src/java/org/apache/sqoop/mapreduce/db/TextSplitter.java d3085cd 
>   src/test/org/apache/sqoop/mapreduce/db/TestTextSplitter.java 9c007d3 
> 
> Diff: https://reviews.apache.org/r/38323/diff/
> 
> 
> Testing
> ---
> 
> Unit test
> 
> 
> Thanks,
> 
> Sowmya Ramesh
> 
>

[jira] [Updated] (SQOOP-2335) Support for Hive External Table in Sqoop - HCatalog

[Rakesh Sharma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-2335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rakesh Sharma updated SQOOP-2335:
-
Attachment: SQOOP-2335_1.patch

Making some changes in the test cases for the column types.

> Support for Hive External Table in Sqoop - HCatalog
> ---
>
> Key: SQOOP-2335
> URL: https://issues.apache.org/jira/browse/SQOOP-2335
> Project: Sqoop
>  Issue Type: New Feature
>Affects Versions: 1.4.5
>Reporter: Atul Gupta
>Assignee: Rakesh Sharma
> Fix For: 1.4.7
>
> Attachments: SQOOP-2335_0.patch, SQOOP-2335_1.patch
>
>
> Currently Apache Sqoop tool support only Hive managed table using Hcatalog. 
> it would be nice if we have support to create a Hive external table in Sqoop 
> Hcatalog.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 38089: SQOOP-2335 : Hcatalog external table support

[Rakesh Sharma]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38089/
---

(Updated Sept. 15, 2015, 6:51 a.m.)


Review request for Sqoop and Venkat Ranganathan.


Changes
---

Making some changes in the test cases


Repository: sqoop-trunk


Description
---

Sqoop currently doesnt support creating external tables while importing data to 
hcatalog tables. This patch adds support to create hcatalog tables ehile 
importing data.


Diffs (updated)
-

  src/docs/user/hcatalog.txt 99ae4f5 
  src/java/org/apache/sqoop/SqoopOptions.java ace90fd 
  src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java 4070c24 
  src/java/org/apache/sqoop/tool/BaseSqoopTool.java a7f6aec 
  src/java/org/apache/sqoop/tool/ImportTool.java 39af42c 
  src/java/org/apache/sqoop/util/HiveUtil.java PRE-CREATION 
  src/test/org/apache/sqoop/hcat/HCatalogExportTest.java 8aa0725 
  src/test/org/apache/sqoop/hcat/HCatalogImportTest.java d97b870 
  src/test/org/apache/sqoop/hcat/HCatalogTestUtils.java 32c267f 
  src/test/org/apache/sqoop/util/TestHiveUtil.java PRE-CREATION 

Diff: https://reviews.apache.org/r/38089/diff/


Testing
---

YES


Thanks,

Rakesh Sharma

Sqoop and Spark

[Aitor Argomaniz]

Hi all,

I've been taking a look at Sqoop on Spark PoC made by Veena Basavaraj:
https://spark-summit.org/2015/events/Sqoop-on-Spark-for-Data-Ingestion/

As you know in order to use Sqoop in environments with existing Spark
deployments it would be great Sqoop in on top of Spark.

I've been reviewing the docs and found some references about support it in
the future, and also i've seen there is an open issue about that since a
year ago:
https://issues.apache.org/jira/browse/SQOOP-1532

Is there any plan in the near future to support Spark in Sqoop?

Thanks in advance.
Best Regards

[jira] [Updated] (SQOOP-2332) Dynamic Partition in Sqoop HCatalog- if Hive table does not exists & add support for Partition Date Format

[Rakesh Sharma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-2332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rakesh Sharma updated SQOOP-2332:
-
Attachment: SQOOP-2332_1.patch

Uploading new patch with cleaning extra spaces.

> Dynamic Partition in Sqoop HCatalog- if Hive table does not exists & add 
> support for Partition Date Format
> --
>
> Key: SQOOP-2332
> URL: https://issues.apache.org/jira/browse/SQOOP-2332
> Project: Sqoop
>  Issue Type: New Feature
>Affects Versions: 1.4.5
>Reporter: Atul Gupta
>Assignee: Rakesh Sharma
> Fix For: 1.4.7
>
> Attachments: SQOOP-2332_0.patch, SQOOP-2332_1.patch
>
>
> Apache Sqoop 1.4.5 with Hcatalog supports Dynamic partitions when hive hive 
> partition table already exists. it would be nice if we have Dynamic Partition 
> support in Apache Sqoop Hcatalog even if Hive table does not exist during 
> data import. as a suggestion could add option like --hive-dynamic-partition 
> Also there is no support for date formats on partition column having type 
> like  date/time/timestamp. As a suggestion it would be great if we add Option 
> like  --hive-partition-key-format to support the partition date formats   
> based on java simple available date formats like -MM-dd, -MM,  
> etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (SQOOP-1369) Avro export ignores --columns option

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-1369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14745567#comment-14745567
 ] 

ASF subversion and git services commented on SQOOP-1369:


Commit 5fc7a680f854c4a03e9a581079172d453d1a059c in sqoop's branch 
refs/heads/trunk from [~jarcec]
[ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=5fc7a68 ]

SQOOP-1369: Avro export ignores --columns option

(Paul Mazak via Jarek Jarcec Cecho)


> Avro export ignores --columns option
> 
>
> Key: SQOOP-1369
> URL: https://issues.apache.org/jira/browse/SQOOP-1369
> Project: Sqoop
>  Issue Type: Bug
>Reporter: Lars Francke
>Assignee: Paul Mazak
> Fix For: 1.4.7
>
> Attachments: SQOOP-1369.patch, SQOOP-1369_2.patch, 
> SQOOP-1369_3.patch, SQOOP-1369_4.patch, SQOOP-1369_5.patch
>
>
> In JdbcExportJob AVRO_COLUMN_TYPES_MAP is being set with the full schema of 
> the output table. This causes the AvroExportMapper to fail with unknown 
> fields if --columns was used to restrict the columns to export (it then tries 
> to set a value on the generated class which doesn't exist).
> There are multiple ways I can see to solve this.
> * Filter the columnTypes passed on to the Mapper in 
> JdbcExportJob.configureInputFormat
> * Pass the --columns value on to the AvroExportMapper and let it ignore 
> things that are not in there
> * Let AvroExportMapper not fail when it can't set a field.
> I might be able to provide a patch and I'd go with the simplest (the first 
> one probably) if there are no objections.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 34104: Fix for SQOOP-1369 Avro export ignores --columns option

[Jarek Cecho]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34104/#review99034
---

Ship it!


Ship It!

- Jarek Cecho


On Aug. 11, 2015, 7:36 p.m., Paul Mazak wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34104/
> ---
> 
> (Updated Aug. 11, 2015, 7:36 p.m.)
> 
> 
> Review request for Sqoop and Yibing Shi.
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> Supplied a patch that worked for us. 
> There are 2 global places to consider - parent class SqlManager and child 
> class CatalogQueryManager - when returning getColumnNames to actually rely on 
> those column names you specified with --columns option. 
> There was 1 place to consider for Avro/Parquet in 
> JdbcExportJob.configureGenericRecordExportInputFormat() to only add the 
> columnType if it's one that was specified with --columns option.
> 
> 
> Diffs
> -
> 
>   src/java/org/apache/sqoop/SqoopOptions.java ace90fd 
>   src/java/org/apache/sqoop/manager/CatalogQueryManager.java 4e063ed 
>   src/java/org/apache/sqoop/manager/OracleManager.java 69b613f 
>   src/java/org/apache/sqoop/manager/SqlManager.java ead581d 
>   src/java/org/apache/sqoop/mapreduce/JdbcExportJob.java 93d438a 
>   src/test/com/cloudera/sqoop/TestAvroExport.java 663828c 
>   src/test/org/apache/sqoop/TestSqoopOptions.java 87b340a 
>   src/test/org/apache/sqoop/manager/TestSqlManager.java PRE-CREATION 
>   src/test/org/apache/sqoop/mapreduce/TestJdbcExportJob.java PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/34104/diff/
> 
> 
> Testing
> ---
> 
> Integration testing passed exporting AVRO to MS SQL Server
> 
> 
> Thanks,
> 
> Paul Mazak
> 
>

[jira] [Created] (SQOOP-2572) MODEL_004:Usage of unsupported data type when using DateTime type

[Jarek Jarcec Cecho (JIRA)]

Jarek Jarcec Cecho created SQOOP-2572:
-

 Summary: MODEL_004:Usage of unsupported data type when using 
DateTime type
 Key: SQOOP-2572
 URL: https://issues.apache.org/jira/browse/SQOOP-2572
 Project: Sqoop
  Issue Type: Bug
Affects Versions: 1.99.7
Reporter: Jarek Jarcec Cecho
 Fix For: 1.99.7


I've tried to incorporate the newly introduced DateTime type from SQOOP-2559 in 
my patch for incremental import in HDFS connector SQOOP-2525. It seems that we 
missed one place where each type needs to be defined:

{code}
81469 [main] INFO  org.apache.sqoop.test.testcases.ConnectorTestCase  - 
Submission finished: MSubmission [jobId=1, status=FAILURE_ON_SUBMIT, 
externalId=null, progress=-1.0, counters=null, externalLink=null, 
error=JobSubmissionError [errorSummary=org.apache.sqoop.common.SqoopException: 
MODEL_004:Usage of unsupported data type - Unsupported type 
org.joda.time.DateTime for input incremental.lastImportedDate, 
errorDetails=org.apache.sqoop.common.SqoopException: MODEL_004:Usage of 
unsupported data type - Unsupported type org.joda.time.DateTime for input 
incremental.lastImportedDate at 
org.apache.sqoop.model.ConfigUtils.toJson(ConfigUtils.java:458)  at 
org.apache.sqoop.job.mr.MRConfigurationUtils.setConnectorJobConfig(MRConfigurationUtils.java:114)
at 
org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.submit(MapreduceSubmissionEngine.java:212)
   at org.apache.sqoop.driver.JobManager.start(JobManager.java:295)at 
org.apache.sqoop.handler.JobRequestHandler.startJob(JobRequestHandler.java:388) 
 at 
org.apache.sqoop.handler.JobRequestHandler.handleEvent(JobRequestHandler.java:117)
   at 
org.apache.sqoop.server.v1.JobServlet.handlePutRequest(JobServlet.java:97)   at 
org.apache.sqoop.server.SqoopProtocolServlet.doPut(SqoopProtocolServlet.java:81)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:646) at 
javax.servlet.http.HttpServlet.service(HttpServlet.java:723) at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at 
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:572)
at 
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.doFilter(DelegationTokenAuthenticationFilter.java:269)
   at 
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:542)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at 
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
  at 
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
  at 
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)   
 at 
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)   
 at 
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:615)at 
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at 
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)  at 
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861)   at 
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:606)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) 
 at java.lang.Thread.run(Thread.java:745)], 
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (SQOOP-2332) Dynamic Partition in Sqoop HCatalog- if Hive table does not exists & add support for Partition Date Format

[Rakesh Sharma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-2332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rakesh Sharma reassigned SQOOP-2332:


Assignee: Rakesh Sharma  (was: Shashank)

> Dynamic Partition in Sqoop HCatalog- if Hive table does not exists & add 
> support for Partition Date Format
> --
>
> Key: SQOOP-2332
> URL: https://issues.apache.org/jira/browse/SQOOP-2332
> Project: Sqoop
>  Issue Type: New Feature
>Affects Versions: 1.4.5
>Reporter: Atul Gupta
>Assignee: Rakesh Sharma
> Fix For: 1.4.7
>
>
> Apache Sqoop 1.4.5 with Hcatalog supports Dynamic partitions when hive hive 
> partition table already exists. it would be nice if we have Dynamic Partition 
> support in Apache Sqoop Hcatalog even if Hive table does not exist during 
> data import. as a suggestion could add option like --hive-dynamic-partition 
> Also there is no support for date formats on partition column having type 
> like  date/time/timestamp. As a suggestion it would be great if we add Option 
> like  --hive-partition-key-format to support the partition date formats   
> based on java simple available date formats like -MM-dd, -MM,  
> etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (SQOOP-2332) Dynamic Partition in Sqoop HCatalog- if Hive table does not exists & add support for Partition Date Format

[Rakesh Sharma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-2332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rakesh Sharma updated SQOOP-2332:
-
Attachment: SQOOP-2332_0.patch

> Dynamic Partition in Sqoop HCatalog- if Hive table does not exists & add 
> support for Partition Date Format
> --
>
> Key: SQOOP-2332
> URL: https://issues.apache.org/jira/browse/SQOOP-2332
> Project: Sqoop
>  Issue Type: New Feature
>Affects Versions: 1.4.5
>Reporter: Atul Gupta
>Assignee: Rakesh Sharma
> Fix For: 1.4.7
>
> Attachments: SQOOP-2332_0.patch
>
>
> Apache Sqoop 1.4.5 with Hcatalog supports Dynamic partitions when hive hive 
> partition table already exists. it would be nice if we have Dynamic Partition 
> support in Apache Sqoop Hcatalog even if Hive table does not exist during 
> data import. as a suggestion could add option like --hive-dynamic-partition 
> Also there is no support for date formats on partition column having type 
> like  date/time/timestamp. As a suggestion it would be great if we add Option 
> like  --hive-partition-key-format to support the partition date formats   
> based on java simple available date formats like -MM-dd, -MM,  
> etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (SQOOP-1369) Avro export ignores --columns option

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-1369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14745757#comment-14745757
 ] 

Hudson commented on SQOOP-1369:
---

SUCCESS: Integrated in Sqoop-hadoop23 #1217 (See 
[https://builds.apache.org/job/Sqoop-hadoop23/1217/])
SQOOP-1369: Avro export ignores --columns option (jarcec: 
https://git-wip-us.apache.org/repos/asf?p=sqoop.git=commit=5fc7a680f854c4a03e9a581079172d453d1a059c)
* src/java/org/apache/sqoop/mapreduce/JdbcExportJob.java
* src/test/org/apache/sqoop/mapreduce/TestJdbcExportJob.java
* src/test/com/cloudera/sqoop/TestAvroExport.java
* src/test/org/apache/sqoop/TestSqoopOptions.java
* src/test/org/apache/sqoop/manager/TestSqlManager.java
* src/java/org/apache/sqoop/manager/CatalogQueryManager.java
* src/java/org/apache/sqoop/manager/SqlManager.java
* src/java/org/apache/sqoop/SqoopOptions.java
* src/java/org/apache/sqoop/manager/OracleManager.java


> Avro export ignores --columns option
> 
>
> Key: SQOOP-1369
> URL: https://issues.apache.org/jira/browse/SQOOP-1369
> Project: Sqoop
>  Issue Type: Bug
>Reporter: Lars Francke
>Assignee: Paul Mazak
> Fix For: 1.4.7
>
> Attachments: SQOOP-1369.patch, SQOOP-1369_2.patch, 
> SQOOP-1369_3.patch, SQOOP-1369_4.patch, SQOOP-1369_5.patch
>
>
> In JdbcExportJob AVRO_COLUMN_TYPES_MAP is being set with the full schema of 
> the output table. This causes the AvroExportMapper to fail with unknown 
> fields if --columns was used to restrict the columns to export (it then tries 
> to set a value on the generated class which doesn't exist).
> There are multiple ways I can see to solve this.
> * Filter the columnTypes passed on to the Mapper in 
> JdbcExportJob.configureInputFormat
> * Pass the --columns value on to the AvroExportMapper and let it ignore 
> things that are not in there
> * Let AvroExportMapper not fail when it can't set a field.
> I might be able to provide a patch and I'd go with the simplest (the first 
> one probably) if there are no objections.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (SQOOP-1369) Avro export ignores --columns option

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-1369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14745800#comment-14745800
 ] 

Hudson commented on SQOOP-1369:
---

FAILURE: Integrated in Sqoop-hadoop100 #981 (See 
[https://builds.apache.org/job/Sqoop-hadoop100/981/])
SQOOP-1369: Avro export ignores --columns option (jarcec: 
https://git-wip-us.apache.org/repos/asf?p=sqoop.git=commit=5fc7a680f854c4a03e9a581079172d453d1a059c)
* src/java/org/apache/sqoop/manager/CatalogQueryManager.java
* src/java/org/apache/sqoop/manager/SqlManager.java
* src/test/org/apache/sqoop/TestSqoopOptions.java
* src/java/org/apache/sqoop/manager/OracleManager.java
* src/test/org/apache/sqoop/mapreduce/TestJdbcExportJob.java
* src/test/org/apache/sqoop/manager/TestSqlManager.java
* src/java/org/apache/sqoop/mapreduce/JdbcExportJob.java
* src/test/com/cloudera/sqoop/TestAvroExport.java
* src/java/org/apache/sqoop/SqoopOptions.java


> Avro export ignores --columns option
> 
>
> Key: SQOOP-1369
> URL: https://issues.apache.org/jira/browse/SQOOP-1369
> Project: Sqoop
>  Issue Type: Bug
>Reporter: Lars Francke
>Assignee: Paul Mazak
> Fix For: 1.4.7
>
> Attachments: SQOOP-1369.patch, SQOOP-1369_2.patch, 
> SQOOP-1369_3.patch, SQOOP-1369_4.patch, SQOOP-1369_5.patch
>
>
> In JdbcExportJob AVRO_COLUMN_TYPES_MAP is being set with the full schema of 
> the output table. This causes the AvroExportMapper to fail with unknown 
> fields if --columns was used to restrict the columns to export (it then tries 
> to set a value on the generated class which doesn't exist).
> There are multiple ways I can see to solve this.
> * Filter the columnTypes passed on to the Mapper in 
> JdbcExportJob.configureInputFormat
> * Pass the --columns value on to the AvroExportMapper and let it ignore 
> things that are not in there
> * Let AvroExportMapper not fail when it can't set a field.
> I might be able to provide a patch and I'd go with the simplest (the first 
> one probably) if there are no objections.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (SQOOP-1369) Avro export ignores --columns option

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-1369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14745822#comment-14745822
 ] 

Hudson commented on SQOOP-1369:
---

FAILURE: Integrated in Sqoop-hadoop200 #1021 (See 
[https://builds.apache.org/job/Sqoop-hadoop200/1021/])
SQOOP-1369: Avro export ignores --columns option (jarcec: 
https://git-wip-us.apache.org/repos/asf?p=sqoop.git=commit=5fc7a680f854c4a03e9a581079172d453d1a059c)
* src/java/org/apache/sqoop/manager/SqlManager.java
* src/java/org/apache/sqoop/manager/CatalogQueryManager.java
* src/java/org/apache/sqoop/mapreduce/JdbcExportJob.java
* src/test/org/apache/sqoop/mapreduce/TestJdbcExportJob.java
* src/java/org/apache/sqoop/SqoopOptions.java
* src/test/org/apache/sqoop/TestSqoopOptions.java
* src/java/org/apache/sqoop/manager/OracleManager.java
* src/test/org/apache/sqoop/manager/TestSqlManager.java
* src/test/com/cloudera/sqoop/TestAvroExport.java


> Avro export ignores --columns option
> 
>
> Key: SQOOP-1369
> URL: https://issues.apache.org/jira/browse/SQOOP-1369
> Project: Sqoop
>  Issue Type: Bug
>Reporter: Lars Francke
>Assignee: Paul Mazak
> Fix For: 1.4.7
>
> Attachments: SQOOP-1369.patch, SQOOP-1369_2.patch, 
> SQOOP-1369_3.patch, SQOOP-1369_4.patch, SQOOP-1369_5.patch
>
>
> In JdbcExportJob AVRO_COLUMN_TYPES_MAP is being set with the full schema of 
> the output table. This causes the AvroExportMapper to fail with unknown 
> fields if --columns was used to restrict the columns to export (it then tries 
> to set a value on the generated class which doesn't exist).
> There are multiple ways I can see to solve this.
> * Filter the columnTypes passed on to the Mapper in 
> JdbcExportJob.configureInputFormat
> * Pass the --columns value on to the AvroExportMapper and let it ignore 
> things that are not in there
> * Let AvroExportMapper not fail when it can't set a field.
> I might be able to provide a patch and I'd go with the simplest (the first 
> one probably) if there are no objections.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 38400: Review for issue SQOOP-2332: Support dynamic partitions while sqoop import. Also support partition key formats for date partitions

[Rakesh Sharma]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38400/
---

(Updated Sept. 15, 2015, 6:02 p.m.)


Review request for Sqoop and Venkat Ranganathan.


Changes
---

Removing extra whitespaces


Repository: sqoop-trunk


Description
---

Review request for support of Dynamic partitons while importing data. Users can 
just tell the list of columns based on which partitioning should happen. Also 
support for date formats is added if user wants to do partitioning based on a 
date column.


Diffs (updated)
-

  src/docs/user/hcatalog.txt 99ae4f5 
  src/java/org/apache/sqoop/SqoopOptions.java ace90fd 
  src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatImportHelper.java 878f765 
  src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java 4070c24 
  src/java/org/apache/sqoop/metastore/TableMetaData.java PRE-CREATION 
  src/java/org/apache/sqoop/tool/BaseSqoopTool.java a7f6aec 
  src/java/org/apache/sqoop/tool/ImportTool.java 39af42c 
  src/java/org/apache/sqoop/util/DbUtil.java PRE-CREATION 
  src/test/org/apache/sqoop/hcat/HCatalogImportTest.java d97b870 

Diff: https://reviews.apache.org/r/38400/diff/


Testing
---

Yes


Thanks,

Rakesh Sharma

[jira] [Commented] (SQOOP-1369) Avro export ignores --columns option

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-1369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14745727#comment-14745727
 ] 

Hudson commented on SQOOP-1369:
---

FAILURE: Integrated in Sqoop-hadoop20 #1014 (See 
[https://builds.apache.org/job/Sqoop-hadoop20/1014/])
SQOOP-1369: Avro export ignores --columns option (jarcec: 
https://git-wip-us.apache.org/repos/asf?p=sqoop.git=commit=5fc7a680f854c4a03e9a581079172d453d1a059c)
* src/test/com/cloudera/sqoop/TestAvroExport.java
* src/java/org/apache/sqoop/mapreduce/JdbcExportJob.java
* src/java/org/apache/sqoop/manager/CatalogQueryManager.java
* src/test/org/apache/sqoop/TestSqoopOptions.java
* src/java/org/apache/sqoop/manager/SqlManager.java
* src/java/org/apache/sqoop/SqoopOptions.java
* src/test/org/apache/sqoop/mapreduce/TestJdbcExportJob.java
* src/java/org/apache/sqoop/manager/OracleManager.java
* src/test/org/apache/sqoop/manager/TestSqlManager.java


> Avro export ignores --columns option
> 
>
> Key: SQOOP-1369
> URL: https://issues.apache.org/jira/browse/SQOOP-1369
> Project: Sqoop
>  Issue Type: Bug
>Reporter: Lars Francke
>Assignee: Paul Mazak
> Fix For: 1.4.7
>
> Attachments: SQOOP-1369.patch, SQOOP-1369_2.patch, 
> SQOOP-1369_3.patch, SQOOP-1369_4.patch, SQOOP-1369_5.patch
>
>
> In JdbcExportJob AVRO_COLUMN_TYPES_MAP is being set with the full schema of 
> the output table. This causes the AvroExportMapper to fail with unknown 
> fields if --columns was used to restrict the columns to export (it then tries 
> to set a value on the generated class which doesn't exist).
> There are multiple ways I can see to solve this.
> * Filter the columnTypes passed on to the Mapper in 
> JdbcExportJob.configureInputFormat
> * Pass the --columns value on to the AvroExportMapper and let it ignore 
> things that are not in there
> * Let AvroExportMapper not fail when it can't set a field.
> I might be able to provide a patch and I'd go with the simplest (the first 
> one probably) if there are no objections.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (SQOOP-2572) MODEL_004:Usage of unsupported data type when using DateTime type

[Jarek Jarcec Cecho (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-2572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14745622#comment-14745622
 ] 

Jarek Jarcec Cecho commented on SQOOP-2572:
---

Would you be open to take a look [~dian.fu]?

> MODEL_004:Usage of unsupported data type when using DateTime type
> -
>
> Key: SQOOP-2572
> URL: https://issues.apache.org/jira/browse/SQOOP-2572
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.99.7
>Reporter: Jarek Jarcec Cecho
> Fix For: 1.99.7
>
>
> I've tried to incorporate the newly introduced DateTime type from SQOOP-2559 
> in my patch for incremental import in HDFS connector SQOOP-2525. It seems 
> that we missed one place where each type needs to be defined:
> {code}
> 81469 [main] INFO  org.apache.sqoop.test.testcases.ConnectorTestCase  - 
> Submission finished: MSubmission [jobId=1, status=FAILURE_ON_SUBMIT, 
> externalId=null, progress=-1.0, counters=null, externalLink=null, 
> error=JobSubmissionError 
> [errorSummary=org.apache.sqoop.common.SqoopException: MODEL_004:Usage of 
> unsupported data type - Unsupported type org.joda.time.DateTime for input 
> incremental.lastImportedDate, 
> errorDetails=org.apache.sqoop.common.SqoopException: MODEL_004:Usage of 
> unsupported data type - Unsupported type org.joda.time.DateTime for input 
> incremental.lastImportedDate   at 
> org.apache.sqoop.model.ConfigUtils.toJson(ConfigUtils.java:458)  at 
> org.apache.sqoop.job.mr.MRConfigurationUtils.setConnectorJobConfig(MRConfigurationUtils.java:114)
> at 
> org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.submit(MapreduceSubmissionEngine.java:212)
>at org.apache.sqoop.driver.JobManager.start(JobManager.java:295)at 
> org.apache.sqoop.handler.JobRequestHandler.startJob(JobRequestHandler.java:388)
>   at 
> org.apache.sqoop.handler.JobRequestHandler.handleEvent(JobRequestHandler.java:117)
>at 
> org.apache.sqoop.server.v1.JobServlet.handlePutRequest(JobServlet.java:97)   
> at 
> org.apache.sqoop.server.SqoopProtocolServlet.doPut(SqoopProtocolServlet.java:81)
>  at javax.servlet.http.HttpServlet.service(HttpServlet.java:646) at 
> javax.servlet.http.HttpServlet.service(HttpServlet.java:723) at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at 
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:572)
> at 
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.doFilter(DelegationTokenAuthenticationFilter.java:269)
>at 
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:542)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at 
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
>   at 
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
>   at 
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) 
>at 
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) 
>at 
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:615)
> at 
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
> at 
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)  
> at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861) 
>   at 
> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:606)
>  at 
> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)  
> at java.lang.Thread.run(Thread.java:745)], 
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 38400: Review for issue SQOOP-2332: Support dynamic partitions while sqoop import. Also support partition key formats for date partitions

[Rakesh Sharma]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38400/
---

Review request for Sqoop and Venkat Ranganathan.


Repository: sqoop-trunk


Description
---

Review request for support of Dynamic partitons while importing data. Users can 
just tell the list of columns based on which partitioning should happen. Also 
support for date formats is added if user wants to do partitioning based on a 
date column.


Diffs
-

  src/docs/user/hcatalog.txt 99ae4f5 
  src/java/org/apache/sqoop/SqoopOptions.java ace90fd 
  src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatImportHelper.java 878f765 
  src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java 4070c24 
  src/java/org/apache/sqoop/metastore/TableMetaData.java PRE-CREATION 
  src/java/org/apache/sqoop/tool/BaseSqoopTool.java a7f6aec 
  src/java/org/apache/sqoop/tool/ImportTool.java 39af42c 
  src/java/org/apache/sqoop/util/DbUtil.java PRE-CREATION 
  src/test/org/apache/sqoop/hcat/HCatalogImportTest.java d97b870 

Diff: https://reviews.apache.org/r/38400/diff/


Testing
---

Yes


Thanks,

Rakesh Sharma