[jira] [Updated] (SQOOP-3487) Remove unintentionally added rowkeys (via PUT in ToStringPutTransformer.java) during HBase (incremental) import

[Attila Szabo (Jira)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo updated SQOOP-3487:

Affects Version/s: (was: no-release)
   1.4.7

> Remove unintentionally added rowkeys (via PUT in ToStringPutTransformer.java) 
> during HBase (incremental) import
> ---
>
> Key: SQOOP-3487
> URL: https://issues.apache.org/jira/browse/SQOOP-3487
> Project: Sqoop
>  Issue Type: Improvement
>  Components: hbase-integration
>Affects Versions: 1.4.7
> Environment: sqoop-version:1.4.7-cdh6.3.2
>Reporter: hong
>Priority: Major
>  Labels: hbase
> Fix For: no-release
>
> Attachments: 
> 0001-SQOOP-3487-Add-PUT-repeatedly-when-importing-to-HBas.patch
>
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> Repeated add of PUT cause the import into HBase to slow down



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (SQOOP-3487) Remove unintentionally added rowkeys (via PUT in ToStringPutTransformer.java) during HBase (incremental) import

[Attila Szabo (Jira)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo updated SQOOP-3487:

Fix Version/s: (was: no-release)
   1.5.0

> Remove unintentionally added rowkeys (via PUT in ToStringPutTransformer.java) 
> during HBase (incremental) import
> ---
>
> Key: SQOOP-3487
> URL: https://issues.apache.org/jira/browse/SQOOP-3487
> Project: Sqoop
>  Issue Type: Improvement
>  Components: hbase-integration
>Affects Versions: 1.4.7
> Environment: sqoop-version:1.4.7-cdh6.3.2
>Reporter: hong
>Priority: Major
>  Labels: hbase
> Fix For: 1.5.0
>
> Attachments: 
> 0001-SQOOP-3487-Add-PUT-repeatedly-when-importing-to-HBas.patch
>
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> Repeated add of PUT cause the import into HBase to slow down



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (SQOOP-3487) Remove unintentionally added rowkeys (via PUT in ToStringPutTransformer.java) during HBase (incremental) import

[Attila Szabo (Jira)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302554#comment-17302554
 ] 

Attila Szabo commented on SQOOP-3487:
-

Hi [~zhou0145] ,

Thank you for your contribution!
Regards,
[~maugli]

> Remove unintentionally added rowkeys (via PUT in ToStringPutTransformer.java) 
> during HBase (incremental) import
> ---
>
> Key: SQOOP-3487
> URL: https://issues.apache.org/jira/browse/SQOOP-3487
> Project: Sqoop
>  Issue Type: Improvement
>  Components: hbase-integration
>Affects Versions: no-release
> Environment: sqoop-version:1.4.7-cdh6.3.2
>Reporter: hong
>Priority: Major
>  Labels: hbase
> Fix For: no-release
>
> Attachments: 
> 0001-SQOOP-3487-Add-PUT-repeatedly-when-importing-to-HBas.patch
>
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> Repeated add of PUT cause the import into HBase to slow down



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (SQOOP-3487) Remove unintentionally added rowkeys (via PUT in ToStringPutTransformer.java) during HBase (incremental) import

[Attila Szabo (Jira)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo updated SQOOP-3487:

Summary: Remove unintentionally added rowkeys (via PUT in 
ToStringPutTransformer.java) during HBase (incremental) import  (was: Add PUT 
repeatedly when importing to HBase)

> Remove unintentionally added rowkeys (via PUT in ToStringPutTransformer.java) 
> during HBase (incremental) import
> ---
>
> Key: SQOOP-3487
> URL: https://issues.apache.org/jira/browse/SQOOP-3487
> Project: Sqoop
>  Issue Type: Improvement
>  Components: hbase-integration
>Affects Versions: no-release
> Environment: sqoop-version:1.4.7-cdh6.3.2
>Reporter: hong
>Priority: Major
>  Labels: hbase
> Fix For: no-release
>
> Attachments: 
> 0001-SQOOP-3487-Add-PUT-repeatedly-when-importing-to-HBas.patch
>
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> Repeated add of PUT cause the import into HBase to slow down



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (SQOOP-3487) Add PUT repeatedly when importing to HBase

[Attila Szabo (Jira)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302253#comment-17302253
 ] 

Attila Szabo commented on SQOOP-3487:
-

Hi [~zhou0145] ,

I've reviewed your changes (both Github and issues.apache.org), but TBH in the 
current state I'm concerned both about the intention of the change, and the 
correctness as well.

First of all:
Could you please provide a bit more detail around what performance gain do you 
expect from this change and how did you measure it? Could you please provide 
also some automated testcase which would show the effect of this gain, and 
would ensure we don't loose it in the future?

On the front of correctness:
SQOOP-3149 introduced the line you'd like to remove, and if I do remember 
correctly absolutely intentionally. Because of this reason:
Could you please provide automated test cases which ensures that SQOOP-3149 
changes won't be undone by your change (so we keep the current correctness 
around NULL column updates)?

Many thanks in advance,
[~maugli]

> Add PUT repeatedly when importing to HBase
> --
>
> Key: SQOOP-3487
> URL: https://issues.apache.org/jira/browse/SQOOP-3487
> Project: Sqoop
>  Issue Type: Improvement
>  Components: hbase-integration
>Affects Versions: no-release
> Environment: sqoop-version:1.4.7-cdh6.3.2
>Reporter: hong
>Priority: Major
>  Labels: hbase
> Fix For: no-release
>
> Attachments: 
> 0001-SQOOP-3487-Add-PUT-repeatedly-when-importing-to-HBas.patch
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Repeated add of PUT cause the import into HBase to slow down



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (SQOOP-3415) Fix gradle test+build when clean applied as the first command + warning issue fixes

[Attila Szabo (JIRA)]

Attila Szabo created SQOOP-3415:
---

 Summary: Fix gradle test+build when clean applied as the first 
command + warning issue fixes
 Key: SQOOP-3415
 URL: https://issues.apache.org/jira/browse/SQOOP-3415
 Project: Sqoop
  Issue Type: Bug
Affects Versions: 1.5.0
Reporter: Attila Szabo
Assignee: Attila Szabo
 Fix For: 1.5.0


If the user wants to build like the following command:

gradlew clean unittest

the gradle process ends up in an exception and the whole process left there 
hanging forever. The root cause of this is the following:

tasks.withType runs in the configuration part of the build, where we ensure the 
neccessary directories exist.

after that clean is executed and all of the dirs got deleted.

Proposed fix:
Apply directory creation as the first step of test tasks.

on the top:
there are some missing options b/c of Junit annotation processors, and also 
Xlint information are swallowed currently. We aim to fix these things as well



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SQOOP-3289) Add .travis.yml

[Attila Szabo (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16377212#comment-16377212
 ] 

Attila Szabo commented on SQOOP-3289:
-

Hi Daniel, 

Though in general the ASF community is great, IMHO  it does not make too much 
sense to compare one project to the other. Especially a very busy active like 
Spark with something quite mature as Sqoop. 

Though I think my message was clear enough, it's up to you if you willing to 
wait until the current builds.apache.org build pipeline will do the real job. 
Just one comment on my side :

If we won't be able to make the life of the contributors ( and the committers) 
easier, any efforts on this front could be in vain. 

My 2 cents, 
Attila 

> Add .travis.yml
> ---
>
> Key: SQOOP-3289
> URL: https://issues.apache.org/jira/browse/SQOOP-3289
> Project: Sqoop
>  Issue Type: Task
>  Components: build
>Affects Versions: 1.4.7
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Minor
> Fix For: 1.5.0
>
>
> Adding a .travis.yml would enable running builds/tests on travis-ci.org. 
> Currently if you wish to use Travis for testing your changes, you have to 
> manually add a .travis.yml to your branch. Having it committed to trunk would 
> save us this extra step.
> I currently have an example 
> [{{.travis.yml}}|https://github.com/dvoros/sqoop/blob/93a4c06c1a3da1fd5305c99e379484507797b3eb/.travis.yml]
>  on my travis branch running unit tests for every commit and every pull 
> request: https://travis-ci.org/dvoros/sqoop/builds
> Later we could add the build status to the project readme as well, see: 
> https://github.com/dvoros/sqoop/tree/travis
> Also, an example of a pull request: https://github.com/dvoros/sqoop/pull/1



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SQOOP-3289) Add .travis.yml

[Attila Szabo (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16375050#comment-16375050
 ] 

Attila Szabo commented on SQOOP-3289:
-

Hey [~dvoros],

I would love the idea to have to finally have a working and  (apache toolchain) 
independent CI/CD system.

Though if don't mind I'd like to aim for something more ambitious.

What do you think::

Would it be possible to setup travis in a way to execute automatically unit and 
integration and 3rd test for trunk after every commit?

The reason why I'm asking is the following:

I do understand that for developers who prefer the Github fork way it would be 
much easier to execute the unit tests with Travis compared to local one, 
although this solution would barely differ from the currently existing 
[https://builds.apache.org/job/Sqoop-hadoop200/] solution, and IMHO we 
definitely NEED a CI system where ALL of the tests are evaluated after every 
commit. I've been trying to get in contact with the responsible ppl to make 
changes in [https://builds.apache.org/job/Sqoop-hadoop200/] though I did not 
make too big achievements sadly.

 

Thats why I'm asking:

What about to aim for running every tests in travis?

 

Thanks,

Attila

> Add .travis.yml
> ---
>
> Key: SQOOP-3289
> URL: https://issues.apache.org/jira/browse/SQOOP-3289
> Project: Sqoop
>  Issue Type: Task
>  Components: build
>Affects Versions: 1.4.7
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Minor
> Fix For: 1.5.0
>
>
> Adding a .travis.yml would enable running builds/tests on travis-ci.org. 
> Currently if you wish to use Travis for testing your changes, you have to 
> manually add a .travis.yml to your branch. Having it committed to trunk would 
> save us this extra step.
> I currently have an example 
> [{{.travis.yml}}|https://github.com/dvoros/sqoop/blob/93a4c06c1a3da1fd5305c99e379484507797b3eb/.travis.yml]
>  on my travis branch running unit tests for every commit and every pull 
> request: https://travis-ci.org/dvoros/sqoop/builds
> Later we could add the build status to the project readme as well, see: 
> https://github.com/dvoros/sqoop/tree/travis
> Also, an example of a pull request: https://github.com/dvoros/sqoop/pull/1



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SQOOP-3288) Incremental import's upper bound ignores session time zone in Oracle

[Attila Szabo (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16371609#comment-16371609
 ] 

Attila Szabo commented on SQOOP-3288:
-

[~dvoros],

Ohh yeah! You're right! And applying the proper timezone was fixed by me in 
-SQOOP-3071-... My old memory...

Approved. Let me sign it on the review ticket as well!

Cheers,

[~maugli]

ps:
Thanks for the b part of the explanation, but that was clear already. But 
thanks!

> Incremental import's upper bound ignores session time zone in Oracle
> 
>
> Key: SQOOP-3288
> URL: https://issues.apache.org/jira/browse/SQOOP-3288
> Project: Sqoop
>  Issue Type: Bug
>  Components: connectors/oracle
>Affects Versions: 1.4.7
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Major
> Fix For: 1.5.0
>
> Attachments: SQOOP-3288.1.patch
>
>
> At the moment we're using [{{SELECT SYSDATE FROM 
> dual}}|https://github.com/apache/sqoop/blob/3153c3610da7e5db388bfb14f3681d308e9e89c6/src/java/org/apache/sqoop/manager/OracleManager.java#L652]
>  when getting current time from Oracle.
> SYSDATE returns the underlying operating system's current time, while 
> CURRENT_TIMESTAMP uses the session time zone. This could lead to problems 
> during incremental imports *when Oracle's time zone is different from the OS*.
> Consider the following scenario when Oracle is configured to {{+0:00}}, while 
> the OS is {{+5:00}}:
> ||Oracle time||OS time||Event||
> |2:00|7:00|{{sqoop import --last-value 1:00 ...}} => imports {{[1:00, 7:00)}}|
> |2:30|7:30|{{update ... set last_updated = current_timestamp ...}} => set to 
> {{2:30}} *Won't be imported!*|
> |3:00|8:00|{{sqoop import --last-value 7:00 ...}} => imports {{[7:00, 8:00)}}|
> This way records updated within 5 hours after the last sqoop import won't get 
> imported.
> Please note, that the example above assumes, that the user/administrator 
> who's updating the Oracle table will use the current session time of Oracle 
> when setting the "last updated" column of the table.
> I think the solution is to use CURRENT_TIMESTAMP instead of SYSDATE. Other 
> connection managers, like MySQL or PostgreSQL are using that as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SQOOP-3288) Incremental import's upper bound ignores session time zone in Oracle

[Attila Szabo (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16371561#comment-16371561
 ] 

Attila Szabo commented on SQOOP-3288:
-

[~dvoros],

You're arguments are absolutely valid, maybe I've not phrased my concerns 
correctly:

Both of CURRENT_TIMESTAMP and LOCALTIMESTAMP fits us, I'm glad we agree in 
this, and I'm also pretty much accepting that you've chosen the first one (BTW: 
fair enough arguments!).

The problem is the following:

Before your patch the behavior is the following:

Any timezone, and time is set in the Oracle's OS, we're using that. So 
regardless which node of the cluster we're executing the import job, the result 
will be the same.

After your patch it's great we would not depend on the ORacle OS time anymore, 
though the result would differ from machine to machine if the cluster's 
timezones are mixed (if I'm not mistaken!). So if somehow we're not enforcing 
setting the session time, or converting and comparing the times in a timezone 
aware way, the new solution could even result in "strange" results.

Do you see any way to ensure that e.g. everything (both last and current times) 
is compared in a timezone aware way? Or are my concerns are "too paranoid"?

Thanks,
[~maugli]

> Incremental import's upper bound ignores session time zone in Oracle
> 
>
> Key: SQOOP-3288
> URL: https://issues.apache.org/jira/browse/SQOOP-3288
> Project: Sqoop
>  Issue Type: Bug
>  Components: connectors/oracle
>Affects Versions: 1.4.7
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Major
> Fix For: 1.5.0
>
> Attachments: SQOOP-3288.1.patch
>
>
> At the moment we're using [{{SELECT SYSDATE FROM 
> dual}}|https://github.com/apache/sqoop/blob/3153c3610da7e5db388bfb14f3681d308e9e89c6/src/java/org/apache/sqoop/manager/OracleManager.java#L652]
>  when getting current time from Oracle.
> SYSDATE returns the underlying operating system's current time, while 
> CURRENT_TIMESTAMP uses the session time zone. This could lead to problems 
> during incremental imports *when Oracle's time zone is different from the OS*.
> Consider the following scenario when Oracle is configured to {{+0:00}}, while 
> the OS is {{+5:00}}:
> ||Oracle time||OS time||Event||
> |2:00|7:00|{{sqoop import --last-value 1:00 ...}} => imports {{[1:00, 7:00)}}|
> |2:30|7:30|{{update ... set last_updated = current_timestamp ...}} => set to 
> {{2:30}} *Won't be imported!*|
> |3:00|8:00|{{sqoop import --last-value 7:00 ...}} => imports {{[7:00, 8:00)}}|
> This way records updated within 5 hours after the last sqoop import won't get 
> imported.
> Please note, that the example above assumes, that the user/administrator 
> who's updating the Oracle table will use the current session time of Oracle 
> when setting the "last updated" column of the table.
> I think the solution is to use CURRENT_TIMESTAMP instead of SYSDATE. Other 
> connection managers, like MySQL or PostgreSQL are using that as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SQOOP-3288) Incremental import's upper bound ignores session time zone in Oracle

[Attila Szabo (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16371491#comment-16371491
 ] 

Attila Szabo commented on SQOOP-3288:
-

Hey [~dvoros],

The reason why I'm a bit concerned:

I've faced already clusters where a specific node was in a very different 
timezone, than the rest of the cluster.

Executing job from that node would end up in different result, than from the 
others.

Do we expect this behaviour? Do we really want to depend on the OS time?

 

Thanks,

Attila

> Incremental import's upper bound ignores session time zone in Oracle
> 
>
> Key: SQOOP-3288
> URL: https://issues.apache.org/jira/browse/SQOOP-3288
> Project: Sqoop
>  Issue Type: Bug
>  Components: connectors/oracle
>Affects Versions: 1.4.7
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Major
> Fix For: 1.5.0
>
> Attachments: SQOOP-3288.1.patch
>
>
> At the moment we're using [{{SELECT SYSDATE FROM 
> dual}}|https://github.com/apache/sqoop/blob/3153c3610da7e5db388bfb14f3681d308e9e89c6/src/java/org/apache/sqoop/manager/OracleManager.java#L652]
>  when getting current time from Oracle.
> SYSDATE returns the underlying operating system's current time, while 
> CURRENT_TIMESTAMP uses the session time zone. This could lead to problems 
> during incremental imports *when Oracle's time zone is different from the OS*.
> Consider the following scenario when Oracle is configured to {{+0:00}}, while 
> the OS is {{+5:00}}:
> ||Oracle time||OS time||Event||
> |2:00|7:00|{{sqoop import --last-value 1:00 ...}} => imports {{[1:00, 7:00)}}|
> |2:30|7:30|{{update ... set last_updated = current_timestamp ...}} => set to 
> {{2:30}} *Won't be imported!*|
> |3:00|8:00|{{sqoop import --last-value 7:00 ...}} => imports {{[7:00, 8:00)}}|
> This way records updated within 5 hours after the last sqoop import won't get 
> imported.
> Please note, that the example above assumes, that the user/administrator 
> who's updating the Oracle table will use the current session time of Oracle 
> when setting the "last updated" column of the table.
> I think the solution is to use CURRENT_TIMESTAMP instead of SYSDATE. Other 
> connection managers, like MySQL or PostgreSQL are using that as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SQOOP-3288) Incremental import's upper bound ignores session time zone in Oracle

[Attila Szabo (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16370164#comment-16370164
 ] 

Attila Szabo commented on SQOOP-3288:
-

[~dvoros],

Please link the review board as well!

Next time please do not forgot to fill out the fix version (it's incredible 
useful when someone grooming these things for a release !!! )

Have you considered the usage of SYSTIMESTAMP? AFAIK it's a timezone aware 
timestamp.

Cheers,
[~maugli]

> Incremental import's upper bound ignores session time zone in Oracle
> 
>
> Key: SQOOP-3288
> URL: https://issues.apache.org/jira/browse/SQOOP-3288
> Project: Sqoop
>  Issue Type: Bug
>  Components: connectors/oracle
>Affects Versions: 1.4.7
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Major
> Fix For: 1.5.0
>
> Attachments: SQOOP-3288.1.patch
>
>
> At the moment we're using [{{SELECT SYSDATE FROM 
> dual}}|https://github.com/apache/sqoop/blob/3153c3610da7e5db388bfb14f3681d308e9e89c6/src/java/org/apache/sqoop/manager/OracleManager.java#L652]
>  when getting current time from Oracle.
> SYSDATE returns the underlying operating system's current time, while 
> CURRENT_TIMESTAMP uses the session time zone. This could lead to problems 
> during incremental imports *when Oracle's time zone is different from the OS*.
> Consider the following scenario when Oracle is configured to {{+0:00}}, while 
> the OS is {{+5:00}}:
> ||Oracle time||OS time||Event||
> |2:00|7:00|{{sqoop import --last-value 1:00 ...}} => imports {{[1:00, 7:00)}}|
> |2:30|7:30|{{update ... set last_updated = current_timestamp ...}} => set to 
> {{2:30}} *Won't be imported!*|
> |3:00|8:00|{{sqoop import --last-value 7:00 ...}} => imports {{[7:00, 8:00)}}|
> This way records updated within 5 hours after the last sqoop import won't get 
> imported.
> Please note, that the example above assumes, that the user/administrator 
> who's updating the Oracle table will use the current session time of Oracle 
> when setting the "last updated" column of the table.
> I think the solution is to use CURRENT_TIMESTAMP instead of SYSDATE. Other 
> connection managers, like MySQL or PostgreSQL are using that as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (SQOOP-3288) Incremental import's upper bound ignores session time zone in Oracle

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo updated SQOOP-3288:

Fix Version/s: 1.5.0

> Incremental import's upper bound ignores session time zone in Oracle
> 
>
> Key: SQOOP-3288
> URL: https://issues.apache.org/jira/browse/SQOOP-3288
> Project: Sqoop
>  Issue Type: Bug
>  Components: connectors/oracle
>Affects Versions: 1.4.7
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Major
> Fix For: 1.5.0
>
> Attachments: SQOOP-3288.1.patch
>
>
> At the moment we're using [{{SELECT SYSDATE FROM 
> dual}}|https://github.com/apache/sqoop/blob/3153c3610da7e5db388bfb14f3681d308e9e89c6/src/java/org/apache/sqoop/manager/OracleManager.java#L652]
>  when getting current time from Oracle.
> SYSDATE returns the underlying operating system's current time, while 
> CURRENT_TIMESTAMP uses the session time zone. This could lead to problems 
> during incremental imports *when Oracle's time zone is different from the OS*.
> Consider the following scenario when Oracle is configured to {{+0:00}}, while 
> the OS is {{+5:00}}:
> ||Oracle time||OS time||Event||
> |2:00|7:00|{{sqoop import --last-value 1:00 ...}} => imports {{[1:00, 7:00)}}|
> |2:30|7:30|{{update ... set last_updated = current_timestamp ...}} => set to 
> {{2:30}} *Won't be imported!*|
> |3:00|8:00|{{sqoop import --last-value 7:00 ...}} => imports {{[7:00, 8:00)}}|
> This way records updated within 5 hours after the last sqoop import won't get 
> imported.
> Please note, that the example above assumes, that the user/administrator 
> who's updating the Oracle table will use the current session time of Oracle 
> when setting the "last updated" column of the table.
> I think the solution is to use CURRENT_TIMESTAMP instead of SYSDATE. Other 
> connection managers, like MySQL or PostgreSQL are using that as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SQOOP-3283) MySQL thirdparty tests hang if there's no USER environment variable

[Attila Szabo (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365629#comment-16365629
 ] 

Attila Szabo commented on SQOOP-3283:
-

Hey [~dvoros],

Let's open the new ticket, and discuss it there (as the existing functionality 
is definitely broken!

 

Cheers,

[~maugli]

 

 

> MySQL thirdparty tests hang if there's no USER environment variable
> ---
>
> Key: SQOOP-3283
> URL: https://issues.apache.org/jira/browse/SQOOP-3283
> Project: Sqoop
>  Issue Type: Bug
>  Components: connectors/mysql, test
>Affects Versions: 1.4.7
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Minor
> Fix For: 1.5.0
>
> Attachments: SQOOP-3283.1.patch, SQOOP-3283.2.patch
>
>
> {{org.apache.sqoop.manager.mysql.MySQLTestUtils#getCurrentUser()}} executes 
> {{whoami}} in a subprocess if there's no USER environment variable (happened 
> to me while running tests from Docker). However, it waits for the Process 
> variable to become null, that never happens:
> {code:java}
> // wait for whoami to exit.
> while (p != null) {
>   try {
> int ret = p.waitFor();
> if (0 != ret) {
>   LOG.error("whoami exited with error status " + ret);
>   // suppress original return value from this method.
>   return null;
> }
>   } catch (InterruptedException ie) {
> continue; // loop around.
>   }
> }
> {code}
> We could get rid of the while loop since {{Process#waitFor()}} blocks while 
> it completes.
> Note, that it's easy to workaround the issue by setting the USER environment 
> variable when running the tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (SQOOP-3283) MySQL thirdparty tests hang if there's no USER environment variable

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo updated SQOOP-3283:

Fix Version/s: 1.5.0

> MySQL thirdparty tests hang if there's no USER environment variable
> ---
>
> Key: SQOOP-3283
> URL: https://issues.apache.org/jira/browse/SQOOP-3283
> Project: Sqoop
>  Issue Type: Bug
>  Components: connectors/mysql, test
>Affects Versions: 1.4.7
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Minor
> Fix For: 1.5.0
>
> Attachments: SQOOP-3283.1.patch, SQOOP-3283.2.patch
>
>
> {{org.apache.sqoop.manager.mysql.MySQLTestUtils#getCurrentUser()}} executes 
> {{whoami}} in a subprocess if there's no USER environment variable (happened 
> to me while running tests from Docker). However, it waits for the Process 
> variable to become null, that never happens:
> {code:java}
> // wait for whoami to exit.
> while (p != null) {
>   try {
> int ret = p.waitFor();
> if (0 != ret) {
>   LOG.error("whoami exited with error status " + ret);
>   // suppress original return value from this method.
>   return null;
> }
>   } catch (InterruptedException ie) {
> continue; // loop around.
>   }
> }
> {code}
> We could get rid of the while loop since {{Process#waitFor()}} blocks while 
> it completes.
> Note, that it's easy to workaround the issue by setting the USER environment 
> variable when running the tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (SQOOP-3283) MySQL thirdparty tests hang if there's no USER environment variable

[Attila Szabo (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364532#comment-16364532
 ] 

Attila Szabo edited comment on SQOOP-3283 at 2/14/18 6:12 PM:
--

Hey [~dvoros],

Thanks for this contribution!

One additional thought:

You might also consider to open a new ticket for changing the behaviour of the 
build.xml file (line 196) b/c that might also cause unexpected behaviour (like 
MySQL complaining around "${env.USER}" username) , especially for those who 
devs/users who are not playing with ant since Java 1.2 (you could find ore 
detials here: [https://ant.apache.org/manual/Tasks/property.html#notes-env] )

 

Thanks,

[~maugli]


was (Author: maugli):
Hey [~dvoros],

Thanks for this contribution!

One additional thought:

You might be considering to open a new ticket for changing the behaviour of the 
build.xml file (line 196) b/c that might also cause unexpected behaviour (like 
MySQL complaining around "${env.USER}" username) , especially for those who 
devs/users who are not playing with ant since Java 1.2 (you could find ore 
detials here: [https://ant.apache.org/manual/Tasks/property.html#notes-env] )

 

Thanks,

[~maugli]

> MySQL thirdparty tests hang if there's no USER environment variable
> ---
>
> Key: SQOOP-3283
> URL: https://issues.apache.org/jira/browse/SQOOP-3283
> Project: Sqoop
>  Issue Type: Bug
>  Components: connectors/mysql, test
>Affects Versions: 1.4.7
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Minor
> Fix For: 1.5.0
>
> Attachments: SQOOP-3283.1.patch, SQOOP-3283.2.patch
>
>
> {{org.apache.sqoop.manager.mysql.MySQLTestUtils#getCurrentUser()}} executes 
> {{whoami}} in a subprocess if there's no USER environment variable (happened 
> to me while running tests from Docker). However, it waits for the Process 
> variable to become null, that never happens:
> {code:java}
> // wait for whoami to exit.
> while (p != null) {
>   try {
> int ret = p.waitFor();
> if (0 != ret) {
>   LOG.error("whoami exited with error status " + ret);
>   // suppress original return value from this method.
>   return null;
> }
>   } catch (InterruptedException ie) {
> continue; // loop around.
>   }
> }
> {code}
> We could get rid of the while loop since {{Process#waitFor()}} blocks while 
> it completes.
> Note, that it's easy to workaround the issue by setting the USER environment 
> variable when running the tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SQOOP-3283) MySQL thirdparty tests hang if there's no USER environment variable

[Attila Szabo (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364532#comment-16364532
 ] 

Attila Szabo commented on SQOOP-3283:
-

Hey [~dvoros],

Thanks for this contribution!

One additional thought:

You might be considering to open a new ticket for changing the behaviour of the 
build.xml file (line 196) b/c that might also cause unexpected behaviour (like 
MySQL complaining around "${env.USER}" username) , especially for those who 
devs/users who are not playing with ant since Java 1.2 (you could find ore 
detials here: [https://ant.apache.org/manual/Tasks/property.html#notes-env] )

 

Thanks,

[~maugli]

> MySQL thirdparty tests hang if there's no USER environment variable
> ---
>
> Key: SQOOP-3283
> URL: https://issues.apache.org/jira/browse/SQOOP-3283
> Project: Sqoop
>  Issue Type: Bug
>  Components: connectors/mysql, test
>Affects Versions: 1.4.7
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Minor
> Attachments: SQOOP-3283.1.patch, SQOOP-3283.2.patch
>
>
> {{org.apache.sqoop.manager.mysql.MySQLTestUtils#getCurrentUser()}} executes 
> {{whoami}} in a subprocess if there's no USER environment variable (happened 
> to me while running tests from Docker). However, it waits for the Process 
> variable to become null, that never happens:
> {code:java}
> // wait for whoami to exit.
> while (p != null) {
>   try {
> int ret = p.waitFor();
> if (0 != ret) {
>   LOG.error("whoami exited with error status " + ret);
>   // suppress original return value from this method.
>   return null;
> }
>   } catch (InterruptedException ie) {
> continue; // loop around.
>   }
> }
> {code}
> We could get rid of the while loop since {{Process#waitFor()}} blocks while 
> it completes.
> Note, that it's easy to workaround the issue by setting the USER environment 
> variable when running the tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SQOOP-2331) Snappy Compression Support in Sqoop-HCatalog

[Attila Szabo (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-2331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350143#comment-16350143
 ] 

Attila Szabo commented on SQOOP-2331:
-

Hi [~standon],

I'd like to help you moving forward with this change.

I've canceled your latest patch b/c it has been diverged from the current trunk 
too much.

I've uploaded a suggestion (mainly your patch just resolved a few conflicts 
with difftools and wiggle, and removed com.clouder.* classes/refs). I'd like to 
kindly ask you to review it, and modify if it is needed.

When you find it satisfying please resubmit the patch, and also please open a 
reviewboard ticket for it, and link the board with this Jira ticket for the 
easier review.

 

Thanks in advance!

[~maugli]

> Snappy Compression Support in Sqoop-HCatalog
> 
>
> Key: SQOOP-2331
> URL: https://issues.apache.org/jira/browse/SQOOP-2331
> Project: Sqoop
>  Issue Type: New Feature
>Affects Versions: 1.4.5
>Reporter: Atul Gupta
>Assignee: Shashank
>Priority: Major
> Fix For: 1.5.0
>
> Attachments: SQOOP-2331_0.patch, SQOOP-2331_1.patch, 
> SQOOP-2331_2.patch
>
>
> Current Apache Sqoop 1.4.5 does not compress in gzip format with 
> --compress option while using with --hcatalog-table option. It also does not 
> support option --compression-codec snappy with --hcatalog-table option. it 
> would be nice if we add both the options in the Sqoop future releases.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (SQOOP-2331) Snappy Compression Support in Sqoop-HCatalog

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-2331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo updated SQOOP-2331:

Attachment: SQOOP-2331_2.patch

> Snappy Compression Support in Sqoop-HCatalog
> 
>
> Key: SQOOP-2331
> URL: https://issues.apache.org/jira/browse/SQOOP-2331
> Project: Sqoop
>  Issue Type: New Feature
>Affects Versions: 1.4.5
>Reporter: Atul Gupta
>Assignee: Shashank
>Priority: Major
> Fix For: 1.5.0
>
> Attachments: SQOOP-2331_0.patch, SQOOP-2331_1.patch, 
> SQOOP-2331_2.patch
>
>
> Current Apache Sqoop 1.4.5 does not compress in gzip format with 
> --compress option while using with --hcatalog-table option. It also does not 
> support option --compression-codec snappy with --hcatalog-table option. it 
> would be nice if we add both the options in the Sqoop future releases.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SQOOP-3054) Get FileSystem from parameter "--target-dir"

[Attila Szabo (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350005#comment-16350005
 ] 

Attila Szabo commented on SQOOP-3054:
-

Hi Angela ([~Ying Cao]),

I'm super glad you've taken the initiative in solving this problem!

I've seen your patch file, though I'm a bit concerned here. From the 
description it's not clear for me, how is this a bug, and how your patch will 
fix it. I'm also missing the information what kind of testing was done, or what 
kind of testing considerations did you make.

To being able to move forward with this ticket and patch I'd like to ask the 
following from you:

Please add more description about the problem (when does it occur, how to 
reproduce it, etc.)
Please open a reviewboard ticket, and link it to this JIRA.

If you have time (though it's not necessary, b/c the current one is applicable 
via wiggle) could you please rebase your patch according to the current trunk 
state?

 

If you have any further questions please do not hesitate to ping me!

 

Thanks in advance,

Attila

 

ps: I've assigned the issue to you, as you're the one who's working on it!

> Get FileSystem from parameter "--target-dir"
> 
>
> Key: SQOOP-3054
> URL: https://issues.apache.org/jira/browse/SQOOP-3054
> Project: Sqoop
>  Issue Type: Bug
>  Components: connectors
>Affects Versions: 1.4.6
> Environment: Redhat 6.6, Sqoop 1.4.6+Hadoop 2.7.2+Hive 1.2.1
>Reporter: Ying Cao
>Assignee: Ying Cao
>Priority: Critical
> Fix For: 1.5.0
>
> Attachments: SQOOP-3054.patch
>
>
> FileSystems gets from conf , update to get filesystem from parateter 
> "--target-dir"



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (SQOOP-3054) Get FileSystem from parameter "--target-dir"

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo reassigned SQOOP-3054:
---

Assignee: Ying Cao

> Get FileSystem from parameter "--target-dir"
> 
>
> Key: SQOOP-3054
> URL: https://issues.apache.org/jira/browse/SQOOP-3054
> Project: Sqoop
>  Issue Type: Bug
>  Components: connectors
>Affects Versions: 1.4.6
> Environment: Redhat 6.6, Sqoop 1.4.6+Hadoop 2.7.2+Hive 1.2.1
>Reporter: Ying Cao
>Assignee: Ying Cao
>Priority: Critical
> Fix For: 1.5.0
>
> Attachments: SQOOP-3054.patch
>
>
> FileSystems gets from conf , update to get filesystem from parateter 
> "--target-dir"



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SQOOP-3052) Introduce Maven/Gradle/etc. based build for Sqoop to make it more developer friendly / open

[Attila Szabo (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349602#comment-16349602
 ] 

Attila Szabo commented on SQOOP-3052:
-

Hey [~anna.szonyi],

I'm a bit uncertain here:

Are these patch files already for review? Could you please move the ticket into 
"Patch available" state if so?

Could you please also open reviewboard ticket, and link it to this Jira ticket?

 

Thanks,

[~maugli]

> Introduce Maven/Gradle/etc. based build for Sqoop to make it more developer 
> friendly / open
> ---
>
> Key: SQOOP-3052
> URL: https://issues.apache.org/jira/browse/SQOOP-3052
> Project: Sqoop
>  Issue Type: Improvement
>Reporter: Attila Szabo
>Assignee: Anna Szonyi
>Priority: Major
> Fix For: 1.5.0
>
> Attachments: SQOOP-3052.patch, SQOOP-3052.patch
>
>
> The current trunk version can only be build with Ant/Ivy combination, which 
> has some painful limitations (resolve is slow / needs to be tweaked to use 
> only caches, the current profile / variable based settings are not working in 
> IDEs out of the box, the current solution does not download the related 
> sources, etc.)
> It would be nice to provide a solution, which would give the possibility for 
> the developers to choose between the nowadays well used build infrsturctures 
> (e.g. Maven, Gradle, etc.). For this solution it would be also essential to 
> keep the different build files (if there is more then one) synchronized 
> easily, and the configuration wouldn't diverege by time. Test execution has 
> to be solved also, and should cover all the available test cases.
> In this scenario:
> If we can provide one good working solution is much better, then provide 
> three different ones which become out of sync easily. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (SQOOP-3277) Sqoop from Oracle RDB database is failing with exception SQLException in nextKeyValue

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo resolved SQOOP-3277.
-
Resolution: Not A Bug

> Sqoop from Oracle RDB database is failing with exception SQLException in 
> nextKeyValue
> -
>
> Key: SQOOP-3277
> URL: https://issues.apache.org/jira/browse/SQOOP-3277
> Project: Sqoop
>  Issue Type: Bug
>  Components: test
>Affects Versions: 1.4.6
> Environment: Production :
> Sqoop from Oracle RDB database is failing with exception SQLException in 
> nextKeyValue
> Sqoop from Oracle RDB database is failing with exception SQLException in 
> nextKeyValue. Below is the error log
> 17/12/22 11:20:35 INFO Configuration.deprecation: mapred.job.queue.name is 
> deprecated. Instead, use mapreduce.job.queuename
> 17/12/22 11:20:35 INFO mapreduce.JobSubmitter: Submitting tokens for job: 
> job_1512111796128_818880
> 17/12/22 11:20:35 INFO security.ExternalTokenManagerFactory: Initialized 
> external token manager class - com.mapr.hadoop.yarn.security.MapRTicketManager
> 17/12/22 11:20:35 INFO impl.YarnClientImpl: Submitted application 
> application_1512111796128_818880
> 17/12/22 11:20:35 INFO mapreduce.Job: The url to track the job: 
> https://dbsld0069.uhc.com:8090/proxy/application_1512111796128_818880/
> 17/12/22 11:20:35 INFO mapreduce.Job: Running job: job_1512111796128_818880
> 17/12/22 11:20:43 INFO mapreduce.Job: Job job_1512111796128_818880 running in 
> uber mode : false
> 17/12/22 11:20:43 INFO mapreduce.Job:  map 0% reduce 0%
> 17/12/22 15:13:18 INFO mapreduce.Job: Task Id : 
> attempt_1512111796128_818880_m_00_0, Status : FAILED
> Error: java.io.IOException: SQLException in nextKeyValue
> at 
> org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:277)
> at 
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:565)
> at 
> org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
> at 
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
> at 
> org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:796)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:346)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: java.sql.SQLRecoverableException: IO Error: End of TNS data channel
> at 
> oracle.jdbc.driver.T4CPreparedStatement.fetch(T4CPreparedStatement.java:1128)
> at 
> oracle.jdbc.driver.OracleResultSetImpl.close_or_fetch_from_next(OracleResultSetImpl.java:373)
> at oracle.jdbc.driver.OracleResultSetImpl.next(OracleResultSetImpl.java:277)
> at 
> org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:237)
> ... 12 more
> Caused by: oracle.net.ns.NetException: End of TNS data channel
> at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:308)
> at oracle.net.ns.NetInputStream.read(NetInputStream.java:260)
> at oracle.net.ns.NetInputStream.read(NetInputStream.java:185)
> at oracle.net.ns.NetInputStream.read(NetInputStream.java:102)
> at 
> oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:124)
> at 
> oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:80)
> at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1137)
> at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:290)
> at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192)
> at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:531)
> at 
> oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:207)
> at 
> oracle.jdbc.driver.T4CPreparedStatement.fetch(T4CPreparedStatement.java:1119)
> ... 15 more
>  
>  
> --> Database from which we are sqooping data is Oracle RDB. We have tried out 
> different work arounds.
> 1. Mapping all the columns to String while sqooping
> 2. Handling the Date/timestamp fields i.e, mapping -00-00 value as null 
> while sqooping.
> This work around successfully sqooped most of the tables but still we are 
> facing the exception for few tables.
> Does any one came across this issue whie sqooping? Could you please provide 
> any suggestions/help on how to resolve this
>Reporter: Chaitanya Krishna Kande
>Assignee: Attila Szabo
>Priority: Blocker
>
> Sqoop from Oracle RDB database is 

[jira] [Commented] (SQOOP-3277) Sqoop from Oracle RDB database is failing with exception SQLException in nextKeyValue

[Attila Szabo (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349582#comment-16349582
 ] 

Attila Szabo commented on SQOOP-3277:
-

Hi Chaitanya,

>From the stack trace it seems the Oracle closed the connection/resultset (End 
>of TNS data channel) by some reason which is quite unknown for us from these 
>information, and also very much unrelated to the Sqoop codebase.

Though I've got some ideas you should check:

The stability/reliability of your network between the Sqoop cluster and your 
Oracle RDBMS.

The timeout settings/values in your database.

 

As this issue doesn't seem to be Sqoop related I'm closing it.

 

Kind regards,

Attila

> Sqoop from Oracle RDB database is failing with exception SQLException in 
> nextKeyValue
> -
>
> Key: SQOOP-3277
> URL: https://issues.apache.org/jira/browse/SQOOP-3277
> Project: Sqoop
>  Issue Type: Bug
>  Components: test
>Affects Versions: 1.4.6
> Environment: Production :
> Sqoop from Oracle RDB database is failing with exception SQLException in 
> nextKeyValue
> Sqoop from Oracle RDB database is failing with exception SQLException in 
> nextKeyValue. Below is the error log
> 17/12/22 11:20:35 INFO Configuration.deprecation: mapred.job.queue.name is 
> deprecated. Instead, use mapreduce.job.queuename
> 17/12/22 11:20:35 INFO mapreduce.JobSubmitter: Submitting tokens for job: 
> job_1512111796128_818880
> 17/12/22 11:20:35 INFO security.ExternalTokenManagerFactory: Initialized 
> external token manager class - com.mapr.hadoop.yarn.security.MapRTicketManager
> 17/12/22 11:20:35 INFO impl.YarnClientImpl: Submitted application 
> application_1512111796128_818880
> 17/12/22 11:20:35 INFO mapreduce.Job: The url to track the job: 
> https://dbsld0069.uhc.com:8090/proxy/application_1512111796128_818880/
> 17/12/22 11:20:35 INFO mapreduce.Job: Running job: job_1512111796128_818880
> 17/12/22 11:20:43 INFO mapreduce.Job: Job job_1512111796128_818880 running in 
> uber mode : false
> 17/12/22 11:20:43 INFO mapreduce.Job:  map 0% reduce 0%
> 17/12/22 15:13:18 INFO mapreduce.Job: Task Id : 
> attempt_1512111796128_818880_m_00_0, Status : FAILED
> Error: java.io.IOException: SQLException in nextKeyValue
> at 
> org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:277)
> at 
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:565)
> at 
> org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
> at 
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
> at 
> org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:796)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:346)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: java.sql.SQLRecoverableException: IO Error: End of TNS data channel
> at 
> oracle.jdbc.driver.T4CPreparedStatement.fetch(T4CPreparedStatement.java:1128)
> at 
> oracle.jdbc.driver.OracleResultSetImpl.close_or_fetch_from_next(OracleResultSetImpl.java:373)
> at oracle.jdbc.driver.OracleResultSetImpl.next(OracleResultSetImpl.java:277)
> at 
> org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:237)
> ... 12 more
> Caused by: oracle.net.ns.NetException: End of TNS data channel
> at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:308)
> at oracle.net.ns.NetInputStream.read(NetInputStream.java:260)
> at oracle.net.ns.NetInputStream.read(NetInputStream.java:185)
> at oracle.net.ns.NetInputStream.read(NetInputStream.java:102)
> at 
> oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:124)
> at 
> oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:80)
> at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1137)
> at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:290)
> at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192)
> at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:531)
> at 
> oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:207)
> at 
> oracle.jdbc.driver.T4CPreparedStatement.fetch(T4CPreparedStatement.java:1119)
> ... 15 more
>  
>  
> --> Database from which we are sqooping data is Oracle RDB. We have tried out 
> different work arounds.
> 1. Mapping 

[jira] [Assigned] (SQOOP-3277) Sqoop from Oracle RDB database is failing with exception SQLException in nextKeyValue

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo reassigned SQOOP-3277:
---

Assignee: Attila Szabo

> Sqoop from Oracle RDB database is failing with exception SQLException in 
> nextKeyValue
> -
>
> Key: SQOOP-3277
> URL: https://issues.apache.org/jira/browse/SQOOP-3277
> Project: Sqoop
>  Issue Type: Bug
>  Components: test
>Affects Versions: 1.4.6
> Environment: Production :
> Sqoop from Oracle RDB database is failing with exception SQLException in 
> nextKeyValue
> Sqoop from Oracle RDB database is failing with exception SQLException in 
> nextKeyValue. Below is the error log
> 17/12/22 11:20:35 INFO Configuration.deprecation: mapred.job.queue.name is 
> deprecated. Instead, use mapreduce.job.queuename
> 17/12/22 11:20:35 INFO mapreduce.JobSubmitter: Submitting tokens for job: 
> job_1512111796128_818880
> 17/12/22 11:20:35 INFO security.ExternalTokenManagerFactory: Initialized 
> external token manager class - com.mapr.hadoop.yarn.security.MapRTicketManager
> 17/12/22 11:20:35 INFO impl.YarnClientImpl: Submitted application 
> application_1512111796128_818880
> 17/12/22 11:20:35 INFO mapreduce.Job: The url to track the job: 
> https://dbsld0069.uhc.com:8090/proxy/application_1512111796128_818880/
> 17/12/22 11:20:35 INFO mapreduce.Job: Running job: job_1512111796128_818880
> 17/12/22 11:20:43 INFO mapreduce.Job: Job job_1512111796128_818880 running in 
> uber mode : false
> 17/12/22 11:20:43 INFO mapreduce.Job:  map 0% reduce 0%
> 17/12/22 15:13:18 INFO mapreduce.Job: Task Id : 
> attempt_1512111796128_818880_m_00_0, Status : FAILED
> Error: java.io.IOException: SQLException in nextKeyValue
> at 
> org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:277)
> at 
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:565)
> at 
> org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
> at 
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
> at 
> org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:796)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:346)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: java.sql.SQLRecoverableException: IO Error: End of TNS data channel
> at 
> oracle.jdbc.driver.T4CPreparedStatement.fetch(T4CPreparedStatement.java:1128)
> at 
> oracle.jdbc.driver.OracleResultSetImpl.close_or_fetch_from_next(OracleResultSetImpl.java:373)
> at oracle.jdbc.driver.OracleResultSetImpl.next(OracleResultSetImpl.java:277)
> at 
> org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:237)
> ... 12 more
> Caused by: oracle.net.ns.NetException: End of TNS data channel
> at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:308)
> at oracle.net.ns.NetInputStream.read(NetInputStream.java:260)
> at oracle.net.ns.NetInputStream.read(NetInputStream.java:185)
> at oracle.net.ns.NetInputStream.read(NetInputStream.java:102)
> at 
> oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:124)
> at 
> oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:80)
> at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1137)
> at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:290)
> at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192)
> at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:531)
> at 
> oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:207)
> at 
> oracle.jdbc.driver.T4CPreparedStatement.fetch(T4CPreparedStatement.java:1119)
> ... 15 more
>  
>  
> --> Database from which we are sqooping data is Oracle RDB. We have tried out 
> different work arounds.
> 1. Mapping all the columns to String while sqooping
> 2. Handling the Date/timestamp fields i.e, mapping -00-00 value as null 
> while sqooping.
> This work around successfully sqooped most of the tables but still we are 
> facing the exception for few tables.
> Does any one came across this issue whie sqooping? Could you please provide 
> any suggestions/help on how to resolve this
>Reporter: Chaitanya Krishna Kande
>Assignee: Attila Szabo
>Priority: Blocker
>
> Sqoop from Oracle RDB database 

[jira] [Resolved] (SQOOP-3199) Sqoop 1.4.7 release preparation

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo resolved SQOOP-3199.
-
Resolution: Fixed

> Sqoop 1.4.7 release preparation
> ---
>
> Key: SQOOP-3199
> URL: https://issues.apache.org/jira/browse/SQOOP-3199
> Project: Sqoop
>  Issue Type: Task
>Reporter: Attila Szabo
>Assignee: Attila Szabo
>Priority: Major
> Fix For: no-release
>
> Attachments: text.html
>
>
> Umbrella jira for 1.4.7 release.
> For reference, the release wikis are:
> https://cwiki.apache.org/confluence/display/SQOOP/How+to+Release
> https://cwiki.apache.org/confluence/display/SQOOP/How+to+Release+Sqoop2



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (SQOOP-3204) Create release artifacts

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo resolved SQOOP-3204.
-
Resolution: Fixed

> Create release artifacts
> 
>
> Key: SQOOP-3204
> URL: https://issues.apache.org/jira/browse/SQOOP-3204
> Project: Sqoop
>  Issue Type: Sub-task
>Reporter: Attila Szabo
>Assignee: Attila Szabo
>Priority: Major
> Fix For: no-release
>
>
> create tarballs
> update keys
> sign and checksum
> upload to private space



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (SQOOP-3270) Introduce specified date parameter for write-version-info.sh

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo updated SQOOP-3270:

Affects Version/s: 1.5.0

> Introduce specified date parameter for write-version-info.sh
> 
>
> Key: SQOOP-3270
> URL: https://issues.apache.org/jira/browse/SQOOP-3270
> Project: Sqoop
>  Issue Type: Improvement
>Affects Versions: 1.5.0
>Reporter: Attila Szabo
>Assignee: Attila Szabo
>
> It's quite difficult to create reproducible build artifacts, if the source 
> itself changes continuously.
> It would make sense to introduce a parameter to fix the build date (e.g. in 
> case of preparing a release candidate, etc.).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (SQOOP-3270) Introduce specified date parameter for write-version-info.sh

[Attila Szabo (JIRA)]

Attila Szabo created SQOOP-3270:
---

 Summary: Introduce specified date parameter for 
write-version-info.sh
 Key: SQOOP-3270
 URL: https://issues.apache.org/jira/browse/SQOOP-3270
 Project: Sqoop
  Issue Type: Improvement
Reporter: Attila Szabo
Assignee: Attila Szabo


It's quite difficult to create reproducible build artifacts, if the source 
itself changes continuously.
It would make sense to introduce a parameter to fix the build date (e.g. in 
case of preparing a release candidate, etc.).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (SQOOP-3269) Update last prev.git.hash property in build.xml

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo resolved SQOOP-3269.
-
Resolution: Fixed

> Update last prev.git.hash property in build.xml
> ---
>
> Key: SQOOP-3269
> URL: https://issues.apache.org/jira/browse/SQOOP-3269
> Project: Sqoop
>  Issue Type: Bug
>Reporter: Attila Szabo
>Assignee: Attila Szabo
>Priority: Critical
>
> It was not mentioned under 
> https://cwiki.apache.org/confluence/display/SQOOP/How+to+Release though it is 
> needed according to the release history.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SQOOP-3269) Update last prev.git.hash property in build.xml

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo updated SQOOP-3269:

Affects Version/s: (was: 1.5.0)

> Update last prev.git.hash property in build.xml
> ---
>
> Key: SQOOP-3269
> URL: https://issues.apache.org/jira/browse/SQOOP-3269
> Project: Sqoop
>  Issue Type: Bug
>Reporter: Attila Szabo
>Assignee: Attila Szabo
>Priority: Critical
>
> It was not mentioned under 
> https://cwiki.apache.org/confluence/display/SQOOP/How+to+Release though it is 
> needed according to the release history.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SQOOP-3269) Update last prev.git.hash property in build.xml

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo updated SQOOP-3269:

Affects Version/s: 1.5.0

> Update last prev.git.hash property in build.xml
> ---
>
> Key: SQOOP-3269
> URL: https://issues.apache.org/jira/browse/SQOOP-3269
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.5.0
>Reporter: Attila Szabo
>Assignee: Attila Szabo
>Priority: Critical
>
> It was not mentioned under 
> https://cwiki.apache.org/confluence/display/SQOOP/How+to+Release though it is 
> needed according to the release history.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (SQOOP-3269) Update last prev.git.hash property in build.xml

[Attila Szabo (JIRA)]

Attila Szabo created SQOOP-3269:
---

 Summary: Update last prev.git.hash property in build.xml
 Key: SQOOP-3269
 URL: https://issues.apache.org/jira/browse/SQOOP-3269
 Project: Sqoop
  Issue Type: Bug
Reporter: Attila Szabo
Assignee: Attila Szabo
Priority: Critical


It was not mentioned under 
https://cwiki.apache.org/confluence/display/SQOOP/How+to+Release though it is 
needed according to the release history.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SQOOP-3267) Incremental import to HBase deletes only last version of column

[Attila Szabo (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16292590#comment-16292590
 ] 

Attila Szabo commented on SQOOP-3267:
-

Hey [~dvoros], [~vasas],

IMHO I would keep the history by default, and if the (b/c of the existing cmd 
line arguments, and b/c as an end user I would really get my data deleted 
without explicitly requesting that).

Aggregating your findings and my thoughts my recommendations are the following:
By default (no other options present) I would insert null value, and keep the 
history.
If the mode aims for the last modified entry only, I would delete the history, 
and only keep the last meaningful value (and of course in case of null value 
delete the column as you've suggested). I would definitely go with this 
direction, b/c we're speaking about incremental mode, and according to the 
existing doucmentation 'mode' is related to incremental mode (and we did not 
made any differentiation for incremental mode with append only tables and 
incremental mode for HBase where we can do "real" modificaitons).

Though if you dislike using and leveraging from the mode cmd line argument, I'm 
not against to introduce new cmd line arguments on this front, for making it 
straightfwd, when we do deletes, when we insert null values, when we keep 
history and when we do not. Although in this case I would also highly recommend 
to introduce some fail fast scenario (form 1.5 version) which would give a 
meaningful error message in case of mode+HBase table+incremental import.

My 2cents,
Attila

ps.: [~vasas] your test cases are very well defined, and very detailed! Nice 
job!!!

> Incremental import to HBase deletes only last version of column
> ---
>
> Key: SQOOP-3267
> URL: https://issues.apache.org/jira/browse/SQOOP-3267
> Project: Sqoop
>  Issue Type: Bug
>  Components: hbase-integration
>Affects Versions: 1.4.7
>Reporter: Daniel Voros
>Assignee: Daniel Voros
> Attachments: SQOOP-3267.1.patch
>
>
> Deletes are supported since SQOOP-3149, but we're only deleting the last 
> version of a column when the corresponding cell was set to NULL in the source 
> table.
> This can lead to unexpected and misleading results if the row has been 
> transferred multiple times, which can easily happen if it's being modified on 
> the source side.
> Also SQOOP-3149 is using a new Put command for every column instead of a 
> single Put per row as before. This could probably lead to a performance drop 
> for wide tables (for which HBase is otherwise usually recommended).
> [~jilani], [~anna.szonyi] could you please comment on what you think would be 
> the expected behavior here?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SQOOP-3199) Sqoop 1.4.7 release preparation

[Attila Szabo (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279773#comment-16279773
 ] 

Attila Szabo commented on SQOOP-3199:
-

[~BoglarkaEgyed],

Requested changes made on BRANCH-1.4.7.

Could you please validated it on your side as well?

What do you think? Until the LICENESE.txt questions are replied by the PMC, 
should we start a vote on an RC0  version?

Many thanks in advance,
[~maugli]

> Sqoop 1.4.7 release preparation
> ---
>
> Key: SQOOP-3199
> URL: https://issues.apache.org/jira/browse/SQOOP-3199
> Project: Sqoop
>  Issue Type: Task
>Reporter: Attila Szabo
>Assignee: Attila Szabo
> Fix For: no-release
>
> Attachments: text.html
>
>
> Umbrella jira for 1.4.7 release.
> For reference, the release wikis are:
> https://cwiki.apache.org/confluence/display/SQOOP/How+to+Release
> https://cwiki.apache.org/confluence/display/SQOOP/How+to+Release+Sqoop2



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SQOOP-3267) Incremental import to HBase deletes only last version of column

[Attila Szabo (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279496#comment-16279496
 ] 

Attila Szabo commented on SQOOP-3267:
-

Hey [~dvoros],

I've checked your proposed changes, and become a bit concerned:
According to your patch you would not differentiate Sqoop's behavior depending 
on cmd line argument mode.

I do understand in case of "lastmodified" mode it would make sense to delete 
all the previous versions of the column (though I guess in this case we should 
delete previous column values even in case of the "normal" updates).

But IMHO in case of "append" mode, we should only delete the last version of 
the column, to keep the history, as that is suggested by the mode itself.

What do you think?
[~maugli]

> Incremental import to HBase deletes only last version of column
> ---
>
> Key: SQOOP-3267
> URL: https://issues.apache.org/jira/browse/SQOOP-3267
> Project: Sqoop
>  Issue Type: Bug
>  Components: hbase-integration
>Affects Versions: 1.4.7
>Reporter: Daniel Voros
>Assignee: Daniel Voros
> Attachments: SQOOP-3267.1.patch
>
>
> Deletes are supported since SQOOP-3149, but we're only deleting the last 
> version of a column when the corresponding cell was set to NULL in the source 
> table.
> This can lead to unexpected and misleading results if the row has been 
> transferred multiple times, which can easily happen if it's being modified on 
> the source side.
> Also SQOOP-3149 is using a new Put command for every column instead of a 
> single Put per row as before. This could probably lead to a performance drop 
> for wide tables (for which HBase is otherwise usually recommended).
> [~jilani], [~anna.szonyi] could you please comment on what you think would be 
> the expected behavior here?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (SQOOP-3268) Remove duplicates and issues with incorrect resolve status from CHANGELOG.txt, PLUS include SQOOP-3226 in 1.4.7 release

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo resolved SQOOP-3268.
-
Resolution: Fixed

> Remove duplicates and issues with incorrect resolve status from 
> CHANGELOG.txt, PLUS include SQOOP-3226 in 1.4.7 release
> ---
>
> Key: SQOOP-3268
> URL: https://issues.apache.org/jira/browse/SQOOP-3268
> Project: Sqoop
>  Issue Type: Sub-task
>Reporter: Attila Szabo
>Assignee: Attila Szabo
> Fix For: no-release
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (SQOOP-3268) Remove duplicates and issues with incorrect resolve status from CHANGELOG.txt, PLUS include SQOOP-3226 in 1.4.7 release

[Attila Szabo (JIRA)]

Attila Szabo created SQOOP-3268:
---

 Summary: Remove duplicates and issues with incorrect resolve 
status from CHANGELOG.txt, PLUS include SQOOP-3226 in 1.4.7 release
 Key: SQOOP-3268
 URL: https://issues.apache.org/jira/browse/SQOOP-3268
 Project: Sqoop
  Issue Type: Sub-task
Reporter: Attila Szabo
Assignee: Attila Szabo
 Fix For: no-release






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (SQOOP-3205) Update KEYS file

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo resolved SQOOP-3205.
-
Resolution: Fixed

> Update KEYS file
> 
>
> Key: SQOOP-3205
> URL: https://issues.apache.org/jira/browse/SQOOP-3205
> Project: Sqoop
>  Issue Type: Sub-task
>Reporter: Attila Szabo
>Assignee: Attila Szabo
> Fix For: no-release
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SQOOP-3205) Update KEYS file

[Attila Szabo (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279366#comment-16279366
 ] 

Attila Szabo commented on SQOOP-3205:
-

Keys file has been updated.

Thank you very much [~kathleen] for your invaluable help with this issue!

> Update KEYS file
> 
>
> Key: SQOOP-3205
> URL: https://issues.apache.org/jira/browse/SQOOP-3205
> Project: Sqoop
>  Issue Type: Sub-task
>Reporter: Attila Szabo
>Assignee: Attila Szabo
> Fix For: no-release
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SQOOP-2939) Extend mainframe module to support GDG, sequential data sets, and data sets stored on tape

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-2939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo updated SQOOP-2939:

Fix Version/s: (was: 1.4.7)
   no-release

> Extend mainframe module to support GDG, sequential data sets, and data sets 
> stored on tape
> --
>
> Key: SQOOP-2939
> URL: https://issues.apache.org/jira/browse/SQOOP-2939
> Project: Sqoop
>  Issue Type: Improvement
>  Components: connectors
>Affects Versions: 1.4.6
> Environment: mainframe
>Reporter: Chris Teoh
>Assignee: Chris Teoh
>Priority: Minor
>  Labels: patch
> Fix For: no-release
>
>
> The current mainframe module supports partitioned data sets only.
> I propose adding the following improvements:- 
> - sequential data set support
> - GDG data set support
> I propose the following parameters to be added:-
> 1. --datasettype   (p = partitioned data set, s = sequential, g = GDG)
> 2. --tape  (true = data set is on tape, false = data set it not 
> on tape)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SQOOP-3187) Sqoop import as PARQUET to S3 failed

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo updated SQOOP-3187:

Fix Version/s: (was: 1.4.7)
   no-release

> Sqoop import as PARQUET to S3 failed
> 
>
> Key: SQOOP-3187
> URL: https://issues.apache.org/jira/browse/SQOOP-3187
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.4.6
>Reporter: Surendra Nichenametla
>Assignee: Sandish Kumar HN
> Fix For: no-release
>
>
> Sqoop import as parquet file to S3 fails. Command and error are give below.
> However, import to a HDFS location works though.
> sqoop import --connect "jdbc:oracle:thin:@:1521/ORCL" --table 
> mytable --username myuser --password mypass --target-dir s3://bucket/foo/bar/ 
> --columns col1,col2 -m1 --as-parquetfile
> 17/05/09 21:00:18 ERROR tool.ImportTool: Imported Failed: Wrong FS: 
> s3://bucket/foo/bar, expected: hdfs://master-ip:8020
> P.S. I tried this from Amazon EMR cluster.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SQOOP-3182) Sqoop1 (import + --incremental + --merge-key + --as-parquetfile) fails with (Can't parse input data: 'PAR1')

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo updated SQOOP-3182:

Fix Version/s: (was: 1.4.7)
   no-release

> Sqoop1 (import + --incremental + --merge-key + --as-parquetfile) fails with 
> (Can't parse input data: 'PAR1')
> 
>
> Key: SQOOP-3182
> URL: https://issues.apache.org/jira/browse/SQOOP-3182
> Project: Sqoop
>  Issue Type: Bug
>Reporter: Markus Kemper
>Assignee: Sandish Kumar HN
> Fix For: no-release
>
>
> Sqoop1 (import + --incremental + --merge-key + --as-parquet) fails with 
> (Can't parse input data: 'PAR1').  See test case below.
> *Test Case*
> {noformat}
> #
> # STEP 01 - Create Table and Data
> #
> export MYCONN=jdbc:oracle:thin:@oracle.sqoop.com:1521/db11g;
> export MYUSER=sqoop
> export MYPSWD=sqoop
> sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query 
> "drop table t1"
> sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query 
> "create table t1 (c1 int, c2 date, c3 varchar(10), c4 timestamp)"
> sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query 
> "insert into t1 values (1, sysdate, 'NEW ROW 1', sysdate)"
> sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query 
> "select * from t1"
> Output:
> -
> | C1   | C2  | C3 | C4  | 
> -
> | 1| 2017-05-06 06:59:02.0 | NEW ROW 1  | 2017-05-06 
> 06:59:02 | 
> -
> #
> # STEP 02 - Import Data into HDFS 
> #
> hdfs dfs -rm -r /user/root/t1
> sqoop import --connect $MYCONN --username $MYUSER --password $MYPSWD --table 
> T1 --target-dir /user/root/t1 --incremental lastmodified --check-column C4 
> --merge-key C1 --last-value '2017-01-01 00:00:00.0' --as-parquetfile 
> --map-column-java C2=String,C4=String --num-mappers 1 --verbose 
> hdfs dfs -ls /user/root/t1/*.parquet
> parquet-tools cat --json 
> 'hdfs://namenode/user/root/t1/b65c1ca5-c8f0-44c6-8c60-8ee83161347f.parquet'
> Output:
> 17/05/06 07:01:34 INFO mapreduce.ImportJobBase: Transferred 2.627 KB in 
> 23.6174 seconds (113.8988 bytes/sec)
> 17/05/06 07:01:34 INFO mapreduce.ImportJobBase: Retrieved 1 records.
> 17/05/06 07:01:34 INFO tool.ImportTool:   --last-value 2017-05-06 07:01:09.0
> ~
> -rw-r--r--   3 root root   1144 2017-05-06 07:01 
> /user/root/t1/b65c1ca5-c8f0-44c6-8c60-8ee83161347f.parquet
> ~
> {"C1":"1","C2":"2017-05-06 06:59:02.0","C3":"NEW ROW 1","C4":"2017-05-06 
> 06:59:02"}
> #
> # STEP 03 - Insert New Row and Update Existing Row
> #
> sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query 
> "insert into t1 values (2, sysdate, 'NEW ROW 2', sysdate)"
> sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query 
> "update t1 set c3 = 'UPDATE 1', c4 = sysdate where c1 = 1"
> sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query 
> "select * from t1 order by c1"
> Output:
> -
> | C1   | C2  | C3 | C4  | 
> -
> | 1| 2017-05-06 06:59:02.0 | UPDATE 1   | 2017-05-06 
> 07:04:40 | 
> | 2| 2017-05-06 07:04:38.0 | NEW ROW 2  | 2017-05-06 
> 07:04:38 | 
> -
> #
> # STEP 04 - Import Data into HDFS and Merge changes 
> #
> sqoop import --connect $MYCONN --username $MYUSER --password $MYPSWD --table 
> T1 --target-dir /user/root/t1 --incremental lastmodified --check-column C4 
> --merge-key C1 --last-value '2017-05-06 07:01:09.0' --as-parquetfile 
> --map-column-java C2=String,C4=String --num-mappers 1 --verbose 
> Output:
> 17/05/06 07:06:43 INFO mapreduce.ImportJobBase: Transferred 2.6611 KB in 
> 27.4934 seconds (99.1148 bytes/sec)
> 17/05/06 07:06:43 INFO mapreduce.ImportJobBase: Retrieved 2 records.
> 17/05/06 07:06:43 DEBUG util.ClassLoaderStack: Restoring classloader: 
> java.net.FactoryURLClassLoader@121fdcee
> 17/05/06 07:06:43 INFO tool.ImportTool: Final destination exists, will run 
> merge job.
> 17/05/06 07:06:43 DEBUG tool.ImportTool: Using temporary folder: 
> 4bc6b65cd0194b81938f4660974ee392_T1
> 17/05/06 07:06:43 DEBUG util.ClassLoaderStack: Checking for existing class: T1
> 17/05/06 07:06:43 DEBUG util.ClassLoaderStack: Attempting to load jar through 
> URL: 
> 

[jira] [Updated] (SQOOP-2850) Append mode for hive imports is not yet supported. Please remove the parameter --append-mode

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-2850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo updated SQOOP-2850:

Fix Version/s: (was: 1.4.7)
   no-release

> Append mode for hive imports is not  yet supported. Please remove the 
> parameter --append-mode
> -
>
> Key: SQOOP-2850
> URL: https://issues.apache.org/jira/browse/SQOOP-2850
> Project: Sqoop
>  Issue Type: Bug
>  Components: hive-integration
>Affects Versions: 1.4.6
> Environment: sqoop-1.4.6
> hadoop-2.6.0
> hive-1.2.0
>Reporter: alexBai
> Fix For: no-release
>
>
> sqoop job --meta-connect jdbc:hsqldb:hsql://xxx:16000/sqoop  --create myjob 
> -- import --connect jdbc:mysql://xxx:3306/newark --table test --username test 
> --password 123456 --hive-import --incremental append --check-column id 
> --last-value 1
> Warning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will 
> fail.
> Please set $ACCUMULO_HOME to the root of your Accumulo installation.
> 16/02/26 11:52:09 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
> 16/02/26 11:52:10 WARN tool.BaseSqoopTool: Setting your password on the 
> command-line is insecure. Consider using -P instead.
> 16/02/26 11:52:10 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for 
> output. You can override
> 16/02/26 11:52:10 INFO tool.BaseSqoopTool: delimiters with 
> --fields-terminated-by, etc.
> Append mode for hive imports is not  yet supported. Please remove the 
> parameter --append-mode



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SQOOP-2751) Test TestIncrementalImport is failing

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-2751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo updated SQOOP-2751:

Fix Version/s: (was: 1.4.7)
   no-release

> Test TestIncrementalImport is failing
> -
>
> Key: SQOOP-2751
> URL: https://issues.apache.org/jira/browse/SQOOP-2751
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.4.6
>Reporter: Jarek Jarcec Cecho
>Assignee: Jarek Jarcec Cecho
> Fix For: no-release
>
> Attachments: SQOOP-2751.patch
>
>
> With exception:
> {code}
> 2015-12-20 08:58:48,059 (main) [ERROR - 
> org.apache.sqoop.tool.ImportTool.run(ImportTool.java:613)] Encountered 
> IOException running import job: java.io.IOException: java.sql.SQLException: 
> Column not found: id in statement [SELECT MAX("id") FROM 
> "incrementalHiveAppendEmptyThenFull"]
>   at 
> org.apache.sqoop.tool.ImportTool.initIncrementalConstraints(ImportTool.java:316)
>   at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:488)
>   at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
>   at org.apache.sqoop.tool.JobTool.execJob(JobTool.java:228)
>   at org.apache.sqoop.tool.JobTool.run(JobTool.java:283)
>   at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>   at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
>   at com.cloudera.sqoop.Sqoop.runSqoop(Sqoop.java:45)
>   at 
> com.cloudera.sqoop.TestIncrementalImport.runJob(TestIncrementalImport.java:632)
>   at 
> com.cloudera.sqoop.TestIncrementalImport.runJob(TestIncrementalImport.java:616)
>   at 
> com.cloudera.sqoop.TestIncrementalImport.testIncrementalHiveAppendEmptyThenFull(TestIncrementalImport.java:1252)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:483)
>   at junit.framework.TestCase.runTest(TestCase.java:176)
>   at junit.framework.TestCase.runBare(TestCase.java:141)
>   at junit.framework.TestResult$1.protect(TestResult.java:122)
>   at junit.framework.TestResult.runProtected(TestResult.java:142)
>   at junit.framework.TestResult.run(TestResult.java:125)
>   at junit.framework.TestCase.run(TestCase.java:129)
>   at junit.framework.TestSuite.runTest(TestSuite.java:255)
>   at junit.framework.TestSuite.run(TestSuite.java:250)
>   at 
> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:535)
>   at 
> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1182)
>   at 
> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1033)
> Caused by: java.sql.SQLException: Column not found: id in statement [SELECT 
> MAX("id") FROM "incrementalHiveAppendEmptyThenFull"]
>   at org.hsqldb.jdbc.Util.sqlException(Unknown Source)
>   at org.hsqldb.jdbc.jdbcStatement.fetchResult(Unknown Source)
>   at org.hsqldb.jdbc.jdbcStatement.executeQuery(Unknown Source)
>   at org.apache.sqoop.tool.ImportTool.getMaxColumnId(ImportTool.java:231)
>   at 
> org.apache.sqoop.tool.ImportTool.initIncrementalConstraints(ImportTool.java:303)
>   ... 26 more
> {code}
> It seems that the problem is that the test 
> {{testIncrementalHiveAppendEmptyThenFull}} is not properly specifying the 
> column name in upper case.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (SQOOP-2937) Sqoop mainframe module does not support sequential data sets, GDG

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo resolved SQOOP-2937.
-
Resolution: Duplicate

> Sqoop mainframe module does not support sequential data sets, GDG
> -
>
> Key: SQOOP-2937
> URL: https://issues.apache.org/jira/browse/SQOOP-2937
> Project: Sqoop
>  Issue Type: Sub-task
>  Components: connectors
>Affects Versions: 1.4.6
> Environment: mainframe
>Reporter: Chris Teoh
>Assignee: Chris Teoh
>Priority: Minor
>  Labels: patch
> Fix For: no-release
>
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> The mainframe module (SQOOP-1272) currently supports partitioned data sets.
> I propose adding the following improvements:-
> - sequential data set support
> - GDG data set support
> I propose the following parameters to be added:-
> 1. --datasettype   (p = partitioned data set, s = sequential, g = GDG)
> Let me know if this should be all in one issue or separate issues



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SQOOP-2937) Sqoop mainframe module does not support sequential data sets, GDG

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo updated SQOOP-2937:

Fix Version/s: (was: 1.4.7)
   no-release

> Sqoop mainframe module does not support sequential data sets, GDG
> -
>
> Key: SQOOP-2937
> URL: https://issues.apache.org/jira/browse/SQOOP-2937
> Project: Sqoop
>  Issue Type: Sub-task
>  Components: connectors
>Affects Versions: 1.4.6
> Environment: mainframe
>Reporter: Chris Teoh
>Assignee: Chris Teoh
>Priority: Minor
>  Labels: patch
> Fix For: no-release
>
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> The mainframe module (SQOOP-1272) currently supports partitioned data sets.
> I propose adding the following improvements:-
> - sequential data set support
> - GDG data set support
> I propose the following parameters to be added:-
> 1. --datasettype   (p = partitioned data set, s = sequential, g = GDG)
> Let me know if this should be all in one issue or separate issues



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Reopened] (SQOOP-2937) Sqoop mainframe module does not support sequential data sets, GDG

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo reopened SQOOP-2937:
-

> Sqoop mainframe module does not support sequential data sets, GDG
> -
>
> Key: SQOOP-2937
> URL: https://issues.apache.org/jira/browse/SQOOP-2937
> Project: Sqoop
>  Issue Type: Sub-task
>  Components: connectors
>Affects Versions: 1.4.6
> Environment: mainframe
>Reporter: Chris Teoh
>Assignee: Chris Teoh
>Priority: Minor
>  Labels: patch
> Fix For: no-release
>
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> The mainframe module (SQOOP-1272) currently supports partitioned data sets.
> I propose adding the following improvements:-
> - sequential data set support
> - GDG data set support
> I propose the following parameters to be added:-
> 1. --datasettype   (p = partitioned data set, s = sequential, g = GDG)
> Let me know if this should be all in one issue or separate issues



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SQOOP-3264) Import JDBC SQL date,time,timestamp to Hive as TIMESTAMP, BIGINT and TIMESTAMP

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo updated SQOOP-3264:

Fix Version/s: (was: 1.4.7)
   1.5.0

> Import JDBC SQL date,time,timestamp to Hive as TIMESTAMP, BIGINT and TIMESTAMP
> --
>
> Key: SQOOP-3264
> URL: https://issues.apache.org/jira/browse/SQOOP-3264
> Project: Sqoop
>  Issue Type: Improvement
>  Components: hive-integration
>Affects Versions: 1.4.6
>Reporter: Michal Klempa
>Priority: Minor
> Fix For: 1.5.0
>
>
> When importing JDBC SQL  Types:
> {code}
> public final static int DATE=  91;
> public final static int TIME=  92;
> public final static int TIMESTAMP   =  93;
> {code}
> Sqoop currently uses the org.apache.sqoop.hive.HiveTypes.toHiveType method, 
> where all of these types are mapped to STRING type.
> Given that in fact, the JDBC value returned is of type Long, let me propose 
> we can output the type for Hive as:
> {code}
> DATE -> TIMESTAMP
> TIME -> BIGINT
> TIMESTAMP -> TIMESTAMP
> {code}
> This is also in line with org.apache.sqoop.manager.ConnManager.toAvroType, 
> where the type is 
> {code}
> case Types.DATE:
> case Types.TIME:
> case Types.TIMESTAMP:
>   return Type.LONG;
> {code}
> Some of the connectors override the toJavaType:
> {code}
> org.apache.sqoop.manager.SQLServerManager
> org.apache.sqoop.manager.oracle.OraOopConnManager
> {code}
> which may indicate different handling.
> The SQLServerManager uses Java String as the output type, because of 
> timezones.
> Same holds true for OraOopConnManager, although it has a separate 
> configuration boolean value 
> 'oraoop.timestamp.string' which controls whether the import will use 
> timezones and convert date types
> to Java String, or timezones are going to be dropped and import will behave 
> the 'sqoop way'.
> Both of these connectors already handle these types as String by default, 
> proposed change would not affect them.
> Other connectors are needed to be checked.
> Some of the connectors override the toHiveType:
> {code}
> org.apache.sqoop.manager.oracle.OraOopConnManager
> {code}
> This connector uses the 'sqoop way':
> {code}
> String hiveType = super.toHiveType(sqlType);
> {code}
> and only when not resolved, the type used is decided:
> {code}
> if (hiveType == null) {
>   // http://wiki.apache.org/hadoop/Hive/Tutorial#Primitive_Types
>   if (sqlType == OraOopOracleQueries.getOracleType("BFILE")
>   || sqlType == OraOopOracleQueries.getOracleType("INTERVALYM")
>   || sqlType == OraOopOracleQueries.getOracleType("INTERVALDS")
>   || sqlType == OraOopOracleQueries.getOracleType("NCLOB")
>   || sqlType == OraOopOracleQueries.getOracleType("NCHAR")
>   || sqlType == OraOopOracleQueries.getOracleType("NVARCHAR")
>   || sqlType == OraOopOracleQueries.getOracleType("OTHER")
>   || sqlType == OraOopOracleQueries.getOracleType("ROWID")
>   || sqlType == OraOopOracleQueries.getOracleType("TIMESTAMPTZ")
>   || sqlType == OraOopOracleQueries.getOracleType("TIMESTAMPLTZ")
>   || sqlType == OraOopOracleQueries.getOracleType("STRUCT")) {
> hiveType = "STRING";
>   }
>   if (sqlType == OraOopOracleQueries.getOracleType("BINARY_FLOAT")) {
> hiveType = "FLOAT";
>   }
>   if (sqlType == OraOopOracleQueries.getOracleType("BINARY_DOUBLE")) {
> hiveType = "DOUBLE";
>   }
> }
> {code}
> This code is affected with proposed change. As the Hive TIMESTAMP is 
> timezone-less, we have to change the handling in this method - respect the 
> property 'oraoop.timestamp.string' - if true, output STRING hive type, if 
> false, go with 'sqoop way'.
> The Hive Type is only used when generating the table ddl (create statement) 
> and Hive can properly recognize the  JDBC compliant java.sql.Timestamp format 
> "-MM-DD HH:MM:SS.f", so no connector should be affected in a way, 
> that Hive would not read the resulting column values.
> However, thorough testing should be done on all connectors before releasing 
> any column type behavior changes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (SQOOP-2937) Sqoop mainframe module does not support sequential data sets, GDG

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo resolved SQOOP-2937.
-
Resolution: Won't Fix

> Sqoop mainframe module does not support sequential data sets, GDG
> -
>
> Key: SQOOP-2937
> URL: https://issues.apache.org/jira/browse/SQOOP-2937
> Project: Sqoop
>  Issue Type: Sub-task
>  Components: connectors
>Affects Versions: 1.4.6
> Environment: mainframe
>Reporter: Chris Teoh
>Assignee: Chris Teoh
>Priority: Minor
>  Labels: patch
> Fix For: 1.4.7
>
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> The mainframe module (SQOOP-1272) currently supports partitioned data sets.
> I propose adding the following improvements:-
> - sequential data set support
> - GDG data set support
> I propose the following parameters to be added:-
> 1. --datasettype   (p = partitioned data set, s = sequential, g = GDG)
> Let me know if this should be all in one issue or separate issues



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Reopened] (SQOOP-2937) Sqoop mainframe module does not support sequential data sets, GDG

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo reopened SQOOP-2937:
-

> Sqoop mainframe module does not support sequential data sets, GDG
> -
>
> Key: SQOOP-2937
> URL: https://issues.apache.org/jira/browse/SQOOP-2937
> Project: Sqoop
>  Issue Type: Sub-task
>  Components: connectors
>Affects Versions: 1.4.6
> Environment: mainframe
>Reporter: Chris Teoh
>Assignee: Chris Teoh
>Priority: Minor
>  Labels: patch
> Fix For: 1.4.7
>
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> The mainframe module (SQOOP-1272) currently supports partitioned data sets.
> I propose adding the following improvements:-
> - sequential data set support
> - GDG data set support
> I propose the following parameters to be added:-
> 1. --datasettype   (p = partitioned data set, s = sequential, g = GDG)
> Let me know if this should be all in one issue or separate issues



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SQOOP-3199) Sqoop 1.4.7 release preparation

[Attila Szabo (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272795#comment-16272795
 ] 

Attila Szabo commented on SQOOP-3199:
-

Hey [~BoglarkaEgyed],

Many thanks for your help again!

As we're generating the release notes from JIRA (and not manually editing it), 
I'm gonna update the status these JIRAs accordingly, and regenerate the release 
notes.

One idea for the future: I you gonna see any closed jira with incorrect 
resolution please do not hesitate to fix it ASAP, I'm gonna do the same (and do 
hope by time the whole community will do the same ;-) ). 
But many thanks once more for this finding, it's been very helpful, and for me 
it was unrecognized!

I'm gonna notify you as soon as it's done + the cherry picking for the Oracle 
test fix.

Cheers,
[~maugli]

> Sqoop 1.4.7 release preparation
> ---
>
> Key: SQOOP-3199
> URL: https://issues.apache.org/jira/browse/SQOOP-3199
> Project: Sqoop
>  Issue Type: Task
>Reporter: Attila Szabo
>Assignee: Attila Szabo
> Fix For: no-release
>
> Attachments: text.html
>
>
> Umbrella jira for 1.4.7 release.
> For reference, the release wikis are:
> https://cwiki.apache.org/confluence/display/SQOOP/How+to+Release
> https://cwiki.apache.org/confluence/display/SQOOP/How+to+Release+Sqoop2



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (SQOOP-3249) Add SQOOP-3248 related data into CHANGELOG.txt

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo resolved SQOOP-3249.
-
Resolution: Fixed

> Add SQOOP-3248 related data into CHANGELOG.txt
> --
>
> Key: SQOOP-3249
> URL: https://issues.apache.org/jira/browse/SQOOP-3249
> Project: Sqoop
>  Issue Type: Sub-task
>Reporter: Attila Szabo
>Assignee: Attila Szabo
> Fix For: no-release
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (SQOOP-3249) Add SQOOP-3248 related data into CHANGELOG.txt

[Attila Szabo (JIRA)]

Attila Szabo created SQOOP-3249:
---

 Summary: Add SQOOP-3248 related data into CHANGELOG.txt
 Key: SQOOP-3249
 URL: https://issues.apache.org/jira/browse/SQOOP-3249
 Project: Sqoop
  Issue Type: Sub-task
Reporter: Attila Szabo
Assignee: Attila Szabo
 Fix For: no-release






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (SQOOP-3248) Add missing Apache 2 license information to TestCompilationManager.java PostgresqlExternalTableImportTest.java

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo resolved SQOOP-3248.
-
Resolution: Fixed

> Add missing Apache 2 license information to TestCompilationManager.java 
> PostgresqlExternalTableImportTest.java
> --
>
> Key: SQOOP-3248
> URL: https://issues.apache.org/jira/browse/SQOOP-3248
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.4.7, 1.5.0
>Reporter: Attila Szabo
>Assignee: Attila Szabo
> Fix For: 1.4.7, 1.5.0
>
>
> It's vital that all files contain the proper Apache 2 license header, and 
> it's also blocking 1.4.7 release



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SQOOP-3248) Add missing Apache 2 license information to TestCompilationManager.java PostgresqlExternalTableImportTest.java

[Attila Szabo (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16227204#comment-16227204
 ] 

Attila Szabo commented on SQOOP-3248:
-

src/test/org/apache/sqoop/orm/TestCompilationManager.java
src/test/com/cloudera/sqoop/manager/PostgresqlExternalTableImportTest.java

affected

> Add missing Apache 2 license information to TestCompilationManager.java 
> PostgresqlExternalTableImportTest.java
> --
>
> Key: SQOOP-3248
> URL: https://issues.apache.org/jira/browse/SQOOP-3248
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.4.7, 1.5.0
>Reporter: Attila Szabo
>Assignee: Attila Szabo
> Fix For: 1.4.7, 1.5.0
>
>
> It's vital that all files contain the proper Apache 2 license header, and 
> it's also blocking 1.4.7 release



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (SQOOP-3248) Add missing Apache 2 license information to TestCompilationManager.java PostgresqlExternalTableImportTest.java

[Attila Szabo (JIRA)]

Attila Szabo created SQOOP-3248:
---

 Summary: Add missing Apache 2 license information to 
TestCompilationManager.java PostgresqlExternalTableImportTest.java
 Key: SQOOP-3248
 URL: https://issues.apache.org/jira/browse/SQOOP-3248
 Project: Sqoop
  Issue Type: Bug
Affects Versions: 1.4.7, 1.5.0
Reporter: Attila Szabo
Assignee: Attila Szabo
 Fix For: 1.4.7, 1.5.0


It's vital that all files contain the proper Apache 2 license header, and it's 
also blocking 1.4.7 release



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SQOOP-3247) Sqoop doesn't handle unsigned bigints at least with MySQL

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo updated SQOOP-3247:

Fix Version/s: (was: 1.4.7)
   1.5.0

> Sqoop doesn't handle unsigned bigints at least with MySQL
> -
>
> Key: SQOOP-3247
> URL: https://issues.apache.org/jira/browse/SQOOP-3247
> Project: Sqoop
>  Issue Type: Bug
>  Components: connectors/mysql
>Affects Versions: 1.4.6
>Reporter: Yulei Wang
>  Labels: patch
> Fix For: 1.5.0
>
> Attachments: SQOOP-3247.patch
>
>
> mysql> desc mysql2hdfs;
> +-+-+--+-+-+---+
> | Field   | Type| Null | Key | Default | Extra |
> +-+-+--+-+-+---+
> | id  | bigint(20) unsigned | YES  | | NULL|   |
> | address | varchar(20) | YES  | | NULL|   |
> +-+-+--+-+-+---+
> 2 rows in set (0.00 sec)
> mysql> select * from mysql2hdfs;
> +--+-+
> | id   | address |
> +--+-+
> | 18446744073709551615 | suzhou  |
> | 18446744073709551615 | suzhou  |
> +--+-+
> 2 rows in set (0.00 sec)
> Get's the following error
> 17/10/30 15:05:01 ERROR tool.ImportTool: Encountered IOException running 
> import job: java.io.IOException: 
> com.mysql.jdbc.exceptions.jdbc4.MySQLDataException: '18446744073709551615' in 
> column '1' is outside valid range for the datatype BIGINT.
>   at 
> org.apache.sqoop.mapreduce.db.DataDrivenDBInputFormat.getSplits(DataDrivenDBInputFormat.java:174)
>   at 
> org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:608)
>   at 
> org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:625)
>   at 
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:494)
>   at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1326)
>   at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1323)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:415)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1858)
>   at org.apache.hadoop.mapreduce.Job.submit(Job.java:1323)
>   at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1344)
>   at 
> org.apache.sqoop.mapreduce.ImportJobBase.doSubmitJob(ImportJobBase.java:196)
>   at 
> org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:169)
>   at 
> org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:266)
>   at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:673)
>   at 
> org.apache.sqoop.manager.MySQLManager.importTable(MySQLManager.java:118)
>   at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:497)
>   at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
>   at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>   at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
>   at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
>   at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
>   at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
> Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLDataException: 
> '18446744073709551615' in column '1' is outside valid range for the datatype 
> BIGINT.
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>   at com.mysql.jdbc.Util.handleNewInstance(Util.java:409)
>   at com.mysql.jdbc.Util.getInstance(Util.java:384)
>   at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1025)
>   at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:987)
>   at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:973)
>   at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:918)
>   at 
> com.mysql.jdbc.ResultSetImpl.throwRangeException(ResultSetImpl.java:7875)
>   at 
> com.mysql.jdbc.ResultSetImpl.parseLongAsDouble(ResultSetImpl.java:7092)
>   at com.mysql.jdbc.ResultSetImpl.getLong(ResultSetImpl.java:2977)
>   at com.mysql.jdbc.ResultSetImpl.getLong(ResultSetImpl.java:2942)
>   at 
> org.apache.sqoop.mapreduce.db.IntegerSplitter.split(IntegerSplitter.java:44)
>   at 
> 

[jira] [Updated] (SQOOP-3199) Sqoop 1.4.7 release preparation

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo updated SQOOP-3199:

Description: 
Umbrella jira for 1.4.7 release.
For reference, the release wikis are:
https://cwiki.apache.org/confluence/display/SQOOP/How+to+Release
https://cwiki.apache.org/confluence/display/SQOOP/How+to+Release+Sqoop2

  was:
Umbrella jira for 1.4.6 release.
For reference, the release wikis are:
https://cwiki.apache.org/confluence/display/SQOOP/How+to+Release
https://cwiki.apache.org/confluence/display/SQOOP/How+to+Release+Sqoop2


> Sqoop 1.4.7 release preparation
> ---
>
> Key: SQOOP-3199
> URL: https://issues.apache.org/jira/browse/SQOOP-3199
> Project: Sqoop
>  Issue Type: Task
>Reporter: Attila Szabo
>Assignee: Attila Szabo
> Fix For: no-release
>
> Attachments: text.html
>
>
> Umbrella jira for 1.4.7 release.
> For reference, the release wikis are:
> https://cwiki.apache.org/confluence/display/SQOOP/How+to+Release
> https://cwiki.apache.org/confluence/display/SQOOP/How+to+Release+Sqoop2



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SQOOP-3199) Sqoop 1.4.7 release preparation

[Attila Szabo (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16225043#comment-16225043
 ] 

Attila Szabo commented on SQOOP-3199:
-

Hey [ ~anna.szonyi],[~BoglarkaEgyed],

I've made some grooming around version 1.4.7 on the JIRA, regenerated the 
release notes, updated the CHANGELOG file, and pushed the changes to the apache 
git repository.

After a double check now it seems to be consistent to me. Could you please 
double check this on your side too, and let me know if you agree with me to 
build the first RC?

Could you please also tell me if you still have concerns around the version 
1.4.7?

Many thanks,
[~maugli]

> Sqoop 1.4.7 release preparation
> ---
>
> Key: SQOOP-3199
> URL: https://issues.apache.org/jira/browse/SQOOP-3199
> Project: Sqoop
>  Issue Type: Task
>Reporter: Attila Szabo
>Assignee: Attila Szabo
> Fix For: no-release
>
> Attachments: text.html
>
>
> Umbrella jira for 1.4.6 release.
> For reference, the release wikis are:
> https://cwiki.apache.org/confluence/display/SQOOP/How+to+Release
> https://cwiki.apache.org/confluence/display/SQOOP/How+to+Release+Sqoop2



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (SQOOP-3246) Update 1.4.7 change log entries + updated the related JIRA tasks (patch available + TODO tasks) by moving them to next version (1.5.0)

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo reassigned SQOOP-3246:
---

Assignee: Attila Szabo

> Update 1.4.7 change log entries + updated the related JIRA tasks (patch 
> available + TODO tasks) by moving them to next version (1.5.0)
> --
>
> Key: SQOOP-3246
> URL: https://issues.apache.org/jira/browse/SQOOP-3246
> Project: Sqoop
>  Issue Type: Sub-task
>Reporter: Attila Szabo
>Assignee: Attila Szabo
> Fix For: no-release
>
> Attachments: SQOOP-3246.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SQOOP-3246) Update 1.4.7 change log entries + updated the related JIRA tasks (patch available + TODO tasks) by moving them to next version (1.5.0)

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo updated SQOOP-3246:

Attachment: SQOOP-3246.patch

> Update 1.4.7 change log entries + updated the related JIRA tasks (patch 
> available + TODO tasks) by moving them to next version (1.5.0)
> --
>
> Key: SQOOP-3246
> URL: https://issues.apache.org/jira/browse/SQOOP-3246
> Project: Sqoop
>  Issue Type: Sub-task
>Reporter: Attila Szabo
> Fix For: no-release
>
> Attachments: SQOOP-3246.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (SQOOP-3246) Update 1.4.7 change log entries + updated the related JIRA tasks (patch available + TODO tasks) by moving them to next version (1.5.0)

[Attila Szabo (JIRA)]

Attila Szabo created SQOOP-3246:
---

 Summary: Update 1.4.7 change log entries + updated the related 
JIRA tasks (patch available + TODO tasks) by moving them to next version (1.5.0)
 Key: SQOOP-3246
 URL: https://issues.apache.org/jira/browse/SQOOP-3246
 Project: Sqoop
  Issue Type: Sub-task
Reporter: Attila Szabo






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SQOOP-3026) Document that Sqoop export with --hcatalog-table is not supported

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo updated SQOOP-3026:

Fix Version/s: (was: 1.4.7)
   1.5.0

> Document that Sqoop export with --hcatalog-table  is not supported
> -
>
> Key: SQOOP-3026
> URL: https://issues.apache.org/jira/browse/SQOOP-3026
> Project: Sqoop
>  Issue Type: Improvement
>Reporter: Anna Szonyi
>Priority: Minor
> Fix For: 1.5.0
>
>
> When using sqoop export with --hcatalog-table, it fails with a NPE, as Hive + 
> MapReduce does not support reading from or writing to views (the same works 
> with HiveCli).
> See:
> https://cwiki.apache.org/confluence/display/Hive/HCatalog+CLI#HCatalogCLI-Create/Drop/AlterView



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SQOOP-2644) Sqoop Export does not offer upserts from HDFS to Teradata DB

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-2644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo updated SQOOP-2644:

Fix Version/s: (was: 1.4.7)
   1.5.0

> Sqoop Export does not offer upserts from HDFS to Teradata DB
> 
>
> Key: SQOOP-2644
> URL: https://issues.apache.org/jira/browse/SQOOP-2644
> Project: Sqoop
>  Issue Type: Sub-task
>Affects Versions: 1.4.6
>Reporter: Kopal Niranjan
>Assignee: Kopal Niranjan
>Priority: Minor
> Fix For: 1.5.0
>
>
> Sqoop export fails when we perform upserts from HDFS to Teradta
> sqoop export --driver com.teradata.jdbc.TeraDriver --connect 
> jdbc:teradata://10.127.54.34/DATABASE=user_db,TMODE=ANSI,LOGMECH=LDAP 
> --username abc -P --table upsert_tbl --export-dir /user/root/exportRec/ 
> --update-key id --update-mode allowinsert
> Above command fails with the following error:
> 15/10/16 02:51:19 ERROR tool.ExportTool: Error during export: Mixed 
> update/insert is not supported against the target database yet



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SQOOP-2534) --password-file option doesn't work Teradata jdbc driver

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-2534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo updated SQOOP-2534:

Fix Version/s: (was: 1.4.7)
   1.5.0

> --password-file option doesn't work Teradata jdbc driver
> 
>
> Key: SQOOP-2534
> URL: https://issues.apache.org/jira/browse/SQOOP-2534
> Project: Sqoop
>  Issue Type: New Feature
>Affects Versions: 1.4.5
>Reporter: Rakesh Sharma
>Priority: Minor
> Fix For: 1.5.0
>
>
> When we use --password-file option while importing with Teradata jdbc driver 
> , we get an exception saying invalid username/password:
>  sqoop import --driver com.teradata.jdbc.TeraDriver --connect 
> jdbc:teradata://10.187.82.34/LOGMECH=LDAP,tmode=ANSI,charset=UTF8 --username 
> raksharma --password-file /user/root/td_dev_pswd --table 
> L_PLTScratchpad.test_employee --split-by id --create-hcatalog-table --verbose 
> --hcatalog-table test_employee_tera
> Exception thrown :
> 15/08/27 08:55:01 ERROR manager.SqlManager: Error executing statement: 
> java.sql.SQLException: [Teradata Database] [TeraJDBC 15.10.00.05] [Error 
> 8017] [SQLState 28000] The UserId, Password or Account is invalid.
> java.sql.SQLException: [Teradata Database] [TeraJDBC 15.10.00.05] [Error 
> 8017] [SQLState 28000] The UserId, Password or Account is invalid.
>   at 
> com.teradata.jdbc.jdbc_4.util.ErrorFactory.makeDatabaseSQLException(ErrorFactory.java:301)
>   at 
> com.teradata.jdbc.jdbc.GenericLogonController.run(GenericLogonController.java:502)
>   at com.teradata.jdbc.jdbc_4.TDSession.(TDSession.java:208)
>   at 
> com.teradata.jdbc.jdk6.JDK6_SQL_Connection.(JDK6_SQL_Connection.java:35)
>   at 
> com.teradata.jdbc.jdk6.JDK6ConnectionFactory.constructSQLConnection(JDK6ConnectionFactory.java:25)
>   at 
> com.teradata.jdbc.jdbc.ConnectionFactory.createConnection(ConnectionFactory.java:179)
>   at 
> com.teradata.jdbc.jdbc.ConnectionFactory.createConnection(ConnectionFactory.java:169)
>   at com.teradata.jdbc.TeraDriver.doConnect(TeraDriver.java:232)
>   at com.teradata.jdbc.TeraDriver.connect(TeraDriver.java:158)
>   at java.sql.DriverManager.getConnection(DriverManager.java:571)
>   at java.sql.DriverManager.getConnection(DriverManager.java:215)
>   at 
> org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:877)
>   at 
> org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
>   at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:736)
>   at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:759)
>   at 
> org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:269)
>   at 
> org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:240)
>   at 
> org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:226)
>   at 
> org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:295)
>   at 
> org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1833)
>   at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1645)
>   at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:96)
>   at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:491)
>   at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:641)
>   at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>   at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
>   at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
>   at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
>   at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
> 15/08/27 08:55:01 ERROR tool.ImportTool: Encountered IOException running 
> import job: java.io.IOException: No columns to generate for ClassWriter
>   at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1651)
>   at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:96)
>   at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:491)
>   at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:641)
>   at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>   at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
>   at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
>   at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
>   at org.apache.sqoop.Sqoop.main(Sqoop.java:236)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SQOOP-1735) Sqoop job fails (but not 'sqoop import') if --create-hive-table is set and the hive table already exists

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-1735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo updated SQOOP-1735:

Fix Version/s: (was: 1.4.7)
   1.5.0

> Sqoop job fails (but not 'sqoop import') if --create-hive-table is set and 
> the hive table already exists
> 
>
> Key: SQOOP-1735
> URL: https://issues.apache.org/jira/browse/SQOOP-1735
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.4.5
> Environment: CentOS 6.6
> Sqoop v. 1.4.5
>Reporter: Joshua Clausen
>Priority: Minor
>  Labels: fail, hive, job, table
> Fix For: 1.5.0
>
>
> If you run the same import config from the command line via "sqoop import" 
> the import will succeed.  Jacek mentioned in a reply to someone that 
> --create-hive-table was intended to create the table if it doesn't already 
> exist, so I'd expect the behavior within a job should match that.
> The cause is because by default the PROPNAME "hive.fail.table.exists" inside 
> the SQOOP_SESSIONS hsqldb table is set to "true" and there's no way to change 
> that (besides running an "update" statement inside a hsqldb client), e.g. 
> "update sqoop_sessions set propval = false where propname = 
> 'hive.fail.table.exists';"
> I'd suggest the best fix would be to add an additional parameter for "sqoop 
> import" that allows the user to explicitly set the behavior if the hive table 
> already exists.  Something like "--hive-fail-table-exists [true/false]".  
> That, or just have the propname = hive.fail.table.exists by default be set to 
> 'false' in the sqoop_sessions table.
> Note, when I submitted this issue I had to select a "Fix Version/s" in order 
> to submit.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SQOOP-3067) Add an cmd line option to support split-by feature for database functions/expressions

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo updated SQOOP-3067:

Fix Version/s: (was: 1.4.7)
   1.5.0

> Add an cmd line option to support split-by feature for database 
> functions/expressions
> -
>
> Key: SQOOP-3067
> URL: https://issues.apache.org/jira/browse/SQOOP-3067
> Project: Sqoop
>  Issue Type: Improvement
>Reporter: Attila Szabo
>Assignee: Anna Szonyi
> Fix For: 1.5.0
>
>
> Due to the fact, if the [SQOOP-2737] escaping feature is enabled, the 
> split-by column name is also escaped and quoted properly, we do need to have 
> an option for split-by which would accept DB related function/expressions, 
> but those expressions should not be quoted/escaped.
> This feature is dependent from the [SQOOP-3066], so first that one has to be 
> implemented and merged to the source.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SQOOP-3052) Introduce Maven/Gradle/etc. based build for Sqoop to make it more developer friendly / open

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo updated SQOOP-3052:

Fix Version/s: (was: 1.4.7)
   1.5.0

> Introduce Maven/Gradle/etc. based build for Sqoop to make it more developer 
> friendly / open
> ---
>
> Key: SQOOP-3052
> URL: https://issues.apache.org/jira/browse/SQOOP-3052
> Project: Sqoop
>  Issue Type: Improvement
>Reporter: Attila Szabo
>Assignee: Anna Szonyi
> Fix For: 1.5.0
>
>
> The current trunk version can only be build with Ant/Ivy combination, which 
> has some painful limitations (resolve is slow / needs to be tweaked to use 
> only caches, the current profile / variable based settings are not working in 
> IDEs out of the box, the current solution does not download the related 
> sources, etc.)
> It would be nice to provide a solution, which would give the possibility for 
> the developers to choose between the nowadays well used build infrsturctures 
> (e.g. Maven, Gradle, etc.). For this solution it would be also essential to 
> keep the different build files (if there is more then one) synchronized 
> easily, and the configuration wouldn't diverege by time. Test execution has 
> to be solved also, and should cover all the available test cases.
> In this scenario:
> If we can provide one good working solution is much better, then provide 
> three different ones which become out of sync easily. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SQOOP-2643) Incremental imports fail in Sqoop when run using Teradata JDBC driver

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-2643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo updated SQOOP-2643:

Fix Version/s: (was: 1.4.7)
   1.5.0

> Incremental imports fail in Sqoop when run using Teradata JDBC driver
> -
>
> Key: SQOOP-2643
> URL: https://issues.apache.org/jira/browse/SQOOP-2643
> Project: Sqoop
>  Issue Type: Sub-task
>Affects Versions: 1.4.6
>Reporter: Kopal Niranjan
>Assignee: Kopal Niranjan
>Priority: Minor
> Fix For: 1.5.0
>
>
> When sqoop incremental import is run over Teradata DB,
> sqoop import --driver com.teradata.jdbc.TeraDriver --connect 
> jdbc:teradata://10.126.62.46/DATABASE=SampleDb,TMODE=ANSI,LOGMECH=LDAP 
> --username abc -P --table test1 --incremental lastmodified --check-column DAT 
> --last-value '2014-10-22 07:00:00' --target-dir 
> /user/$USER/hive_data_dynapart/test1dir
> then the following error occurs:
> 15/09/24 03:57:10 ERROR manager.SqlManager: SQL exception accessing current 
> timestamp: com.teradata.jdbc.jdbc_4.util.JDBCException: [Teradata Database] 
> [TeraJDBC 14.10.00.26] [Error 3706] [SQLState 42000] Syntax error: expected 
> something between '(' and ')'.
> com.teradata.jdbc.jdbc_4.util.JDBCException: [Teradata Database] [TeraJDBC 
> 14.10.00.26] [Error 3706] [SQLState 42000] Syntax error: expected something 
> between '(' and ')'.
> at 
> com.teradata.jdbc.jdbc_4.util.ErrorFactory.makeDatabaseSQLException(ErrorFactory.java:307)
> at 
> com.teradata.jdbc.jdbc_4.statemachine.ReceiveInitSubState.action(ReceiveInitSubState.java:109)
> at 
> com.teradata.jdbc.jdbc_4.statemachine.StatementReceiveState.subStateMachine(StatementReceiveState.java:314)
> at 
> com.teradata.jdbc.jdbc_4.statemachine.StatementReceiveState.action(StatementReceiveState.java:202)
> at 
> com.teradata.jdbc.jdbc_4.statemachine.StatementController.runBody(StatementController.java:123)
> at 
> com.teradata.jdbc.jdbc_4.statemachine.StatementController.run(StatementController.java:114)
> at com.teradata.jdbc.jdbc_4.TDStatement.executeStatement(TDStatement.java:384)
> at com.teradata.jdbc.jdbc_4.TDStatement.executeStatement(TDStatement.java:326)
> at 
> com.teradata.jdbc.jdbc_4.TDStatement.doNonPrepExecuteQuery(TDStatement.java:314)
> at com.teradata.jdbc.jdbc_4.TDStatement.executeQuery(TDStatement.java:1091)
> at 
> org.apache.sqoop.manager.SqlManager.getCurrentDbTimestamp(SqlManager.java:987)
> at 
> org.apache.sqoop.tool.ImportTool.initIncrementalConstraints(ImportTool.java:328)
> at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:488)
> at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
> at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
> 15/09/24 03:57:10 ERROR tool.ImportTool: Encountered IOException running 
> import job: java.io.IOException: Could not get current time from database
> at 
> org.apache.sqoop.tool.ImportTool.initIncrementalConstraints(ImportTool.java:330)
> at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:488)
> at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
> at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:236)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SQOOP-3131) Docuemtn support for DB2 XML data type when importing to hdfs

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo updated SQOOP-3131:

Fix Version/s: (was: 1.4.7)
   1.5.0

> Docuemtn support for DB2 XML data type when importing to hdfs
> -
>
> Key: SQOOP-3131
> URL: https://issues.apache.org/jira/browse/SQOOP-3131
> Project: Sqoop
>  Issue Type: Improvement
>  Components: docs
>Affects Versions: 1.4.5
> Environment: RedHat6.4 + Sqoop 1.4.5 + hadoop 2.4.1
>Reporter: Ying Cao
>Priority: Minor
>  Labels: feature
> Fix For: 1.5.0
>
>
> After SQOOP-1904, Sqoop should supoort for DB2 XML data type importing.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SQOOP-3085) Add support for client side (JVM) timezone settings

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo updated SQOOP-3085:

Fix Version/s: (was: 1.4.7)
   1.5.0

> Add support for client side (JVM) timezone settings
> ---
>
> Key: SQOOP-3085
> URL: https://issues.apache.org/jira/browse/SQOOP-3085
> Project: Sqoop
>  Issue Type: Improvement
>Affects Versions: 1.4.6
>Reporter: Attila Szabo
>Assignee: Attila Szabo
> Fix For: 1.5.0
>
>
> Currently in OracleManager and OraOopManager there is the capability to set 
> time zone to support "TIMESTAMP WITH LOCAL TIME ZONE" data types (with 
> "oracle.sessionTimezone" Hadoop option).
> As the current user expectation (+3rd party test case) is to get back the 
> date in the given local time zone, both OracleManager and OraOopManager sets 
> the default time zone of the underlying JVM to the same time zone, what is 
> passed to the Oracle connection. 
> This is a very straightforward and consistent expectation, however there's a 
> need to set the time zone settings in Sqoop through command line arguments.
> There are two arguments for that:
> A - passing the -Duser.timezone to the JVM is not always that straightforward 
> (as sqoop is executed by some shell/batch script, and user would have to 
> modify that file, or modify the hadoop/sqoop site xml files, or pass it with 
> env)
> B - if the user wants to set the client side time zone, it should be explicit 
> on the CMD line, and also warning+validation should be returned if the client 
> side+db side time zones are different (by intention).
> The goal of this task is to fill in this gap, and add the option+the required 
> validation too.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SQOOP-1347) High performance oracle connector should depend on --direct flag only to disable/enable feature

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-1347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo updated SQOOP-1347:

Fix Version/s: (was: 1.4.7)
   1.5.0

> High performance oracle connector should depend on --direct flag only to 
> disable/enable feature
> ---
>
> Key: SQOOP-1347
> URL: https://issues.apache.org/jira/browse/SQOOP-1347
> Project: Sqoop
>  Issue Type: Bug
>  Components: connectors/oracle
>Affects Versions: 1.4.5
>Reporter: Venkat Ranganathan
>Priority: Minor
> Fix For: 1.5.0
>
>
> SQOOP-1287 introduces the Oracle High Performance Connector as part of Sqoop. 
>   Right now, two flags are used to enable this feature.   Sqoop command line 
> option --direct and a connector specific config option.  As this connector is 
> getting integrated with Sqoop,  we should make it only depend on --direct 
> option alone to enable/disable the feature.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SQOOP-2980) Export to DB2 z/OS fails unless --batch mode is used

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-2980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo updated SQOOP-2980:

Fix Version/s: (was: 1.4.7)
   1.5.0

> Export to DB2 z/OS fails unless --batch mode is used
> 
>
> Key: SQOOP-2980
> URL: https://issues.apache.org/jira/browse/SQOOP-2980
> Project: Sqoop
>  Issue Type: Bug
>  Components: connectors
>Affects Versions: 1.4.6
>Reporter: Senthil Ganesh
> Fix For: 1.5.0
>
>
> Export fails for DB2 Z/OS without --batch option with the below error
> java.lang.Exception: java.io.IOException: com.ibm.db2.jcc.am.SqlException: 
> [jcc][t4][10251][10308][4.19.49] java.sql.Connection.close() requested while 
> a transaction is in progress on the connection.
> May 25, 2016 8:14:37 PM,eMSG_TRACE,TASK_140119021643520-MAPPING,
> APPSDK_Msg_1762,[TRACE] The transaction remains active, and the 
> connection cannot be closed. ERRORCODE=-4471, SQLSTATE=null
> at 
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
> at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
> May 25, 2016 8:14:37 PM,eMSG_TRACE,TASK_140119021643520-MAPPING,
> APPSDK_Msg_1762,[TRACE] Caused by: java.io.IOException: 
> com.ibm.db2.jcc.am.SqlException: [jcc][t4][10251][10308][4.19.49] 
> java.sql.Connection.close() requested while a transaction is in progress on 
> the connection.
> May 25, 2016 8:14:37 PM,eMSG_TRACE,TASK_140119021643520-MAPPING,
> APPSDK_Msg_1762,[TRACE] The transaction remains active, and the 
> connection cannot be closed. ERRORCODE=-4471, SQLSTATE=null
> at 
> org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:211)
> at 
> org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:670)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at 
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SQOOP-2946) Netezza export fails when BOOLEAN column is mapped with INTEGER

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-2946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo updated SQOOP-2946:

Fix Version/s: (was: 1.4.7)
   1.5.0

> Netezza export fails when BOOLEAN column is mapped with INTEGER
> ---
>
> Key: SQOOP-2946
> URL: https://issues.apache.org/jira/browse/SQOOP-2946
> Project: Sqoop
>  Issue Type: Bug
>  Components: connectors
>Affects Versions: 1.4.6
>Reporter: Senthil Ganesh
> Fix For: 1.5.0
>
>
> When we try to export to Netezza for BOOLEAN data type column using 
> --map-java-column INTERGER, it fails with below exception (same work for 
> import case):
> org.netezza.error.NzSQLException: ERROR: Attribute 'COL_BOOLEAN' is of type 
> 'BOOL' but expression is of type 'INT4'



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SQOOP-2979) Oracle direct mode do not allow FLOAT data type (java.lang.ClassCastException: java.lang.Double cannot be cast to java.math.BigDecimal)

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-2979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo updated SQOOP-2979:

Fix Version/s: (was: 1.4.7)
   1.5.0

> Oracle direct mode do not allow FLOAT data type 
> (java.lang.ClassCastException: java.lang.Double cannot be cast to 
> java.math.BigDecimal)
> ---
>
> Key: SQOOP-2979
> URL: https://issues.apache.org/jira/browse/SQOOP-2979
> Project: Sqoop
>  Issue Type: Bug
>  Components: connectors
>Affects Versions: 1.4.6
>Reporter: Senthil Ganesh
> Fix For: 1.5.0
>
>
> Using --direct option with Oracle table having float data type sees below 
> exception:
> Error: java.io.IOException: java.sql.SQLException: 
> java.lang.ClassCastException: java.lang.Double cannot be cast to 
> java.math.BigDecimal



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SQOOP-2978) Netezza import/export fails when TIME column is mapped with TIMESTAMP

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-2978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo updated SQOOP-2978:

Fix Version/s: (was: 1.4.7)
   1.5.0

> Netezza import/export fails when TIME column is mapped with TIMESTAMP
> -
>
> Key: SQOOP-2978
> URL: https://issues.apache.org/jira/browse/SQOOP-2978
> Project: Sqoop
>  Issue Type: Bug
>  Components: connectors
>Affects Versions: 1.4.6
>Reporter: Senthil Ganesh
> Fix For: 1.5.0
>
>
> We have scenarios where we need to map TIME data type column of Netezza with 
> TIMESTAMP from Sqoop. However, that fails with java.lang.ClassCastException: 
> java.sql.Time cannot be cast to java.sql.Timestamp error. Same approach works 
> with other databases.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SQOOP-2947) Oracle direct mode do not allow --jar-file to have fewer columns to export the data

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-2947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo updated SQOOP-2947:

Fix Version/s: (was: 1.4.7)
   1.5.0

> Oracle direct mode do not allow --jar-file to have fewer columns to export 
> the data
> ---
>
> Key: SQOOP-2947
> URL: https://issues.apache.org/jira/browse/SQOOP-2947
> Project: Sqoop
>  Issue Type: Bug
>  Components: connectors/oracle
>Affects Versions: 1.4.6
>Reporter: Senthil Ganesh
> Fix For: 1.5.0
>
>
> Oracle direct mode does not allow --jar-file option, in which case it expects 
> all columns of the target table be present in the export file. This is an 
> issue as my source has fewer column which I have imported to a sequence file 
> and need to use that file for exporting to another Oracle table which has 
> more columns.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SQOOP-3029) Add an option for uppercase/lowercase column name mapping between HCatalog and RDBMS cloumn name list

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo updated SQOOP-3029:

Fix Version/s: (was: 1.4.7)
   1.5.0

> Add an option for uppercase/lowercase column name mapping between HCatalog 
> and RDBMS cloumn name list
> -
>
> Key: SQOOP-3029
> URL: https://issues.apache.org/jira/browse/SQOOP-3029
> Project: Sqoop
>  Issue Type: Improvement
>Reporter: Attila Szabo
> Fix For: 1.5.0
>
>
> The current implementation of HCatalog related export works out of the box 
> with RDBMS systems (e.g. Oracle) if the column names match case sensitive, 
> although in several cases the column names are received as an UPPER_CASE list 
> from the RDBMS, and thus the export fails, within the ClassWriter.
> Although the users have the possibility to define the name of the columns 
> explicit with the --columns option, in case of 100+ columns it's not 
> practical.
> It would be great to have an option which could do the matching in lower and 
> UPPER case as well. The start point for this implementation should start from 
> SqoopHCatUtilities#configureHCat method with respect to ensure HCatalog api 
> is able to handle the different cased column names.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SQOOP-2943) Make sqoop able to import to Parquet file format in case of HDFS encryption zones are turned on

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-2943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo updated SQOOP-2943:

Fix Version/s: (was: 1.4.7)
   1.5.0

> Make sqoop able to import to Parquet file format in case of HDFS encryption 
> zones are turned on
> ---
>
> Key: SQOOP-2943
> URL: https://issues.apache.org/jira/browse/SQOOP-2943
> Project: Sqoop
>  Issue Type: Improvement
>Affects Versions: 1.4.7
>Reporter: Attila Szabo
> Fix For: 1.5.0
>
>
> If HDFS encryption zones are turned on, and the user tries to import into 
> Parquet format, where the target location is in a very different encryption 
> zone, than the zone of the /tmp/ location (typical use case for that 
> encrypted hive warehouse directory), even the mapper jobs are executed 
> successfully, and the partial results stored on the temp storage correctly, 
> the MergeOutputMapper class of Kite SDK dies with an HDFS related exception 
> ("can't be moved into an encryption zone").
> The problem does not appear in case of clear text output formats.
> Please make Sqoop able to solve this scenario as well!



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SQOOP-2945) Oracle CLOB mapped to String is unable to import the data (SQLException in nextKeyValue)

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-2945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo updated SQOOP-2945:

Fix Version/s: (was: 1.4.7)
   1.5.0

> Oracle CLOB mapped to String is unable to import the data (SQLException in 
> nextKeyValue)
> 
>
> Key: SQOOP-2945
> URL: https://issues.apache.org/jira/browse/SQOOP-2945
> Project: Sqoop
>  Issue Type: Bug
>  Components: connectors/oracle
>Affects Versions: 1.4.6
>Reporter: Senthil Ganesh
> Fix For: 1.5.0
>
>
> If we map any CLOB column from Oracle to String with "--map-column-java 
> CLOB_COL=String" during Sqoop import, we are hitting below exception:
> ...
> map 0% reduce 0%
> map 50% reduce 0%
> Task Id : attempt_1462336475649_1097_m_00_0, Status : FAILED
> Error: java.io.IOException: SQLException in nextKeyValue
>   at 
> org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:277)
>   at 
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:556)
>   at 
> org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
>   at 
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
>   at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>   at 
> org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
>   at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
>   at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
>   at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> ...
> As we like to store the result in Hive (which do not support CLOB), we need 
> to map this column to String.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SQOOP-2944) AURORA direct mode to support Sequence file format

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-2944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo updated SQOOP-2944:

Fix Version/s: (was: 1.4.7)
   1.5.0

> AURORA direct mode to support Sequence file format
> --
>
> Key: SQOOP-2944
> URL: https://issues.apache.org/jira/browse/SQOOP-2944
> Project: Sqoop
>  Issue Type: Improvement
>  Components: connectors
>Affects Versions: 1.4.6
>Reporter: Senthil Ganesh
> Fix For: 1.5.0
>
>
> Currently --direct mode option for Aurora do not support sequence file 
> format. As our consumption of import and export is based on sequence file, we 
> are unable to leverage --direct option with Aurora.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SQOOP-1907) export support for --staging-table against db2

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-1907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo updated SQOOP-1907:

Fix Version/s: (was: 1.4.7)
   1.5.0

> export support for --staging-table against db2
> --
>
> Key: SQOOP-1907
> URL: https://issues.apache.org/jira/browse/SQOOP-1907
> Project: Sqoop
>  Issue Type: Improvement
>  Components: connectors
>Affects Versions: 1.4.5
> Environment: RedHat6.4 + Sqoop1.4.5 + Hadoop2.4.1 + Hive0.13.1 + DB2 
> 10.5
>Reporter: xieshiju
>Assignee: xieshiju
>  Labels: features
> Fix For: 1.5.0
>
> Attachments: SQOOP-1907.patch
>
>   Original Estimate: 504h
>  Remaining Estimate: 504h
>
>  export has an option --staging-table which allows the data to be exported 
> into an interim staging table and then if successful it is moved into the 
> final table. This prevents partial data loaded since each map task will 
> commit it work separately instead of in one transaction. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SQOOP-1933) CryptoFileLoader does not work for saved jobs

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo updated SQOOP-1933:

Fix Version/s: (was: 1.4.7)
   1.5.0

> CryptoFileLoader does not work for saved jobs
> -
>
> Key: SQOOP-1933
> URL: https://issues.apache.org/jira/browse/SQOOP-1933
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.4.5
> Environment: Red Hat Enterprise Linux Server release 6.3 (Santiago)
>Reporter: Karthik S
> Fix For: 1.5.0
>
> Attachments: fixpatch_v1.patch
>
>
> I am able to use the CryptoFile Loader to read encrypted passwords from a 
> simple import command. However, the same command does not work from saved jobs
> The below command works. There are no problems with it. It successfully 
> decrypts the password and imports into hdfs
> sqoop import 
> -Dorg.apache.sqoop.credentials.loader.class=org.apache.sqoop.util.password.CryptoFileLoader
>  -Dorg.apache.sqoop.credentials.loader.crypto.passphrase=sqooppass --driver 
> com.mysql.jdbc.Driver --connect jdbc:mysql://00.00.000.000:/test 
> --username user --password-file /user/user1/temp.txt  --table emp 
> --target-dir /user/user1/karthik/e129 --split-by emp_id
> However, When i try using saved jobs, it does not work.
> sqoop job 
> -Dorg.apache.sqoop.credentials.loader.class=org.apache.sqoop.util.password.CryptoFileLoader
>  -Dorg.apache.sqoop.credentials.loader.crypto.passphrase=sqooppass --create 
> job9 -- import --driver com.mysql.jdbc.Driver --connect 
> jdbc:mysql://00.00.000.000:3306/test --username user --password-file 
> /user/user1/temp.txt  --table emp --target-dir /user/user1/karthik/e130 
> --split-by emp_id
> sqoop job -exec job9 -- --target-dir /user/user1/emp_103
> It gives error 
> 14/12/22 15:39:48 ERROR sqoop.Sqoop: Got exception running Sqoop: 
> java.lang.RuntimeException: Unable to fetch password from file.
> java.lang.RuntimeException: Unable to fetch password from file.
> at 
> org.apache.sqoop.SqoopOptions.loadPasswordProperty(SqoopOptions.java:667)
> at org.apache.sqoop.SqoopOptions.loadProperties(SqoopOptions.java:617)
> at 
> org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage.read(HsqldbJobStorage.java:299)
> at org.apache.sqoop.tool.JobTool.execJob(JobTool.java:198)
> at org.apache.sqoop.tool.JobTool.run(JobTool.java:283)
> at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
> Caused by: java.io.IOException: Can't decrypt the password
> at 
> org.apache.sqoop.util.password.CryptoFileLoader.loadPassword(CryptoFileLoader.java:151)
> at 
> org.apache.sqoop.util.CredentialsUtil.fetchPasswordFromLoader(CredentialsUtil.java:81)
> at 
> org.apache.sqoop.SqoopOptions.loadPasswordProperty(SqoopOptions.java:664)
> ... 10 more
> Caused by: javax.crypto.BadPaddingException: Given final block not properly 
> padded
> at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:811)
> at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:676)
> at com.sun.crypto.provider.AESCipher.engineDoFinal(AESCipher.java:313)
> at javax.crypto.Cipher.doFinal(Cipher.java:2087)
> at 
> org.apache.sqoop.util.password.CryptoFileLoader.loadPassword(CryptoFileLoader.java:149)
> ... 12 more



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SQOOP-1932) fix authorization failuare when the --hive-table option contaion a database name for import

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-1932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo updated SQOOP-1932:

Fix Version/s: (was: 1.4.7)
   1.5.0

> fix authorization failuare when the --hive-table option contaion a database 
> name for import
> ---
>
> Key: SQOOP-1932
> URL: https://issues.apache.org/jira/browse/SQOOP-1932
> Project: Sqoop
>  Issue Type: Bug
>  Components: hive-integration
>Affects Versions: 1.4.5
> Environment: RedHat6.4 + Sqoop1.4.5 + Hive0.13 + DB2 10.5
>Reporter: xieshiju
>Assignee: xieshiju
> Fix For: 1.5.0
>
> Attachments: SQOOP-1932.patch, SQOOP-1932.patch
>
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> I create a new user user123 and a hive database DB1 in the hive using this 
> user user, then execute command grant all on database DB1 to user user123. 
> Following I ran a sqoop import command and want to import data from db2 table 
> to hive table:
> ./sqoop import --connect 'jdbc:db2://:5/XIEDB' --username  
> --password  --as-textfile --hive-import --hive-table DB1.hv_test --table 
> DB3BOOK --split-by ID --num-mappers 1  
> I got below error:
> 14/12/19 23:00:09 INFO hive.HiveImport: Authorization failed:No privilege 
> 'Create' found for outputs { database:default}. Use show grant to get more 
> details.
> 14/12/19 23:00:09 ERROR tool.ImportTool: Encountered IOException running 
> import job: java.io.IOException: Hive exited with status 147
>   at 
> org.apache.sqoop.hive.HiveImport.executeExternalHiveScript(HiveImport.java:392)
>   at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:341)
>   at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:245)
>   at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:511)
>   at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:601)
>   at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>   at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
>   at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
>   at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
>   at org.apache.sqoop.Sqoop.main(Sqoop.java:236)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SQOOP-2346) compression isn't honored in incremental imports

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo updated SQOOP-2346:

Fix Version/s: (was: 1.4.7)
   1.5.0

> compression isn't honored in incremental imports
> 
>
> Key: SQOOP-2346
> URL: https://issues.apache.org/jira/browse/SQOOP-2346
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.4.5
>Reporter: Abraham Elmahrek
> Fix For: 1.5.0
>
>
> See 
> http://mail-archives.apache.org/mod_mbox/sqoop-user/201505.mbox/%3C57703C6A-7F93-4067-8029-060E4D212096%40paytronix.com%3E
>  for more details.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SQOOP-2328) Sqoop import does not recognize Primary Key of a IBM DB2 table

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-2328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo updated SQOOP-2328:

Fix Version/s: (was: 1.4.7)
   1.5.0

> Sqoop import does not recognize Primary Key of a IBM DB2 table
> --
>
> Key: SQOOP-2328
> URL: https://issues.apache.org/jira/browse/SQOOP-2328
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.4.5
> Environment: IBM DB2 9,.7V
>Reporter: Atul Gupta
>Assignee: Shashank
> Fix For: 1.5.0
>
> Attachments: SQOOP-2328_0.patch
>
>
> Currently Sqoop import query does not recognize the PK of IBM DB2 table.
> When any sqoop query runs for DB2 table, it is not able to recognize the 
> primary key(PK) of that table, which is used as --split-by column implicitly. 
> To run the Sqoop query it is mandatory to give --split-by  
> explicitly. 
> Query Given below:
> {code}
> sqoop import -Dmapred.job.queue.name=edwdev --connect 
> jdbc:db2://dle-db2edw01.dl.karmalab.net:50001/EXPDEV01 --username='' 
> -P --table edwdev.mytable --hive-import --hive-table platdev.mytable_dynapart 
> --hive-external-table --target-dir /user/lbansal/hive_data_dynapart/testdir 
> --hive-dynamic-partition --hive-partition-key DATE 
> --hive-partition-key-format aa- --incremental lastmodified --check-column 
> DATE --last-value '2014-10-22' --append
> gives the following error message:
> 14/11/21 02:17:32 INFO manager.SqlManager: Executing SQL statement: SELECT 
> t.* FROM edwdev.mytable AS t WHERE 1=0
> 14/11/21 02:17:33 INFO tool.ImportTool: Incremental import based on column 
> DATE
> 14/11/21 02:17:33 INFO tool.ImportTool: Lower bound value: '2014-10-22'
> 14/11/21 02:17:33 INFO tool.ImportTool: Upper bound value: '2014-11-21 
> 02:17:33.680951'
> 14/11/21 02:17:34 INFO hive.metastore: Trying to connect to metastore with 
> URI thrift://cheledwhdd004.karmalab.net:9083
> 14/11/21 02:17:34 INFO hive.metastore: Connected to metastore.
> 14/11/21 02:17:34 WARN tool.HiveUtil: platdev.mytable_dynapart table not found
> 14/11/21 02:17:34 WARN tool.HiveUtil: platdev.mytable_dynapart table not found
> 14/11/21 02:17:34 ERROR tool.ImportTool: Error during import: No primary key 
> could be found for table edwdev.mytable. Please specify one with --split-by 
> or perform a sequential import with '-m 1'.
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SQOOP-2438) Use Class.cast when creating HiveConf object in ParquetJob

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-2438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo updated SQOOP-2438:

Fix Version/s: (was: 1.4.7)
   1.5.0

> Use Class.cast when creating HiveConf object in ParquetJob
> --
>
> Key: SQOOP-2438
> URL: https://issues.apache.org/jira/browse/SQOOP-2438
> Project: Sqoop
>  Issue Type: Bug
>  Components: hive-integration
>Affects Versions: 1.4.7
>Reporter: Abraham Elmahrek
>  Labels: newbie
> Fix For: 1.5.0
>
>
> To create an instance of {{HiveMetaStoreClient}}, Sqoop needs to create an 
> instance of the HiveConf class. That class is difficult to pass around, so we 
> have specific code in the {{ParquetJob.addHiveDelegationToken}} method to 
> create an instance of that class specifically. Using {{Class.cast(...)}}, 
> we should be able to achieve the same result, with less code.
> {code}
> // Need to use reflection since there's no compile time dependency on the 
> client libs.
> Class HiveConfClass;
> Class HiveMetaStoreClientClass;
> try {
>   HiveMetaStoreClientClass = Class.forName(HIVE_METASTORE_CLIENT_CLASS);
> } catch (ClassNotFoundException ex) {
>   LOG.error("Could not load " + HIVE_METASTORE_CLIENT_CLASS
>   + " when adding hive delegation token. "
>   + "Make sure HIVE_CONF_DIR is set correctly.", ex);
>   throw new RuntimeException("Couldn't fetch delegation token.", ex);
> }
> try {
>   HiveConfClass = Class.forName(HiveConfig.HIVE_CONF_CLASS);
> } catch (ClassNotFoundException ex) {
>   LOG.error("Could not load " + HiveConfig.HIVE_CONF_CLASS
>   + " when adding hive delegation token."
>   + " Make sure HIVE_CONF_DIR is set correctly.", ex);
>   throw new RuntimeException("Couldn't fetch delegation token.", ex);
> }
> try {
>   Object client = 
> HiveMetaStoreClientClass.getConstructor(HiveConfClass).newInstance(
>   HiveConfClass.getConstructor(Configuration.class, 
> Class.class).newInstance(conf, Configuration.class)
>   );
>   // getDelegationToken(String kerberosPrincial)
>   Method getDelegationTokenMethod = 
> HiveMetaStoreClientClass.getMethod("getDelegationToken", String.class);
>   Object tokenStringForm = getDelegationTokenMethod.invoke(client, 
> UserGroupInformation.getLoginUser().getShortUserName());
>   // Load token
>   Token metastoreToken = new 
> Token();
>   metastoreToken.decodeFromUrlString(tokenStringForm.toString());
>   conf.getCredentials().addToken(new Text(HIVE_METASTORE_TOKEN_ALIAS), 
> metastoreToken);
>   LOG.debug("Successfully fetched hive metastore delegation token. " + 
> metastoreToken);
> } catch (Exception ex) {
>   LOG.error("Couldn't fetch delegation token.", ex);
>   throw new RuntimeException("Couldn't fetch delegation token.", ex);
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SQOOP-2457) Add option to automatically compute statistics after loading date into a hive table

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-2457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo updated SQOOP-2457:

Fix Version/s: (was: 1.4.7)
   1.5.0

> Add option to  automatically compute statistics after loading date into a 
> hive table
> 
>
> Key: SQOOP-2457
> URL: https://issues.apache.org/jira/browse/SQOOP-2457
> Project: Sqoop
>  Issue Type: Improvement
>Affects Versions: 1.4.6
>Reporter: Venkat Ranganathan
>Assignee: Venkat Ranganathan
> Fix For: 1.5.0
>
>
> With CBO and different execution engines like Tez depedning on statistics 
> like row count heavily, it is important that we provide the option to update 
> stats on data loaded into Hive as part of the --hive-import option.  Ideally 
> these should be Hive managed, but there are use cases where this is not 
> automatic and hence this option will help in those cases
> Will be disabled by default.   Enabled by a flag 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SQOOP-2649) Support for importing data onto Apache Phoenix tables

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo updated SQOOP-2649:

Fix Version/s: (was: 1.4.7)
   1.5.0

> Support for importing data onto Apache Phoenix tables
> -
>
> Key: SQOOP-2649
> URL: https://issues.apache.org/jira/browse/SQOOP-2649
> Project: Sqoop
>  Issue Type: New Feature
>  Components: hbase-integration
>Reporter: maghamravikiran
> Fix For: 1.5.0
>
> Attachments: phoenix_sqoop.patch, phoenix_sqoop_integration.diff
>
>
> Support importing data to Phoenix backed HBase . 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SQOOP-2894) Hive import with Parquet failed in Kerberos enabled cluster

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo updated SQOOP-2894:

Fix Version/s: (was: 1.4.7)
   1.5.0

> Hive import with Parquet failed in Kerberos enabled cluster
> ---
>
> Key: SQOOP-2894
> URL: https://issues.apache.org/jira/browse/SQOOP-2894
> Project: Sqoop
>  Issue Type: Bug
>  Components: hive-integration, tools
>Affects Versions: 1.4.6
> Environment: Redhat 6.6, Sqoop 1.4.6+Hadoop 2.7.2+Hive 1.2.1
>Reporter: Ping Wang
>  Labels: security
> Fix For: 1.5.0
>
>
> Importing data from external database to hive with Parquet option failed in 
> the kerberos environment. (It can success without kerberos). 

> The sqoop command I used:
> sqoop import --connect jdbc:db2://xxx:5/testdb --username xxx --password 
> xxx --table users --hive-import -hive-table users3 --as-parquetfile -m 1
> The import job failed:

> ..
> 2016-02-26 04:20:07,020 INFO [main] 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Using mapred newApiCommitter.
> 2016-02-26 04:20:08,088 INFO [main] 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config 
> null
> 2016-02-26 04:20:08,918 INFO [main] hive.metastore: Trying to connect to 
> metastore with URI thrift://xxx:9083
> 2016-02-26 04:30:09,207 WARN [main] hive.metastore: set_ugi() not successful, 
> Likely cause: new client talking to old server. Continuing without it.
> org.apache.thrift.transport.TTransportException: 
> java.net.SocketTimeoutException: Read timed out
> at 
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
> at 
> org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:380)
> at 
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:230)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
> at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_set_ugi(ThriftHiveMetastore.java:3688)
> at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.set_ugi(ThriftHiveMetastore.java:3674)
> at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:448)
> at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:237)
> at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:182)
> at org.kitesdk.data.spi.hive.MetaStoreUtil.(MetaStoreUtil.java:82)
> at 
> org.kitesdk.data.spi.hive.HiveAbstractMetadataProvider.getMetaStoreUtil(HiveAbstractMetadataProvider.java:63)
> at 
> org.kitesdk.data.spi.hive.HiveAbstractMetadataProvider.resolveNamespace(HiveAbstractMetadataProvider.java:270)
> at 
> org.kitesdk.data.spi.hive.HiveAbstractMetadataProvider.resolveNamespace(HiveAbstractMetadataProvider.java:255)
> at 
> org.kitesdk.data.spi.hive.HiveAbstractMetadataProvider.load(HiveAbstractMetadataProvider.java:102)
> at 
> org.kitesdk.data.spi.filesystem.FileSystemDatasetRepository.load(FileSystemDatasetRepository.java:192)
> at org.kitesdk.data.Datasets.load(Datasets.java:108)
> at org.kitesdk.data.Datasets.load(Datasets.java:165)
> at 
> org.kitesdk.data.mapreduce.DatasetKeyOutputFormat.load(DatasetKeyOutputFormat.java:510)
> at 
> org.kitesdk.data.mapreduce.DatasetKeyOutputFormat.getOutputCommitter(DatasetKeyOutputFormat.java:473)
> at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.call(MRAppMaster.java:476)
> at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.call(MRAppMaster.java:458)
> at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.callWithJobClassLoader(MRAppMaster.java:1560)
> at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:458)
> at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:377)
> at 
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java:1518)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1515)
> at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1448)
> ... 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SQOOP-1906) Export support for mixed update/insert against db2

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-1906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo updated SQOOP-1906:

Fix Version/s: (was: 1.4.7)
   1.5.0

>  Export support for mixed  update/insert against db2
> 
>
> Key: SQOOP-1906
> URL: https://issues.apache.org/jira/browse/SQOOP-1906
> Project: Sqoop
>  Issue Type: Improvement
>  Components: connectors
>Affects Versions: 1.4.5
> Environment: RedHat6.4+Sqoop1.4.5+Hadoop2.4.1+Hive0.13.0+DB2 10.5
>Reporter: xieshiju
>Assignee: xieshiju
>  Labels: features
> Fix For: 1.5.0
>
> Attachments: SQOOP-1906.patch
>
>   Original Estimate: 504h
>  Remaining Estimate: 504h
>
> The DB2 connector need to implemented the option --update-mode allowinsert. 
> option --update-mode allowinsert which can be used when loading into a table 
> that is not empty. 
> In allowinsert mode export will try to update the row but if ithe row does 
> not exist it will insert it. Without this option only rows that already exist 
> in the table will be updated and any new rows are ignored.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SQOOP-1301) 'run' method in Sqoop.java is not thread-safe

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo updated SQOOP-1301:

Fix Version/s: (was: 1.4.7)
   1.5.0

> 'run' method in Sqoop.java is not thread-safe
> -
>
> Key: SQOOP-1301
> URL: https://issues.apache.org/jira/browse/SQOOP-1301
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.4.4
>Reporter: Ajay Chitre
> Fix For: 1.5.0
>
>
> It seems the ‘run’ method in Sqoop is not thread-safe.  When multiple Sqoop 
> jobs are triggered exactly at the same time, they end up stepping on each 
> other.  Here’s what seems to be the problem:  The ‘run’ method calls 
> tool.parseArguments which in turn calls 
> ConfigurationHelper.parseGenericOptions which in turn creates a new instance 
> of GenericOptionsParser.  The constructor for GenericOptionsParser is not 
> thread-safe because it ends up calling ‘buildGeneralOptions’ which uses 
> OptionBuilder.withArgName.  This method uses instance variables thereby 
> making it thread unsafe.
> The way we’ve got around it is by creating a ‘Lock’ object.  This seems to be 
> working.  If there’s a better way, please let us know.  If not, please 
> consider adding this feature.  We can create a patch if there’s an interest.  
> Thanks.
> Ajay Chitre (achi...@cisco.com)
> Virendra Singh (virsi...@cisco.com)
> Anyway, here’s what we’ve done:
> 1)  Created a class that extends Sqoop.java
> public class DlSqoop extends Sqoop {
> 2)  Created a Lock object:
> private static Lock monitor = new ReentrantLock();
> 3)  Overridden ‘run’ method as follows:
>   public int run(String [] args) {
> if (options.getConf() == null) {
>   // Configuration wasn't initialized until after the ToolRunner
>   // got us to this point. ToolRunner gave Sqoop itself a Conf
>   // though.
>   options.setConf(getConf());
> }
> try {
>   monitor.lock();
>   options = tool.parseArguments(args, null, options, false);
>   tool.appendArgs(this.childPrgmArgs);
>   tool.validateOptions(options);
> } catch (Exception e) {
>   // Couldn't parse arguments.
>   // Log the stack trace for this exception
>   LOG.debug(e.getMessage(), e);
>   // Print exception message.
>   System.err.println(e.getMessage());
>   // Print the tool usage message and exit.
>   ToolOptions toolOpts = new ToolOptions();
>   tool.configureOptions(toolOpts);
>   tool.printHelp(toolOpts);
>   return 1; // Exit on exception here.
>} finally {
> monitor.unlock();
> }
> return tool.run(options);
>   }



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SQOOP-1295) Sqoop Merge supports only one column merge key. Merge fails if source table as composite key

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-1295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo updated SQOOP-1295:

Fix Version/s: (was: 1.4.7)
   1.5.0

> Sqoop Merge supports only one column merge key. Merge fails if source table 
> as composite key
> 
>
> Key: SQOOP-1295
> URL: https://issues.apache.org/jira/browse/SQOOP-1295
> Project: Sqoop
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 1.4.4
> Environment: Linux
>Reporter: Virendra Singh
> Fix For: 1.5.0
>
> Attachments: DlMergeMapperBase.java, DlMergeMapperBase_Test.java
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SQOOP-1600) Exception when import data using Data Connector for Oracle with TIMESTAMP column type to Parquet files

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-1600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo updated SQOOP-1600:

Fix Version/s: (was: 1.4.7)
   1.5.0

> Exception when import data using Data Connector for Oracle with TIMESTAMP 
> column type to Parquet files
> --
>
> Key: SQOOP-1600
> URL: https://issues.apache.org/jira/browse/SQOOP-1600
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.4.6
> Environment: Hadoop version: 2.5.0-cdh5.2.0
> Sqoop: 1.4.5
>Reporter: Daniel Lanza García
>Assignee: Qian Xu
>  Labels: Connector, Oracle, Parquet, Timestamp
> Fix For: 1.5.0
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> A error is thrown in each mapper when a import job is run using Quest data 
> connector for Oracle (-direct argument), the source table has a column of the 
> type timestamp and the destination files are of Parquet format.
> The mapper's log show that the error is the following:
> {code}
> WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : 
> org.apache.avro.UnresolvedUnionException: Not in union ["long","null"]: 
> 2012-7-1 0:4:44. 40300
> {code}
> Which means the data obtained by the mapper (by the connector) is not of the 
> same type that the schema describe in this field. As we can read in the 
> error, the problem is related with the column UTC_STAMP (the unique column in 
> the source table that store a time stamp).
> If we check the generated schema for this column, we can observe that the 
> column is of the type long and SQL data type TIMESTAMP (93), which is correct.
> {code}
> Schema: {"name" : "UTC_STAMP","type" : [ "long", "null" ],"columnName" : 
> "UTC_STAMP","sqlType" : "93"}
> {code}
> If we debug the method where the exception is thrown 
> {{org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:605)}}, 
> we can see that the problem comes when the type of the data obtained by the 
> mapper is of the type String which doesn't correspond with the type described 
> by the schema (long). The exception is not thrown when the destination files 
> are text files. The reason is that when you import to text files, a schema is 
> not generated.
> Solution:
> In the documentation, there is a section which describe how manage data and 
> timestamps when you use the Data Connector for Oracle and Hadoop. As we can 
> read in this section, this connector has a different way to manage this type 
> of data. However, this behavior can be disabled as describe this section with 
> the below parameter.
> -Doraoop.timestamp.string=false
> Although the problem is solved with this parameter (mandatory if you are in 
> this conditions), the software should deal with this types of column and 
> doesn't throw an exception.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SQOOP-1289) Sqoop hbase-bulkload does not work with composite key

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-1289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo updated SQOOP-1289:

Fix Version/s: (was: 1.4.7)
   1.5.0

> Sqoop hbase-bulkload does not work with composite key
> -
>
> Key: SQOOP-1289
> URL: https://issues.apache.org/jira/browse/SQOOP-1289
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.4.5
>Reporter: hahakubile
>  Labels: hbase-bulkload
> Fix For: 1.5.0
>
>
> when executing command below:
> ./sqoop import  --connect jdbc:mysql://$MYSQL_SERVER/$DATABASE --driver 
> com.mysql.jdbc.Driver --username $USER --password $PASSWORD --table $TABLE 
> --hbase-table $HBASE_TABLE --column-family f --hbase-row-key id,proid 
> --hbase-bulkload
>  
> java.io.IOException: Could not insert row with null value for row-key column: 
> id,proid
> at 
> org.apache.sqoop.hbase.ToStringPutTransformer.getPutCommand(ToStringPutTransformer.java:146)
> at 
> org.apache.sqoop.mapreduce.HBaseBulkImportMapper.map(HBaseBulkImportMapper.java:86)
> at 
> org.apache.sqoop.mapreduce.HBaseBulkImportMapper.map(HBaseBulkImportMapper.java:44)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> at 
> org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:648)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:322)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278)
> at org.apache.hadoop.mapred.Child.main(Child.java:260)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SQOOP-1807) Incremental import to HBase with free form query is broken

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-1807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo updated SQOOP-1807:

Fix Version/s: (was: 1.4.7)
   1.5.0

> Incremental import to HBase with free form query is broken
> --
>
> Key: SQOOP-1807
> URL: https://issues.apache.org/jira/browse/SQOOP-1807
> Project: Sqoop
>  Issue Type: Bug
>  Components: hbase-integration
>Affects Versions: 1.4.5
>Reporter: Abraham Elmahrek
> Fix For: 1.5.0
>
>
> See 
> http://mail-archives.apache.org/mod_mbox/sqoop-user/201411.mbox/%3CCAH76rPar1gQF0_9FFTaTOd-7uVosdvXgyv%3Dc5xoNGCmV-zOyFQ%40mail.gmail.com%3E
>  for more info.
> It appears Sqoop is trying to append to a directory that doesn't exist. It 
> shouldn't be appending to a directory because it's writing to HBase. When 
> stepping through the code, the trace is:
> {noformat}
> status: 'RUNNING'
> at org.apache.sqoop.util.AppendUtils.append(AppendUtils.java:78)
> at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:504)
> at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
> at org.apache.sqoop.tool.JobTool.execJob(JobTool.java:228)
> at org.apache.sqoop.tool.JobTool.run(JobTool.java:283)
> at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SQOOP-1252) Sqoop import from db2. Schema of import not in expected format.

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-1252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo updated SQOOP-1252:

Fix Version/s: (was: 1.4.7)
   1.5.0

> Sqoop import from db2. Schema of import not in expected format.
> ---
>
> Key: SQOOP-1252
> URL: https://issues.apache.org/jira/browse/SQOOP-1252
> Project: Sqoop
>  Issue Type: Bug
>  Components: connectors
>Affects Versions: 1.4.5
>Reporter: Arijit Choudhury
>Assignee: Nick White
> Fix For: 1.5.0
>
> Attachments: SQOOP-1252.patch
>
>
> When I try to sqoop import (Datastax 3.2.0) from DB2 database using the below 
> command:
> ./dse sqoop import --connect jdbc:db2://172.29.252.40:4922/DSNN --username 
> tst -P --table tstschema."dsn_filter_table" --cassandra-keyspace SqoopTest 
> --cassandra-column-family actest2 --cassandra-row-key PREDNO 
> --cassandra-thrift-host 10.247.31.42 --cassandra-create-schema --split-by 
> PREDNO
> [ DB2 Select query: select * from SchemaName.TableName with ur; ]
>  
> Why I am not getting the schema in proper format as it is in DB2? Why in 
> Cassandra column name are getting into rows? Request your help to resolve the 
> issue.
>  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SQOOP-1760) Relocated hadoop installation

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-1760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo updated SQOOP-1760:

Fix Version/s: (was: 1.4.7)
   1.5.0

> Relocated hadoop installation
> -
>
> Key: SQOOP-1760
> URL: https://issues.apache.org/jira/browse/SQOOP-1760
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.4.5
>Reporter: Abraham Elmahrek
>Assignee: Abraham Elmahrek
> Fix For: 1.5.0
>
>
> It's possible to get:
> {noformat}
> /usr/lib/hadoop/bin/hadoop: No such file or directory
> {noformat}
> The fix is to check for the binary.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (SQOOP-1198) Make Sqoop build aware of the protobuf library

[Attila Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-1198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Szabo updated SQOOP-1198:

Fix Version/s: (was: 1.4.7)
   1.5.0

> Make Sqoop build aware of the protobuf library
> --
>
> Key: SQOOP-1198
> URL: https://issues.apache.org/jira/browse/SQOOP-1198
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.4.4
>Reporter: Jarek Jarcec Cecho
> Fix For: 1.5.0
>
> Attachments: SQOOP-1198.patch
>
>
> Currently Sqoop build is using protobuf from local machine. As different 
> Hadoop versions are using different protobuf versions, I would suggest to 
> make the build aware of the version and pull in version that is required.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)