[jira] [Commented] (SQOOP-1270) Sqoop PostgreSQL Export Upsert Functionality Request
[ https://issues.apache.org/jira/browse/SQOOP-1270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15348718#comment-15348718 ] Bolke de Bruin commented on SQOOP-1270: --- Postgres has this functionality now natively since 9.5 . Can this please be activated? > Sqoop PostgreSQL Export Upsert Functionality Request > > > Key: SQOOP-1270 > URL: https://issues.apache.org/jira/browse/SQOOP-1270 > Project: Sqoop > Issue Type: New Feature > Components: connectors/postgresql >Affects Versions: 1.4.3 > Environment: CDH 4.5 >Reporter: Hari Sekhon > > Hi, > One of my Hadoop developers is trying to re-export to PostgreSQL but is > hitting the following error: > ERROR tool.ExportTool: Error during export: Mixed update/insert is not > supported against the target database yet > Could we please add upsert functionality to the PostgreSQL connector? > Although Postgres doesn't have direct upsert functionality it's possible to > construct SQL to handle this like so: > http://www.postgresql.org/docs/current/static/plpgsql-control-structures.html#PLPGSQL-UPSERT-EXAMPLE > A similar request to add upsert functionality for MySQL was addressed here: > https://issues.apache.org/jira/browse/SQOOP-621 > Thanks > Hari Sekhon > http://www.linkedin.com/in/harisekhon -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (SQOOP-2976) Flag to expand decimal values to fit AVRO schema
Thomas Scott created SQOOP-2976: --- Summary: Flag to expand decimal values to fit AVRO schema Key: SQOOP-2976 URL: https://issues.apache.org/jira/browse/SQOOP-2976 Project: Sqoop Issue Type: Improvement Affects Versions: 1.4.6 Reporter: Thomas Scott As per https://issues.apache.org/jira/browse/AVRO-1864 when importing from Oracle (or any other database that truncates decimals) Sqoop jobs can fail because the scale of the decimal produced by the database does not match the scale in the AVRO file. For instance if the value 3.15 is produced by Oracle and the AVRO decimal scale is 3 (this can happen even if the Oracle column is defined with scale of 3) then the job will fail. Can we have a flag (--pad-decimals) that pads incoming values with zeros to fit the AVRO schema (e.g. 3.15 becomes 3.150). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SQOOP-2971) OraOop does not close connections properly
[ https://issues.apache.org/jira/browse/SQOOP-2971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15348391#comment-15348391 ] Markus Kemper commented on SQOOP-2971: -- Thank you [~vasas], In the field we have seen this issue result in the following Oracle exception thus limited the degree of parallelism possible when SESSIONS are capped. 16/04/12 13:26:22 INFO mapreduce.Job: Task Id : attempt_1459758835311_12642_m_07_0, Status : FAILED Error: java.lang.RuntimeException: java.lang.RuntimeException: java.sql.SQLException: ORA-02391: exceeded simultaneous SESSIONS_PER_USER limit at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:749) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: java.lang.RuntimeException: java.sql.SQLException: ORA-02391: exceeded simultaneous SESSIONS_PER_USER limit > OraOop does not close connections properly > -- > > Key: SQOOP-2971 > URL: https://issues.apache.org/jira/browse/SQOOP-2971 > Project: Sqoop > Issue Type: Bug >Reporter: Attila Szabo >Assignee: Szabolcs Vasas > Fix For: 1.4.6, 1.4.7 > > Attachments: SQOOP-2971.patch > > > When importing with OraOop for every mapper task two connections are created, > and those connections are not closed until the end of the import process, > thus Sqoop use twice of the necessary amount, and kinda leaks the expensive > resources. This could cause problem when the number of the connections is > limited on the DB server side, however the user would like to leverage from > having more mappers (e.g. right now with 10 avail connections the user cannot > use 10 mappers, but instead only 4). > Please provide a fix for this issue either by sharing the connection, or > closing the connections when they are not needed anymore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Review Request 49192: OraOop does not close connections properly
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49192/ --- Review request for Sqoop. Bugs: SQOOP-2971 https://issues.apache.org/jira/browse/SQOOP-2971 Repository: sqoop-trunk Description --- Connections are now closed properly in OraOopDataDrivenDBInputFormat class. Diffs - src/java/org/apache/sqoop/manager/oracle/OraOopDataDrivenDBInputFormat.java 13f05d5 src/java/org/apache/sqoop/mapreduce/db/DBInputFormat.java 3a8e5d0 Diff: https://reviews.apache.org/r/49192/diff/ Testing --- We have started an import job on a cluster with a patched Sqoop version and we have verified that 1 map task now uses only 1 Oracle connection. Thanks, Szabolcs Vasas
[jira] [Updated] (SQOOP-2971) OraOop does not close connections properly
[ https://issues.apache.org/jira/browse/SQOOP-2971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szabolcs Vasas updated SQOOP-2971: -- Attachment: SQOOP-2971.patch > OraOop does not close connections properly > -- > > Key: SQOOP-2971 > URL: https://issues.apache.org/jira/browse/SQOOP-2971 > Project: Sqoop > Issue Type: Bug >Reporter: Attila Szabo >Assignee: Szabolcs Vasas > Fix For: 1.4.6, 1.4.7 > > Attachments: SQOOP-2971.patch > > > When importing with OraOop for every mapper task two connections are created, > and those connections are not closed until the end of the import process, > thus Sqoop use twice of the necessary amount, and kinda leaks the expensive > resources. This could cause problem when the number of the connections is > limited on the DB server side, however the user would like to leverage from > having more mappers (e.g. right now with 10 avail connections the user cannot > use 10 mappers, but instead only 4). > Please provide a fix for this issue either by sharing the connection, or > closing the connections when they are not needed anymore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SQOOP-2975) Sqoop loses part of data during import
[
https://issues.apache.org/jira/browse/SQOOP-2975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Svinarchuk updated SQOOP-2975:
-
Attachment: SQOOP-2975.patch
> Sqoop loses part of data during import
> ---
>
> Key: SQOOP-2975
> URL: https://issues.apache.org/jira/browse/SQOOP-2975
> Project: Sqoop
> Issue Type: Bug
>Affects Versions: 1.4.6
>Reporter: Sergey Svinarchuk
> Attachments: SQOOP-2975.patch
>
>
> Sqoop wrong splits columns with double(float) type.
> For example:
> If we split by column that looks like this:
> {code}
> 22.8
> 29.3
> 29.9
> 39.1
> 56.7
> 66.1
> 88.8
> 88.2
> 88.2
> 95.9
> {code}
> Sqoop splits it on the next ranges:
> First range - from 22.8 to 56.7
> Second range - from 95.9 to 95.9
> And then data will be lose
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (SQOOP-2975) Sqoop loses part of data during import
Sergey Svinarchuk created SQOOP-2975:
Summary: Sqoop loses part of data during import
Key: SQOOP-2975
URL: https://issues.apache.org/jira/browse/SQOOP-2975
Project: Sqoop
Issue Type: Bug
Affects Versions: 1.4.6
Reporter: Sergey Svinarchuk
Sqoop wrong splits columns with double(float) type.
For example:
If we split by column that looks like this:
{code}
22.8
29.3
29.9
39.1
56.7
66.1
88.8
88.2
88.2
95.9
{code}
Sqoop splits it on the next ranges:
First range - from 22.8 to 56.7
Second range - from 95.9 to 95.9
And then data will be lose
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
