[jira] [Assigned] (SQOOP-3215) Sqoop create hive table to support other formats(avro,parquet)

[Eric Lin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Lin reassigned SQOOP-3215:
---

Assignee: Eric Lin

> Sqoop create hive table to support other formats(avro,parquet)
> --
>
> Key: SQOOP-3215
> URL: https://issues.apache.org/jira/browse/SQOOP-3215
> Project: Sqoop
>  Issue Type: Improvement
>Affects Versions: 1.4.6
>Reporter: Nitish Khanna
>Assignee: Eric Lin
>
> Hi Team,
> Sqoop doesn't support any other format apart from text format when we make 
> use of "create-hive-table".
> It would be great if sqoop could create avro,parquet etc format table(schema 
> only).
> I tried below command to create avro format table in hive.
> [root@host-10-17-81-13 ~]# sqoop create-hive-table --connect $MYCONN 
> --username $MYUSER --password $MYPSWD --table test_table --hive-table 
> test_table_avro --as-avrodatafile
> Warning: 
> /opt/cloudera/parcels/CDH-5.8.3-1.cdh5.8.3.p0.2/bin/../lib/sqoop/../accumulo 
> does not exist! Accumulo imports will fail.
> Please set $ACCUMULO_HOME to the root of your Accumulo installation.
> 17/07/26 21:23:38 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.8.3
> 17/07/26 21:23:38 WARN tool.BaseSqoopTool: Setting your password on the 
> command-line is insecure. Consider using -P instead.
> 17/07/26 21:23:38 ERROR tool.BaseSqoopTool: Error parsing arguments for 
> create-hive-table:
> 17/07/26 21:23:38 ERROR tool.BaseSqoopTool: Unrecognized argument: 
> --as-avrodatafile
> Please correct me if i missed anything.
> Regards
> Nitish Khanna



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (SQOOP-3215) Sqoop create hive table to support other formats(avro,parquet)

[Nitish Khanna (JIRA)]

Nitish Khanna created SQOOP-3215:


 Summary: Sqoop create hive table to support other 
formats(avro,parquet)
 Key: SQOOP-3215
 URL: https://issues.apache.org/jira/browse/SQOOP-3215
 Project: Sqoop
  Issue Type: Improvement
Affects Versions: 1.4.6
Reporter: Nitish Khanna


Hi Team,

Sqoop doesn't support any other format apart from text format when we make use 
of "create-hive-table".

It would be great if sqoop could create avro,parquet etc format table(schema 
only).

I tried below command to create avro format table in hive.

[root@host-10-17-81-13 ~]# sqoop create-hive-table --connect $MYCONN --username 
$MYUSER --password $MYPSWD --table test_table --hive-table test_table_avro 
--as-avrodatafile
Warning: 
/opt/cloudera/parcels/CDH-5.8.3-1.cdh5.8.3.p0.2/bin/../lib/sqoop/../accumulo 
does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
17/07/26 21:23:38 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.8.3
17/07/26 21:23:38 WARN tool.BaseSqoopTool: Setting your password on the 
command-line is insecure. Consider using -P instead.
17/07/26 21:23:38 ERROR tool.BaseSqoopTool: Error parsing arguments for 
create-hive-table:
17/07/26 21:23:38 ERROR tool.BaseSqoopTool: Unrecognized argument: 
--as-avrodatafile

Please correct me if i missed anything.

Regards
Nitish Khanna



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Using pull requests for reviews

[Boglarka Egyed]

Hi Szabolcs,

I think this is pretty natural question.

I did some quick research on the topic and found one limitation with the
pull requests so far: people can not be assigned specifically as reviewers,
however, I don't think this would be a problem as every pull request and
related updates (comments, changes) would be sent to the dev@ mailing list.

>From my point of view the most important features of reviewing would be:

   - downloadable patch file
   - commenting on specific line(s)
   - editable description and comments
   - tracked history (including comments and changes)

As far as I see it all of these are available via pull requests too.

To the best of my knowledge Flume also uses pull requests beside using
ReviewBoard for example.

However, it would be nice to see what has driven the community to the
ReviewBoard initially as we could miss something here.

Thanks,
Bogi


On Mon, Jul 24, 2017 at 3:30 PM, Szabolcs Vasas  wrote:

> Hi All,
>
> I have used GitHub's pull request feature a couple of times recently and I
> found it very convenient so I was wondering if we could start using it for
> the Sqoop patches as well instead of Review board. I think submitting and
> committing patches is much more simple with pull requests since you do not
> have to manually create the patch file and upload it to the JIRA and the
> review board.
> What do you guys think? Does RB have an advantage over pull request or did
> we use it because of historical reasons?
>
> Regards,
> Szabolcs
>

[GitHub] sqoop issue #39: Testpullrequest

[ebogi]

Github user ebogi commented on the issue:

https://github.com/apache/sqoop/pull/39
  
Test comment.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Changing behavior on trunk

[Boglarka Egyed]

Hi All,

+1 to Szabi's comment. As it is a bugfix I wouldn't think that a new option
should be introduced to switch it on and off. However, it should be
emphasized in the release notes (in whichever release it will be included)
that this is a behavior changing fix.

Thanks for bringing it up Anna!

Cheers,
Bogi

On Fri, Jul 21, 2017 at 3:14 PM, Szabolcs Vasas  wrote:

> Hi guys,
>
> I think that this is basically a bug fix and because of that I would not
> add a new flag to switch it on/off.
> It is true that it changes behavior but many other bug fixes do the same so
> if we add a switch now we might want to do the same thing later and that
> could lead to a large number of extra command line arguments.
>
> Regards,
> Szabolcs
>
> On Thu, Jul 20, 2017 at 8:29 PM, Anna Szonyi  wrote:
>
> > Hi @dev list,
> >
> > Jilani has created a great patch for SQOOP-3149
> > : Sqoop incremental
> > import - NULL column updates are not pulled into HBase table.
> >
> > This is a patch that (in my opinion) corrects previously incorrect
> behavior
> > in sqoop, however it can also be seen as just changing behavior from the
> > users perspective ("we did not have nulls there before, now we have").
> >
> > The patch is ready to be committed, however I wanted to ask the dev list
> on
> > thoughts and opinions on whether we should merge the change as-is or
> should
> > we include a flag to turn this feature on and off? And if we need to
> > include a flag, should it be in a separate commit or should we amend the
> > original patch?
> >
> > Please find the review request below:
> > https://reviews.apache.org/r/57499/
> >
> > Please chime in with your thoughts - if there are no objections on this
> > thread, I plan to merge the change as-is in a week or so and pick up this
> > conversation again after.
> >
> > Thanks,
> > Anna
> >
>
>
>
> --
> Szabolcs Vasas
> Software Engineer
> 
>

[GitHub] sqoop pull request #39: Testpullrequest

[szvasas]

Github user szvasas closed the pull request at:

https://github.com/apache/sqoop/pull/39


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] sqoop pull request #39: Testpullrequest

[szvasas]

GitHub user szvasas opened a pull request:

https://github.com/apache/sqoop/pull/39

Testpullrequest



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/szvasas/sqoop testpullrequest

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/sqoop/pull/39.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #39






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] sqoop pull request #38: Metastore

[szvasas]

Github user szvasas commented on a diff in the pull request:

https://github.com/apache/sqoop/pull/38#discussion_r129515098
  
--- Diff: src/docs/user/metastore-purpose.txt ---
@@ -22,6 +22,8 @@ Multiple users and/or remote users can define and execute 
saved jobs (created
 with +sqoop job+) defined in this metastore.
 
 Clients must be configured to connect to the metastore in +sqoop-site.xml+ 
or
-with the +--meta-connect+ argument.
+with the +--meta-connect+ argument. The +sqoop-site.xml+ configuration 
supports
--- End diff --

Thank you for adding documentation!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] sqoop pull request #38: Metastore

[szvasas]

Github user szvasas commented on a diff in the pull request:

https://github.com/apache/sqoop/pull/38#discussion_r129511691
  
--- Diff: src/java/com/cloudera/sqoop/metastore/GeneralJobStorage.java ---
@@ -15,22 +15,22 @@
  * See the License for the specific language governing permissions and
  * limitations under the License.
  */
-package com.cloudera.sqoop.metastore.hsqldb;
+package com.cloudera.sqoop.metastore;
 
 /**
  * @deprecated Moving to use org.apache.sqoop namespace.
  */
-public class HsqldbJobStorage
-extends org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage {
+public class GeneralJobStorage
--- End diff --

Can we use the GenericJobStorage?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---