[jira] [Updated] (ATLAS-1051) Sqoop Hook does not package HDFS model jars which is required
[
https://issues.apache.org/jira/browse/ATLAS-1051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Madhan Neethiraj updated ATLAS-1051:
Fix Version/s: 0.7.1-incubating
Committed to 0.7-incubating branch:
http://git-wip-us.apache.org/repos/asf/incubator-atlas/commit/3f853af0892cef927bf737ef35e0eddc968e8b9d
> Sqoop Hook does not package HDFS model jars which is required
> -
>
> Key: ATLAS-1051
> URL: https://issues.apache.org/jira/browse/ATLAS-1051
> Project: Atlas
> Issue Type: Bug
>Affects Versions: 0.7-incubating, 0.8-incubating
>Reporter: Suma Shivaprasad
>Assignee: Suma Shivaprasad
> Fix For: 0.8-incubating, 0.7.1-incubating
>
> Attachments: ATLAS-1051.patch
>
>
> Sqoop hook should package hdfs model jar and when sqoop commands to import
> are run, see this error since hdfs model jar is missing resulting in errors
> while loading data.
> {noformat}
> 2016-07-22 17:06:36,181 DEBUG [main]: ql.Driver
> (DriverContext.java:shutdown(132)) - Shutting down query
> LOAD DATA INPATH 'hdfs://localhost.localdomain:8020/user/hive/x_service' INTO
> TABLE `x_service`
> 2016-07-22 17:06:36,181 INFO [main]: CliDriver
> (SessionState.java:printInfo(939)) - Time taken: 4.791 seconds
> 2016-07-22 17:06:36,181 INFO [main]: log.PerfLogger
> (PerfLogger.java:PerfLogBegin(148)) - from=org.apache.hadoop.hive.ql.Driver>
> 2016-07-22 17:06:36,181 INFO [main]: log.PerfLogger
> (PerfLogger.java:PerfLogEnd(176)) - start=1469232396181 end=1469232396181 duration=0
> from=org.apache.hadoop.hive.ql.Driver>
> 2016-07-22 17:06:36,182 DEBUG [main]: session.SessionState
> (SessionState.java:close(1452)) - Removing resource dir
> /tmp/f60f6488-691f-440d-8fa0-01e26873f8a5_resources
> 2016-07-22 17:06:36,191 DEBUG [Atlas Logger 0]:
> classloader.AtlasPluginClassLoader
> (AtlasPluginClassLoader.java:loadClass(115)) - ==>
> AtlasPluginClassLoader.loadClass(org.apache.hadoop.fs.Path)
> 2016-07-22 17:06:36,192 DEBUG [Atlas Logger 0]:
> classloader.AtlasPluginClassLoader
> (AtlasPluginClassLoader.java:loadClass(123)) -
> AtlasPluginClassLoader.loadClass(org.apache.hadoop.fs.Path): calling
> pluginClassLoader.loadClass()
> 2016-07-22 17:06:36,193 DEBUG [Atlas Logger 0]:
> classloader.AtlasPluginClassLoader
> (AtlasPluginClassLoader.java:findClass(79)) - ==>
> AtlasPluginClassLoader.findClass(org.apache.hadoop.fs.Path)
> 2016-07-22 17:06:36,193 DEBUG [Atlas Logger 0]:
> classloader.AtlasPluginClassLoader
> (AtlasPluginClassLoader.java:findClass(87)) -
> AtlasPluginClassLoader.findClass(org.apache.hadoop.fs.Path): calling
> pluginClassLoader.findClass()
> 2016-07-22 17:06:36,193 DEBUG [Atlas Logger 0]:
> classloader.AtlasPluginClassLoader
> (AtlasPluginClassLoader.java:findClass(97)) -
> AtlasPluginClassLoader.findClass(org.apache.hadoop.fs.Path): calling
> componentClassLoader.findClass()
> 2016-07-22 17:06:36,193 DEBUG [Atlas Logger 0]:
> classloader.AtlasPluginClassLoader
> (AtlasPluginClassLoader.java:loadClass(133)) -
> AtlasPluginClassLoader.loadClass(org.apache.hadoop.fs.Path): calling
> componentClassLoader.loadClass()
> 2016-07-22 17:06:36,193 DEBUG [Atlas Logger 0]:
> classloader.AtlasPluginClassLoader
> (AtlasPluginClassLoader.java:loadClass(142)) - <==
> AtlasPluginClassLoader.loadClass(org.apache.hadoop.fs.Path): class
> org.apache.hadoop.fs.Path
> 2016-07-22 17:06:36,193 DEBUG [Atlas Logger 0]: hook.HiveHook
> (HiveHook.java:processHiveEntity(632)) - Registering DFS Path
> hdfs://localhost.localdomain:8020/user/hive/x_service
> 2016-07-22 17:06:36,193 DEBUG [Atlas Logger 0]:
> classloader.AtlasPluginClassLoader
> (AtlasPluginClassLoader.java:loadClass(115)) - ==>
> AtlasPluginClassLoader.loadClass(org.apache.atlas.fs.model.FSDataTypes)
> 2016-07-22 17:06:36,193 DEBUG [Atlas Logger 0]:
> classloader.AtlasPluginClassLoader
> (AtlasPluginClassLoader.java:loadClass(123)) -
> AtlasPluginClassLoader.loadClass(org.apache.atlas.fs.model.FSDataTypes):
> calling pluginClassLoader.loadClass()
> 2016-07-22 17:06:36,193 DEBUG [Atlas Logger 0]:
> classloader.AtlasPluginClassLoader
> (AtlasPluginClassLoader.java:findClass(79)) - ==>
> AtlasPluginClassLoader.findClass(org.apache.atlas.fs.model.FSDataTypes)
> 2016-07-22 17:06:36,194 DEBUG [Atlas Logger 0]:
> classloader.AtlasPluginClassLoader
> (AtlasPluginClassLoader.java:findClass(87)) -
> AtlasPluginClassLoader.findClass(org.apache.atlas.fs.model.FSDataTypes):
> calling pluginClassLoader.findClass()
> 2016-07-22 17:06:
[jira] [Resolved] (ATLAS-1050) Sqoop Hook does not package HDFS model jars which is required
[
https://issues.apache.org/jira/browse/ATLAS-1050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suma Shivaprasad resolved ATLAS-1050.
-
Resolution: Duplicate
> Sqoop Hook does not package HDFS model jars which is required
> -
>
> Key: ATLAS-1050
> URL: https://issues.apache.org/jira/browse/ATLAS-1050
> Project: Atlas
> Issue Type: Bug
>Reporter: Suma Shivaprasad
>Assignee: Suma Shivaprasad
>
> Sqoop hook should package hdfs model jar and when sqoop commands to import
> are run, see this error since hdfs model jar is missing resulting in errors
> while loading data.
> {noformat}
> 2016-07-22 17:06:36,181 DEBUG [main]: ql.Driver
> (DriverContext.java:shutdown(132)) - Shutting down query
> LOAD DATA INPATH 'hdfs://localhost.localdomain:8020/user/hive/x_service' INTO
> TABLE `x_service`
> 2016-07-22 17:06:36,181 INFO [main]: CliDriver
> (SessionState.java:printInfo(939)) - Time taken: 4.791 seconds
> 2016-07-22 17:06:36,181 INFO [main]: log.PerfLogger
> (PerfLogger.java:PerfLogBegin(148)) - from=org.apache.hadoop.hive.ql.Driver>
> 2016-07-22 17:06:36,181 INFO [main]: log.PerfLogger
> (PerfLogger.java:PerfLogEnd(176)) - start=1469232396181 end=1469232396181 duration=0
> from=org.apache.hadoop.hive.ql.Driver>
> 2016-07-22 17:06:36,182 DEBUG [main]: session.SessionState
> (SessionState.java:close(1452)) - Removing resource dir
> /tmp/f60f6488-691f-440d-8fa0-01e26873f8a5_resources
> 2016-07-22 17:06:36,191 DEBUG [Atlas Logger 0]:
> classloader.AtlasPluginClassLoader
> (AtlasPluginClassLoader.java:loadClass(115)) - ==>
> AtlasPluginClassLoader.loadClass(org.apache.hadoop.fs.Path)
> 2016-07-22 17:06:36,192 DEBUG [Atlas Logger 0]:
> classloader.AtlasPluginClassLoader
> (AtlasPluginClassLoader.java:loadClass(123)) -
> AtlasPluginClassLoader.loadClass(org.apache.hadoop.fs.Path): calling
> pluginClassLoader.loadClass()
> 2016-07-22 17:06:36,193 DEBUG [Atlas Logger 0]:
> classloader.AtlasPluginClassLoader
> (AtlasPluginClassLoader.java:findClass(79)) - ==>
> AtlasPluginClassLoader.findClass(org.apache.hadoop.fs.Path)
> 2016-07-22 17:06:36,193 DEBUG [Atlas Logger 0]:
> classloader.AtlasPluginClassLoader
> (AtlasPluginClassLoader.java:findClass(87)) -
> AtlasPluginClassLoader.findClass(org.apache.hadoop.fs.Path): calling
> pluginClassLoader.findClass()
> 2016-07-22 17:06:36,193 DEBUG [Atlas Logger 0]:
> classloader.AtlasPluginClassLoader
> (AtlasPluginClassLoader.java:findClass(97)) -
> AtlasPluginClassLoader.findClass(org.apache.hadoop.fs.Path): calling
> componentClassLoader.findClass()
> 2016-07-22 17:06:36,193 DEBUG [Atlas Logger 0]:
> classloader.AtlasPluginClassLoader
> (AtlasPluginClassLoader.java:loadClass(133)) -
> AtlasPluginClassLoader.loadClass(org.apache.hadoop.fs.Path): calling
> componentClassLoader.loadClass()
> 2016-07-22 17:06:36,193 DEBUG [Atlas Logger 0]:
> classloader.AtlasPluginClassLoader
> (AtlasPluginClassLoader.java:loadClass(142)) - <==
> AtlasPluginClassLoader.loadClass(org.apache.hadoop.fs.Path): class
> org.apache.hadoop.fs.Path
> 2016-07-22 17:06:36,193 DEBUG [Atlas Logger 0]: hook.HiveHook
> (HiveHook.java:processHiveEntity(632)) - Registering DFS Path
> hdfs://localhost.localdomain:8020/user/hive/x_service
> 2016-07-22 17:06:36,193 DEBUG [Atlas Logger 0]:
> classloader.AtlasPluginClassLoader
> (AtlasPluginClassLoader.java:loadClass(115)) - ==>
> AtlasPluginClassLoader.loadClass(org.apache.atlas.fs.model.FSDataTypes)
> 2016-07-22 17:06:36,193 DEBUG [Atlas Logger 0]:
> classloader.AtlasPluginClassLoader
> (AtlasPluginClassLoader.java:loadClass(123)) -
> AtlasPluginClassLoader.loadClass(org.apache.atlas.fs.model.FSDataTypes):
> calling pluginClassLoader.loadClass()
> 2016-07-22 17:06:36,193 DEBUG [Atlas Logger 0]:
> classloader.AtlasPluginClassLoader
> (AtlasPluginClassLoader.java:findClass(79)) - ==>
> AtlasPluginClassLoader.findClass(org.apache.atlas.fs.model.FSDataTypes)
> 2016-07-22 17:06:36,194 DEBUG [Atlas Logger 0]:
> classloader.AtlasPluginClassLoader
> (AtlasPluginClassLoader.java:findClass(87)) -
> AtlasPluginClassLoader.findClass(org.apache.atlas.fs.model.FSDataTypes):
> calling pluginClassLoader.findClass()
> 2016-07-22 17:06:36,194 DEBUG [Atlas Logger 0]:
> classloader.AtlasPluginClassLoader
> (AtlasPluginClassLoader.java:findClass(97)) -
> AtlasPluginClassLoader.findClass(org.apache.atlas.fs.model.FSDataTypes):
> calling componentClassLoader.findClass()
> 2016-07-22 17:06:36,194 DEBUG [Atlas Logger 0]:
> classloader.AtlasPluginClassLoader
> (AtlasPluginClassLoader.java:loadClass(133)) -
> AtlasPluginClassLoader.loadClass(org.apache.atlas.fs.model.FSDataTypes):
> calling componentClassLoader.loadClass()
> 2016-07-22 17:06:36,197 ERROR [Atlas Logger 0]: hook.HiveHook
> (HiveHook.java:run(188)) - Atlas hook
[jira] [Commented] (ATLAS-1051) Sqoop Hook does not package HDFS model jars which is required
[
https://issues.apache.org/jira/browse/ATLAS-1051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392685#comment-15392685
]
Madhan Neethiraj commented on ATLAS-1051:
-
+1 for the patch. Thanks.
> Sqoop Hook does not package HDFS model jars which is required
> -
>
> Key: ATLAS-1051
> URL: https://issues.apache.org/jira/browse/ATLAS-1051
> Project: Atlas
> Issue Type: Bug
>Affects Versions: 0.7-incubating, 0.8-incubating
>Reporter: Suma Shivaprasad
>Assignee: Suma Shivaprasad
> Fix For: 0.8-incubating
>
> Attachments: ATLAS-1051.patch
>
>
> Sqoop hook should package hdfs model jar and when sqoop commands to import
> are run, see this error since hdfs model jar is missing resulting in errors
> while loading data.
> {noformat}
> 2016-07-22 17:06:36,181 DEBUG [main]: ql.Driver
> (DriverContext.java:shutdown(132)) - Shutting down query
> LOAD DATA INPATH 'hdfs://localhost.localdomain:8020/user/hive/x_service' INTO
> TABLE `x_service`
> 2016-07-22 17:06:36,181 INFO [main]: CliDriver
> (SessionState.java:printInfo(939)) - Time taken: 4.791 seconds
> 2016-07-22 17:06:36,181 INFO [main]: log.PerfLogger
> (PerfLogger.java:PerfLogBegin(148)) - from=org.apache.hadoop.hive.ql.Driver>
> 2016-07-22 17:06:36,181 INFO [main]: log.PerfLogger
> (PerfLogger.java:PerfLogEnd(176)) - start=1469232396181 end=1469232396181 duration=0
> from=org.apache.hadoop.hive.ql.Driver>
> 2016-07-22 17:06:36,182 DEBUG [main]: session.SessionState
> (SessionState.java:close(1452)) - Removing resource dir
> /tmp/f60f6488-691f-440d-8fa0-01e26873f8a5_resources
> 2016-07-22 17:06:36,191 DEBUG [Atlas Logger 0]:
> classloader.AtlasPluginClassLoader
> (AtlasPluginClassLoader.java:loadClass(115)) - ==>
> AtlasPluginClassLoader.loadClass(org.apache.hadoop.fs.Path)
> 2016-07-22 17:06:36,192 DEBUG [Atlas Logger 0]:
> classloader.AtlasPluginClassLoader
> (AtlasPluginClassLoader.java:loadClass(123)) -
> AtlasPluginClassLoader.loadClass(org.apache.hadoop.fs.Path): calling
> pluginClassLoader.loadClass()
> 2016-07-22 17:06:36,193 DEBUG [Atlas Logger 0]:
> classloader.AtlasPluginClassLoader
> (AtlasPluginClassLoader.java:findClass(79)) - ==>
> AtlasPluginClassLoader.findClass(org.apache.hadoop.fs.Path)
> 2016-07-22 17:06:36,193 DEBUG [Atlas Logger 0]:
> classloader.AtlasPluginClassLoader
> (AtlasPluginClassLoader.java:findClass(87)) -
> AtlasPluginClassLoader.findClass(org.apache.hadoop.fs.Path): calling
> pluginClassLoader.findClass()
> 2016-07-22 17:06:36,193 DEBUG [Atlas Logger 0]:
> classloader.AtlasPluginClassLoader
> (AtlasPluginClassLoader.java:findClass(97)) -
> AtlasPluginClassLoader.findClass(org.apache.hadoop.fs.Path): calling
> componentClassLoader.findClass()
> 2016-07-22 17:06:36,193 DEBUG [Atlas Logger 0]:
> classloader.AtlasPluginClassLoader
> (AtlasPluginClassLoader.java:loadClass(133)) -
> AtlasPluginClassLoader.loadClass(org.apache.hadoop.fs.Path): calling
> componentClassLoader.loadClass()
> 2016-07-22 17:06:36,193 DEBUG [Atlas Logger 0]:
> classloader.AtlasPluginClassLoader
> (AtlasPluginClassLoader.java:loadClass(142)) - <==
> AtlasPluginClassLoader.loadClass(org.apache.hadoop.fs.Path): class
> org.apache.hadoop.fs.Path
> 2016-07-22 17:06:36,193 DEBUG [Atlas Logger 0]: hook.HiveHook
> (HiveHook.java:processHiveEntity(632)) - Registering DFS Path
> hdfs://localhost.localdomain:8020/user/hive/x_service
> 2016-07-22 17:06:36,193 DEBUG [Atlas Logger 0]:
> classloader.AtlasPluginClassLoader
> (AtlasPluginClassLoader.java:loadClass(115)) - ==>
> AtlasPluginClassLoader.loadClass(org.apache.atlas.fs.model.FSDataTypes)
> 2016-07-22 17:06:36,193 DEBUG [Atlas Logger 0]:
> classloader.AtlasPluginClassLoader
> (AtlasPluginClassLoader.java:loadClass(123)) -
> AtlasPluginClassLoader.loadClass(org.apache.atlas.fs.model.FSDataTypes):
> calling pluginClassLoader.loadClass()
> 2016-07-22 17:06:36,193 DEBUG [Atlas Logger 0]:
> classloader.AtlasPluginClassLoader
> (AtlasPluginClassLoader.java:findClass(79)) - ==>
> AtlasPluginClassLoader.findClass(org.apache.atlas.fs.model.FSDataTypes)
> 2016-07-22 17:06:36,194 DEBUG [Atlas Logger 0]:
> classloader.AtlasPluginClassLoader
> (AtlasPluginClassLoader.java:findClass(87)) -
> AtlasPluginClassLoader.findClass(org.apache.atlas.fs.model.FSDataTypes):
> calling pluginClassLoader.findClass()
> 2016-07-22 17:06:36,194 DEBUG [Atlas Logger 0]:
> classloader.AtlasPluginClassLoader
> (AtlasPluginClassLoader.java:findClass(97)) -
&
[jira] [Updated] (ATLAS-1051) Sqoop Hook does not package HDFS model jars which is required
[
https://issues.apache.org/jira/browse/ATLAS-1051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suma Shivaprasad updated ATLAS-1051:
Attachment: ATLAS-1051.patch
> Sqoop Hook does not package HDFS model jars which is required
> -
>
> Key: ATLAS-1051
> URL: https://issues.apache.org/jira/browse/ATLAS-1051
> Project: Atlas
> Issue Type: Bug
>Reporter: Suma Shivaprasad
>Assignee: Suma Shivaprasad
> Attachments: ATLAS-1051.patch
>
>
> Sqoop hook should package hdfs model jar and when sqoop commands to import
> are run, see this error since hdfs model jar is missing resulting in errors
> while loading data.
> {noformat}
> 2016-07-22 17:06:36,181 DEBUG [main]: ql.Driver
> (DriverContext.java:shutdown(132)) - Shutting down query
> LOAD DATA INPATH 'hdfs://localhost.localdomain:8020/user/hive/x_service' INTO
> TABLE `x_service`
> 2016-07-22 17:06:36,181 INFO [main]: CliDriver
> (SessionState.java:printInfo(939)) - Time taken: 4.791 seconds
> 2016-07-22 17:06:36,181 INFO [main]: log.PerfLogger
> (PerfLogger.java:PerfLogBegin(148)) - from=org.apache.hadoop.hive.ql.Driver>
> 2016-07-22 17:06:36,181 INFO [main]: log.PerfLogger
> (PerfLogger.java:PerfLogEnd(176)) - start=1469232396181 end=1469232396181 duration=0
> from=org.apache.hadoop.hive.ql.Driver>
> 2016-07-22 17:06:36,182 DEBUG [main]: session.SessionState
> (SessionState.java:close(1452)) - Removing resource dir
> /tmp/f60f6488-691f-440d-8fa0-01e26873f8a5_resources
> 2016-07-22 17:06:36,191 DEBUG [Atlas Logger 0]:
> classloader.AtlasPluginClassLoader
> (AtlasPluginClassLoader.java:loadClass(115)) - ==>
> AtlasPluginClassLoader.loadClass(org.apache.hadoop.fs.Path)
> 2016-07-22 17:06:36,192 DEBUG [Atlas Logger 0]:
> classloader.AtlasPluginClassLoader
> (AtlasPluginClassLoader.java:loadClass(123)) -
> AtlasPluginClassLoader.loadClass(org.apache.hadoop.fs.Path): calling
> pluginClassLoader.loadClass()
> 2016-07-22 17:06:36,193 DEBUG [Atlas Logger 0]:
> classloader.AtlasPluginClassLoader
> (AtlasPluginClassLoader.java:findClass(79)) - ==>
> AtlasPluginClassLoader.findClass(org.apache.hadoop.fs.Path)
> 2016-07-22 17:06:36,193 DEBUG [Atlas Logger 0]:
> classloader.AtlasPluginClassLoader
> (AtlasPluginClassLoader.java:findClass(87)) -
> AtlasPluginClassLoader.findClass(org.apache.hadoop.fs.Path): calling
> pluginClassLoader.findClass()
> 2016-07-22 17:06:36,193 DEBUG [Atlas Logger 0]:
> classloader.AtlasPluginClassLoader
> (AtlasPluginClassLoader.java:findClass(97)) -
> AtlasPluginClassLoader.findClass(org.apache.hadoop.fs.Path): calling
> componentClassLoader.findClass()
> 2016-07-22 17:06:36,193 DEBUG [Atlas Logger 0]:
> classloader.AtlasPluginClassLoader
> (AtlasPluginClassLoader.java:loadClass(133)) -
> AtlasPluginClassLoader.loadClass(org.apache.hadoop.fs.Path): calling
> componentClassLoader.loadClass()
> 2016-07-22 17:06:36,193 DEBUG [Atlas Logger 0]:
> classloader.AtlasPluginClassLoader
> (AtlasPluginClassLoader.java:loadClass(142)) - <==
> AtlasPluginClassLoader.loadClass(org.apache.hadoop.fs.Path): class
> org.apache.hadoop.fs.Path
> 2016-07-22 17:06:36,193 DEBUG [Atlas Logger 0]: hook.HiveHook
> (HiveHook.java:processHiveEntity(632)) - Registering DFS Path
> hdfs://localhost.localdomain:8020/user/hive/x_service
> 2016-07-22 17:06:36,193 DEBUG [Atlas Logger 0]:
> classloader.AtlasPluginClassLoader
> (AtlasPluginClassLoader.java:loadClass(115)) - ==>
> AtlasPluginClassLoader.loadClass(org.apache.atlas.fs.model.FSDataTypes)
> 2016-07-22 17:06:36,193 DEBUG [Atlas Logger 0]:
> classloader.AtlasPluginClassLoader
> (AtlasPluginClassLoader.java:loadClass(123)) -
> AtlasPluginClassLoader.loadClass(org.apache.atlas.fs.model.FSDataTypes):
> calling pluginClassLoader.loadClass()
> 2016-07-22 17:06:36,193 DEBUG [Atlas Logger 0]:
> classloader.AtlasPluginClassLoader
> (AtlasPluginClassLoader.java:findClass(79)) - ==>
> AtlasPluginClassLoader.findClass(org.apache.atlas.fs.model.FSDataTypes)
> 2016-07-22 17:06:36,194 DEBUG [Atlas Logger 0]:
> classloader.AtlasPluginClassLoader
> (AtlasPluginClassLoader.java:findClass(87)) -
> AtlasPluginClassLoader.findClass(org.apache.atlas.fs.model.FSDataTypes):
> calling pluginClassLoader.findClass()
> 2016-07-22 17:06:36,194 DEBUG [Atlas Logger 0]:
> classloader.AtlasPluginClassLoader
> (AtlasPluginClassLoader.java:findClass(97)) -
> AtlasPluginClassLoader.findClass(org.apache.atlas.fs.model.FSDataTypes):
> calling componentClassLoader.findClass()
> 2016-07-22 1
[jira] [Created] (ATLAS-1050) Sqoop Hook does not package HDFS model jars which is required
Suma Shivaprasad created ATLAS-1050:
---
Summary: Sqoop Hook does not package HDFS model jars which is
required
Key: ATLAS-1050
URL: https://issues.apache.org/jira/browse/ATLAS-1050
Project: Atlas
Issue Type: Bug
Reporter: Suma Shivaprasad
Assignee: Suma Shivaprasad
Sqoop hook should package hdfs model jar and when sqoop commands to import are
run, see this error since hdfs model jar is missing resulting in errors while
loading data.
{noformat}
2016-07-22 17:06:36,181 DEBUG [main]: ql.Driver
(DriverContext.java:shutdown(132)) - Shutting down query
LOAD DATA INPATH 'hdfs://localhost.localdomain:8020/user/hive/x_service' INTO
TABLE `x_service`
2016-07-22 17:06:36,181 INFO [main]: CliDriver
(SessionState.java:printInfo(939)) - Time taken: 4.791 seconds
2016-07-22 17:06:36,181 INFO [main]: log.PerfLogger
(PerfLogger.java:PerfLogBegin(148)) -
2016-07-22 17:06:36,181 INFO [main]: log.PerfLogger
(PerfLogger.java:PerfLogEnd(176)) -
2016-07-22 17:06:36,182 DEBUG [main]: session.SessionState
(SessionState.java:close(1452)) - Removing resource dir
/tmp/f60f6488-691f-440d-8fa0-01e26873f8a5_resources
2016-07-22 17:06:36,191 DEBUG [Atlas Logger 0]:
classloader.AtlasPluginClassLoader (AtlasPluginClassLoader.java:loadClass(115))
- ==> AtlasPluginClassLoader.loadClass(org.apache.hadoop.fs.Path)
2016-07-22 17:06:36,192 DEBUG [Atlas Logger 0]:
classloader.AtlasPluginClassLoader (AtlasPluginClassLoader.java:loadClass(123))
- AtlasPluginClassLoader.loadClass(org.apache.hadoop.fs.Path): calling
pluginClassLoader.loadClass()
2016-07-22 17:06:36,193 DEBUG [Atlas Logger 0]:
classloader.AtlasPluginClassLoader (AtlasPluginClassLoader.java:findClass(79))
- ==> AtlasPluginClassLoader.findClass(org.apache.hadoop.fs.Path)
2016-07-22 17:06:36,193 DEBUG [Atlas Logger 0]:
classloader.AtlasPluginClassLoader (AtlasPluginClassLoader.java:findClass(87))
- AtlasPluginClassLoader.findClass(org.apache.hadoop.fs.Path): calling
pluginClassLoader.findClass()
2016-07-22 17:06:36,193 DEBUG [Atlas Logger 0]:
classloader.AtlasPluginClassLoader (AtlasPluginClassLoader.java:findClass(97))
- AtlasPluginClassLoader.findClass(org.apache.hadoop.fs.Path): calling
componentClassLoader.findClass()
2016-07-22 17:06:36,193 DEBUG [Atlas Logger 0]:
classloader.AtlasPluginClassLoader (AtlasPluginClassLoader.java:loadClass(133))
- AtlasPluginClassLoader.loadClass(org.apache.hadoop.fs.Path): calling
componentClassLoader.loadClass()
2016-07-22 17:06:36,193 DEBUG [Atlas Logger 0]:
classloader.AtlasPluginClassLoader (AtlasPluginClassLoader.java:loadClass(142))
- <== AtlasPluginClassLoader.loadClass(org.apache.hadoop.fs.Path): class
org.apache.hadoop.fs.Path
2016-07-22 17:06:36,193 DEBUG [Atlas Logger 0]: hook.HiveHook
(HiveHook.java:processHiveEntity(632)) - Registering DFS Path
hdfs://localhost.localdomain:8020/user/hive/x_service
2016-07-22 17:06:36,193 DEBUG [Atlas Logger 0]:
classloader.AtlasPluginClassLoader (AtlasPluginClassLoader.java:loadClass(115))
- ==> AtlasPluginClassLoader.loadClass(org.apache.atlas.fs.model.FSDataTypes)
2016-07-22 17:06:36,193 DEBUG [Atlas Logger 0]:
classloader.AtlasPluginClassLoader (AtlasPluginClassLoader.java:loadClass(123))
- AtlasPluginClassLoader.loadClass(org.apache.atlas.fs.model.FSDataTypes):
calling pluginClassLoader.loadClass()
2016-07-22 17:06:36,193 DEBUG [Atlas Logger 0]:
classloader.AtlasPluginClassLoader (AtlasPluginClassLoader.java:findClass(79))
- ==> AtlasPluginClassLoader.findClass(org.apache.atlas.fs.model.FSDataTypes)
2016-07-22 17:06:36,194 DEBUG [Atlas Logger 0]:
classloader.AtlasPluginClassLoader (AtlasPluginClassLoader.java:findClass(87))
- AtlasPluginClassLoader.findClass(org.apache.atlas.fs.model.FSDataTypes):
calling pluginClassLoader.findClass()
2016-07-22 17:06:36,194 DEBUG [Atlas Logger 0]:
classloader.AtlasPluginClassLoader (AtlasPluginClassLoader.java:findClass(97))
- AtlasPluginClassLoader.findClass(org.apache.atlas.fs.model.FSDataTypes):
calling componentClassLoader.findClass()
2016-07-22 17:06:36,194 DEBUG [Atlas Logger 0]:
classloader.AtlasPluginClassLoader (AtlasPluginClassLoader.java:loadClass(133))
- AtlasPluginClassLoader.loadClass(org.apache.atlas.fs.model.FSDataTypes):
calling componentClassLoader.loadClass()
2016-07-22 17:06:36,197 ERROR [Atlas Logger 0]: hook.HiveHook
(HiveHook.java:run(188)) - Atlas hook failed due to error
java.lang.NoClassDefFoundError: org/apache/atlas/fs/model/FSDataTypes
at
org.apache.atlas.hive.bridge.HiveMetaStoreBridge.fillHDFSDataSet(HiveMetaStoreBridge.java:556)
at
org.apache.atlas.hive.hook.HiveHook.processHiveEntity(HiveHook.java:634)
at
org.apache.atlas.hive.hook.HiveHook.registerProcess(HiveHook.java:599)
at org.apache.atlas.hive.hook.HiveHook.fireAndForget(HiveHook.java:227)
at org.apache.atlas.hive.hook.Hive
Re: Atlas and sqoop hook
Make sure you have latest sqoop, built from sqoop trunk. Use ‹verbose in sqoop command to get debug logs. From the logs, you can verify if sqoop atlas hook is invoked. Also make sure that the sqoop hook picks the right atlas conf Regards, Shwetha On 20/06/16, 11:58 PM, "Margus Roo" <mar...@roo.ee> wrote: >Hi > >Downloaded and compiled atlas 0.7. >Hive hook is working - create table [tablename] as select * from [src >tablename] is working and data lineage is generated in atlas. >Next I tried sqoop hook and followed >http://atlas.incubator.apache.org/Bridge-Sqoop.html > >Command: >sqoop-import --connect jdbc:mysql://mysqlhost/test --table sqoop_test >--split-by id --hive-import -hive-table sqoop_test19 --username margusja >--P >creates a new table in Hive and new table is in atlas also but no data >lineage > >I see from >http://hortonworks.com/hadoop-tutorial/cross-component-lineage-apache-atla >s/ >There I can see that extra config parameters are loaded (in picture >https://raw.githubusercontent.com/hortonworks/tutorials/atlas-ranger-tp/as >sets/cross-component-lineage-with-atlas/8-sqoop-import-finish.png) >and kafka producer creating ouutput but in my command: >sqoop-import --connect jdbc:mysql://mysqlhost/test --table sqoop_test >--split-by id --hive-import -hive-table sqoop_test19 --username margusja >--P >there is no extra output only: > >Warning: /usr/hdp/2.4.0.0-169/accumulo does not exist! Accumulo imports >will fail. >Please set $ACCUMULO_HOME to the root of your Accumulo installation. >16/06/20 21:25:47 INFO sqoop.Sqoop: Running Sqoop version: >1.4.6.2.4.0.0-169 >16/06/20 21:25:47 WARN tool.BaseSqoopTool: Setting your password on the >command-line is insecure. Consider using -P instead. >16/06/20 21:25:47 INFO tool.BaseSqoopTool: Using Hive-specific >delimiters for output. You can override >16/06/20 21:25:47 INFO tool.BaseSqoopTool: delimiters with >--fields-terminated-by, etc. >16/06/20 21:25:47 INFO manager.MySQLManager: Preparing to use a MySQL >streaming resultset. >16/06/20 21:25:47 INFO tool.CodeGenTool: Beginning code generation >16/06/20 21:25:47 INFO manager.SqlManager: Executing SQL statement: >SELECT t.* FROM `sqoop_test` AS t LIMIT 1 >16/06/20 21:25:47 INFO manager.SqlManager: Executing SQL statement: >SELECT t.* FROM `sqoop_test` AS t LIMIT 1 >16/06/20 21:25:47 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is >/usr/hdp/2.4.0.0-169/hadoop-mapreduce >Note: >/tmp/sqoop-root/compile/49b525e14ebd68542d86b68dc399bd84/sqoop_test.java >uses or overrides a deprecated API. >Note: Recompile with -Xlint:deprecation for details. >16/06/20 21:25:48 INFO orm.CompilationManager: Writing jar file: >/tmp/sqoop-root/compile/49b525e14ebd68542d86b68dc399bd84/sqoop_test.jar >16/06/20 21:25:48 WARN manager.MySQLManager: It looks like you are >importing from mysql. >16/06/20 21:25:48 WARN manager.MySQLManager: This transfer can be >faster! Use the --direct >16/06/20 21:25:48 WARN manager.MySQLManager: option to exercise a >MySQL-specific fast path. >16/06/20 21:25:48 INFO manager.MySQLManager: Setting zero DATETIME >behavior to convertToNull (mysql) >16/06/20 21:25:48 INFO mapreduce.ImportJobBase: Beginning import of >sqoop_test >SLF4J: Class path contains multiple SLF4J bindings. >SLF4J: Found binding in >[jar:file:/usr/hdp/2.4.0.0-169/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/sl >f4j/impl/StaticLoggerBinder.class] >SLF4J: Found binding in >[jar:file:/usr/hdp/2.4.0.0-169/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/ >slf4j/impl/StaticLoggerBinder.class] >SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an >explanation. >SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] >16/06/20 21:25:50 INFO impl.TimelineClientImpl: Timeline service >address: http://bigdata21.webmedia.int:8188/ws/v1/timeline/ >16/06/20 21:25:50 INFO client.RMProxy: Connecting to ResourceManager at >bigdata21.webmedia.int/192.168.81.110:8050 >16/06/20 21:25:52 INFO db.DBInputFormat: Using read commited transaction >isolation >16/06/20 21:25:52 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: >SELECT MIN(`id`), MAX(`id`) FROM `sqoop_test` >16/06/20 21:25:52 INFO mapreduce.JobSubmitter: number of splits:2 >16/06/20 21:25:52 INFO mapreduce.JobSubmitter: Submitting tokens for >job: job_1460979043517_0118 >16/06/20 21:25:53 INFO impl.YarnClientImpl: Submitted application >application_1460979043517_0118 >16/06/20 21:25:53 INFO mapreduce.Job: The url to track the job: >http://bigdata21.webmedia.int:8088/proxy/application_1460979043517_0118/ >16/06/20 21:25:53 INFO mapreduce.Job: Running job: job_1460979043517_0118 >16/06/20 21:25:58 INFO mapreduce.Job: Job job_1460979043517_0118 running >in uber mode : fals
Atlas and sqoop hook
Hi Downloaded and compiled atlas 0.7. Hive hook is working - create table [tablename] as select * from [src tablename] is working and data lineage is generated in atlas. Next I tried sqoop hook and followed http://atlas.incubator.apache.org/Bridge-Sqoop.html Command: sqoop-import --connect jdbc:mysql://mysqlhost/test --table sqoop_test --split-by id --hive-import -hive-table sqoop_test19 --username margusja --P creates a new table in Hive and new table is in atlas also but no data lineage I see from http://hortonworks.com/hadoop-tutorial/cross-component-lineage-apache-atlas/ There I can see that extra config parameters are loaded (in picture https://raw.githubusercontent.com/hortonworks/tutorials/atlas-ranger-tp/assets/cross-component-lineage-with-atlas/8-sqoop-import-finish.png) and kafka producer creating ouutput but in my command: sqoop-import --connect jdbc:mysql://mysqlhost/test --table sqoop_test --split-by id --hive-import -hive-table sqoop_test19 --username margusja --P there is no extra output only: Warning: /usr/hdp/2.4.0.0-169/accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. 16/06/20 21:25:47 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6.2.4.0.0-169 16/06/20 21:25:47 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead. 16/06/20 21:25:47 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for output. You can override 16/06/20 21:25:47 INFO tool.BaseSqoopTool: delimiters with --fields-terminated-by, etc. 16/06/20 21:25:47 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset. 16/06/20 21:25:47 INFO tool.CodeGenTool: Beginning code generation 16/06/20 21:25:47 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `sqoop_test` AS t LIMIT 1 16/06/20 21:25:47 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `sqoop_test` AS t LIMIT 1 16/06/20 21:25:47 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/hdp/2.4.0.0-169/hadoop-mapreduce Note: /tmp/sqoop-root/compile/49b525e14ebd68542d86b68dc399bd84/sqoop_test.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. 16/06/20 21:25:48 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/49b525e14ebd68542d86b68dc399bd84/sqoop_test.jar 16/06/20 21:25:48 WARN manager.MySQLManager: It looks like you are importing from mysql. 16/06/20 21:25:48 WARN manager.MySQLManager: This transfer can be faster! Use the --direct 16/06/20 21:25:48 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path. 16/06/20 21:25:48 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql) 16/06/20 21:25:48 INFO mapreduce.ImportJobBase: Beginning import of sqoop_test SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/hdp/2.4.0.0-169/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/2.4.0.0-169/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 16/06/20 21:25:50 INFO impl.TimelineClientImpl: Timeline service address: http://bigdata21.webmedia.int:8188/ws/v1/timeline/ 16/06/20 21:25:50 INFO client.RMProxy: Connecting to ResourceManager at bigdata21.webmedia.int/192.168.81.110:8050 16/06/20 21:25:52 INFO db.DBInputFormat: Using read commited transaction isolation 16/06/20 21:25:52 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN(`id`), MAX(`id`) FROM `sqoop_test` 16/06/20 21:25:52 INFO mapreduce.JobSubmitter: number of splits:2 16/06/20 21:25:52 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1460979043517_0118 16/06/20 21:25:53 INFO impl.YarnClientImpl: Submitted application application_1460979043517_0118 16/06/20 21:25:53 INFO mapreduce.Job: The url to track the job: http://bigdata21.webmedia.int:8088/proxy/application_1460979043517_0118/ 16/06/20 21:25:53 INFO mapreduce.Job: Running job: job_1460979043517_0118 16/06/20 21:25:58 INFO mapreduce.Job: Job job_1460979043517_0118 running in uber mode : false 16/06/20 21:25:58 INFO mapreduce.Job: map 0% reduce 0% 16/06/20 21:26:02 INFO mapreduce.Job: map 50% reduce 0% 16/06/20 21:26:03 INFO mapreduce.Job: map 100% reduce 0% 16/06/20 21:26:03 INFO mapreduce.Job: Job job_1460979043517_0118 completed successfully 16/06/20 21:26:03 INFO mapreduce.Job: Counters: 30 File System Counters FILE: Number of bytes read=0 FILE: Number of bytes written=310818 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=197
[jira] [Commented] (ATLAS-859) Sqoop hook can also extend from AtlasHook
[
https://issues.apache.org/jira/browse/ATLAS-859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15329546#comment-15329546
]
Hemanth Yamijala commented on ATLAS-859:
It does. Sorry - my comment is completely garbled. It should read:
However, more important is that *it sends* entities to Kafka using methods of
the {{AtlasHook}}. Hence things like the retries and other logic is indeed
taken care of still for the Sqoop hook. Hence, I am closing this as not a bug.
Thanks for pointing this out, by asking the question.
> Sqoop hook can also extend from AtlasHook
> -
>
> Key: ATLAS-859
> URL: https://issues.apache.org/jira/browse/ATLAS-859
> Project: Atlas
> Issue Type: Bug
>Reporter: Hemanth Yamijala
>Assignee: Hemanth Yamijala
>
> Of the 4 hooks integrating with Atlas in the Atlas code base, the Sqoop hook
> alone does not extend from a common base class - {{AtlasHook}}. As a result,
> enhancements made to Atlas hook might be missed out by Sqoop hook. Fixing
> this inconsistency would be a good thing, IMO.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (ATLAS-859) Sqoop hook can also extend from AtlasHook
[
https://issues.apache.org/jira/browse/ATLAS-859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15329540#comment-15329540
]
Madhan Neethiraj commented on ATLAS-859:
bq. However, more important is that to send entities to Kafka using methods of
the AtlasHook. Hence things like the retries and other logic is indeed taken
care of still for the Sqoop hook. Hence, I am closing this as not a bug.
[~yhemanth] does Sqoop hook currently use AtlasHook methods to send
notifications to Kafka? If not, is there a separate JIRA to track this?
> Sqoop hook can also extend from AtlasHook
> -
>
> Key: ATLAS-859
> URL: https://issues.apache.org/jira/browse/ATLAS-859
> Project: Atlas
> Issue Type: Bug
>Reporter: Hemanth Yamijala
>Assignee: Hemanth Yamijala
>
> Of the 4 hooks integrating with Atlas in the Atlas code base, the Sqoop hook
> alone does not extend from a common base class - {{AtlasHook}}. As a result,
> enhancements made to Atlas hook might be missed out by Sqoop hook. Fixing
> this inconsistency would be a good thing, IMO.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (ATLAS-859) Sqoop hook can also extend from AtlasHook
[
https://issues.apache.org/jira/browse/ATLAS-859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hemanth Yamijala resolved ATLAS-859.
Resolution: Not A Bug
[~shwethags] pointed out that the Sqoop hook class already extends from another
class called {{SqoopJobDataPublisher}}. Hence, it would not be possible to make
it extend from {{AtlasHook}}.
However, more important is that to send entities to Kafka using methods of the
{{AtlasHook}}. Hence things like the retries and other logic is indeed taken
care of still for the Sqoop hook. Hence, I am closing this as not a bug.
> Sqoop hook can also extend from AtlasHook
> -
>
> Key: ATLAS-859
> URL: https://issues.apache.org/jira/browse/ATLAS-859
> Project: Atlas
> Issue Type: Bug
>Reporter: Hemanth Yamijala
>Assignee: Hemanth Yamijala
>
> Of the 4 hooks integrating with Atlas in the Atlas code base, the Sqoop hook
> alone does not extend from a common base class - {{AtlasHook}}. As a result,
> enhancements made to Atlas hook might be missed out by Sqoop hook. Fixing
> this inconsistency would be a good thing, IMO.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (ATLAS-859) Sqoop hook can also extend from AtlasHook
[
https://issues.apache.org/jira/browse/ATLAS-859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hemanth Yamijala reassigned ATLAS-859:
--
Assignee: Hemanth Yamijala
> Sqoop hook can also extend from AtlasHook
> -
>
> Key: ATLAS-859
> URL: https://issues.apache.org/jira/browse/ATLAS-859
> Project: Atlas
> Issue Type: Bug
>Reporter: Hemanth Yamijala
>Assignee: Hemanth Yamijala
>
> Of the 4 hooks integrating with Atlas in the Atlas code base, the Sqoop hook
> alone does not extend from a common base class - {{AtlasHook}}. As a result,
> enhancements made to Atlas hook might be missed out by Sqoop hook. Fixing
> this inconsistency would be a good thing, IMO.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (ATLAS-859) Sqoop hook can also extend from AtlasHook
Hemanth Yamijala created ATLAS-859:
--
Summary: Sqoop hook can also extend from AtlasHook
Key: ATLAS-859
URL: https://issues.apache.org/jira/browse/ATLAS-859
Project: Atlas
Issue Type: Bug
Reporter: Hemanth Yamijala
Of the 4 hooks integrating with Atlas in the Atlas code base, the Sqoop hook
alone does not extend from a common base class - {{AtlasHook}}. As a result,
enhancements made to Atlas hook might be missed out by Sqoop hook. Fixing this
inconsistency would be a good thing, IMO.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
