Using sqoop on hdp 3.1 to import Oracle DB data into HDFS as parquet and
expecting it to appear in atlas as an entity (I assume it should be the
default sqoop_process type). However, after running the sqoop job, nothing
appears in atlas. Sqoop job looks like...
sqoop import \
-Dmapreduce.map.memory.mb=3144 -Dmapreduce.map.java.opts=-Xmx1048m \
-Dyarn.app.mapreduce.am.log.level=DEBUG \
-Dmapreduce.map.log.level=DEBUG \
-Dmapreduce.reduce.log.level=DEBUG \
-Dmapred.job.name="Ora import table $tablename" \
-Djava.security.egd=file:///dev/urandom \
-Djava.security.egd=file:///dev/urandom \
-Doraoop.timestamp.string=false \
-Dmapreduce.map.max.attempts=10 \
$oracle_cnxn_str \
--as-parquetfile \
--target-dir $importdir \
-query "select a.*, current_date as etl_date from $tablename a
where 1=1 AND \$CONDITIONS" \
--split-by $splitby \
--where "1=1" \
--num-mappers 12 \
--delete-target-dir
Looking in the hdp ambari interface for atlas, I can see that the sqoop
plugin for atlas is configured[image: enter image description here]
<https://i.stack.imgur.com/U7qYp.png>(did not initially do any manual
tweaking to it after installing atlas according to the docs
<https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/installing-ranger/content/ranger_install_ranger_plugin_settings.html>).
Adding the property[image: enter image description here]
<https://i.stack.imgur.com/LRukd.png>based on another Hortonworks post
<http://community.hortonworks.com/answers/52257/view.html> and attempting
the sqoop job again still shows nothing in the atlas UI. Anyone with more
experience with hdp and atlas have any debugging suggestions or fixes?
--
This electronic message is intended only for the named
recipient, and may
contain information that is confidential or
privileged. If you are not the
intended recipient, you are
hereby notified that any disclosure, copying,
distribution or
use of the contents of this message is strictly
prohibited. If
you have received this message in error or are not the
named
recipient, please notify us immediately by contacting the
sender at
the electronic mail address noted above, and delete
and destroy all copies
of this message. Thank you.
