[
https://issues.apache.org/jira/browse/SQOOP-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15281819#comment-15281819
]
Attila Szabo commented on SQOOP-2906:
-------------------------------------
Dear Joeri,
I'd like to run a set of measurements on my own ( to check if your solution +
mine on this issue are synergies or not and to check further improvements )
I found this hprofiler quite exciting however I had difficulties in using it (
my bad ).
Could you help with a the following questions:
- How did you ensure that the extra parameters are passed to the mappers and
the Sqoop1 cmd line tool?
- In case of -c which server/service IP is needed there and in what kind of
format?
- Is it enough to use only with -h ?
- If I don't wanna use SSH keys is it a valid solution if I type my root
password all the time when it executes SSH?
- Should it work just out of box ( e.g. defining -h -j -f -t ) or is any
postprocess needed?
- Did you use a hadoop distribution like CDH or just pure hdfs and sqoop1?
Many thanks in advance!
Maugli
> Optimization of AvroUtil.toAvroIdentifier
> -----------------------------------------
>
> Key: SQOOP-2906
> URL: https://issues.apache.org/jira/browse/SQOOP-2906
> Project: Sqoop
> Issue Type: Improvement
> Reporter: Joeri Hermans
> Assignee: Joeri Hermans
> Labels: avro, hadoop, optimization
> Attachments: diff.txt
>
>
> Hi all
> Our distributed profiler indicated some inefficiencies in the
> AvroUtil.toAvroIdentifier method, more specifically, the use of Regex
> patterns. This can be directly observed from the FlameGraph generated by this
> profiler (https://jhermans.web.cern.ch/jhermans/sqoop_avro_flamegraph.svg).
> We implemented an optimization, and compared this with the original method.
> On our testing machine, the optimization by itself is about 500% (on average)
> more efficient compared to the original implementation. We have yet to test
> how this optimization will influence the performance of user jobs.
> Any suggestions or remarks are welcome.
> Kind regards,
> Joeri
> https://github.com/apache/sqoop/pull/18
> Writeup:
> https://db-blog.web.cern.ch/blog/joeri-hermans/2016-04-hadoop-performance-troubleshooting-stack-tracing-introduction
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
