Ian,
Thanks for helping. This example now worked for me using a single node. Now
I'm trying to run the word count example in a real cluster.
I read in the "Hyracks: A Flexible and Extensible Foundation for
Data-Intensive Computing" article:
Hyracks currently allows users to choose between two ways of describing the
> scheduling choices for each of the operators in a job. One option is to
> list the exact number of partitions of an operator to be created at runtime
> along with a list of choices of worker machines for each partition. Another
> option is just to specify the number of partitions to use for an operator,
> leaving Hyracks to decide the location assignments for each of its
> partitions. We are currently working on a third, longer-term option
> involving automatic partitioning and placement based on the estimated
> resource requirements for operators and the current availability of
> resources at the worker machines.
>
The third way is already implemented? And also, how to choose between the
first and the second ones while writing the Java code expressing the DAG?
I've noticed that in the text-example some partitions are created using the
method *createPartitionConstraint(spec, op, splits); *which receives the
job, an operator and file splits and next calls the method:
public static void addAbsoluteLocationConstraint(JobSpecification spec,
>> IOperatorDescriptor op,
>
> String... locations) {
>
> addPartitionCountConstraint(spec, op, locations.length);
>
> for (int i = 0; i < locations.length; ++i) {
>
> spec.addUserConstraint(new Constraint(new
>> PartitionLocationExpression(op.getOperatorId(), i),
>
> new ConstantExpression(locations[i])));
>
> }
>
> }
>
>
So if I want to create more partitions of the same operator I just need to
add new User Constraints with the location (node) where I want it to run?
Thanks!
Ígor Chagas Marques
Em qua, 24 de abr de 2019 às 02:35, Ian Maxon <[email protected]> escreveu:
> Hey Igor,
> Unfortunately those docs are a bit outdated. You need a config in most
> cases. We test it like this but the docs and packaging never caught up
> after we fixed our config story a bit. I have a patch up to refresh things
> a bit but what you have in the source as is will work. For example, if you
> had this as single.conf:
>
> [nc]
> address = 127.0.0.1
> cluster.address = 127.0.0.1
> cluster.port = 1099
>
> [cc]
> address = 127.0.0.1
> app.class = org.apache.hyracks.control.cc.BaseCCApplication
>
> [localnc]
> node.id = textserver-nc1
>
> And you started the scripts like:
> bin/hyrackscc -config-file conf/single.conf
> bin/hyracksnc -config-file conf/single.conf
>
> It would start a basic cluster.
>
> The wordcount example needs quite a few more args though, you would execute
> it like so with a local cluster:
>
> textclient -host localhost \
> -infile-splits textserver-nc1:data/file1.txt \
> -outfile-splits textserver-nc1:data/file1.txt.out \
> -algo -hash
>
> Where the paths are relative to the pwd of the NC process. Also the text
> example is only really distributed in the source as it stands right now, so
> you'd have to compile from the top level directory in the source (mvn clean
> package -DskipTests) and find the proper executable in
>
> hyracks-fullstack/hyracks/hyracks-examples/text-example/textclient/target/appassenbler/bin
>
> - Ian
>
>
> > Hi, devs
> >
> > I'm trying to work with hyracks without Asterix. However, I got a little
> > confused with how to run the simple WordCount example
> > (hyracks-examples/text-example).
> > I've read in a README file (hyracks-server/docs/README) that I should
> begin
> > starting CC and NC with bin/hyrackscc.sh and bin/hyracksnc.sh scripts.
> When
> > I tried to run these I received the following output:
> > "my-host-name
> > eth0: error fetching interface information: Device not found
> > em1: error fetching interface information: Device not found"
> > They are running? Is that ok? How to check it out?
> > And at last but not least, if the CC and NC are going ok, how to run the
> > WordCount example job on it?
> >
> > Thanks!
> > Ígor Chagas Marques
> >
>
